JP2006003263A - Visual information processor and application system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a visual information processor and an application system capable of automatically generating and utilizing as-built 3D-CAD data in real time. <P>SOLUTION: The visual information processor has the function of inputting image information of a movable imaging apparatus and processes the input image to automatically generate CAD data of an object existing in a movement space, and the function of automatically updating the CAD data, and also has the function of inputting a command signal for retrieving the CAD data and retrieves the CAD data in accordance with the command signal, and the function of outputting information of the retrieved CAD data. Furthermore, a variety of autonomous control type automatic machines are made up by combining the visual information processor with a robot control device and the like. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a visual information processing apparatus and a system to which the visual information processing apparatus is applied. In particular, this is a visual information processing device suitable for automatic generation of as-built 3D-CAD data, visual guidance control devices, mobile inspection devices, and monitoring service systems for automatic machines such as robots.

  As a technique for generating 3D-CAD data based on image information obtained by photographing with an imaging device, there is a technique for obtaining three-dimensional coordinates from a plurality of photographs. For example, there is a software called 3D Builder, which was developed by 3D Construction Company in the US and sold in Japan by K.G. When the operator inputs corresponding points between a plurality of photos one by one, the point and the position and orientation of the camera of the imaging device that took the image are calculated as three-dimensional coordinates, and CAD data can be created interactively. Texture mapping can be performed.

  In Japanese Patent Laid-Open Nos. 2002-32741 and 2002-32742, if feature points (windows) are designated in an initial image or feature points are first set based on feature point position setting data, continuous Feature points are automatically tracked in the input image, so that the operator does not have to input corresponding points one by one, three-dimensional coordinates are calculated, and the point data are connected to perform texture mapping. A concept of a system that can be disclosed is disclosed.

  In addition, as a method for searching for corresponding points in stereo measurement, Japanese Examined Patent Publication No. 8-16930 discloses the correlation between images, and an operator inputs corresponding points at locations where matching is not good due to the correlation coefficient and the sum of squared residuals. Such a method is disclosed.

JP 2002-32741 A JP 2002-32742 A Japanese Patent Publication No. 8-16930 Japanese Patent Laid-Open No. 02-137072

Software sold by 3D Builder, KK TG IEICE Transactions D-II Vol. J80-D-II No. 5 PP. 1136-1143, May 1997 IPSJ Research Report Computer Vision and Image Media, No. 131-022, 2001

  However, a corresponding feature point in a plurality of photographs can be searched automatically and accurately, an operator can avoid input setting, accurate CAD data can be generated without texture mapping, and an imaging device The CAD camera is actively moved so that CAD data can be generated more accurately, and it can be used for control of robots, etc. so that the camera can be generated with a certain degree of accuracy at the beginning to bring the shooting camera closer to the front, back, left, and right It has not been considered to make it possible to improve the accuracy in real time by moving or moving, or to integrate CAD data generated by moving a plurality of photographing cameras. In addition, after or while generating CAD data of as-built, combined with the function to search for arbitrary CAD data, it can be combined with a function to search for arbitrary CAD data, navigation systems such as automobiles, navigation systems, mobile inspection devices, monitoring, etc. It was not considered to be applied to service systems or to autonomous control robots.

  An object of the present invention is to enable automatic generation of as-built 3D-CAD data in real time, and in combination with a CAD data search function, as a visual guidance control device for various automatic machines such as robots, navigation systems, and mobile inspections An object of the present invention is to provide a visual information processing apparatus applicable as a device, a monitoring service system, and the like, and a system thereof.

  The inventors have developed a visual information processing apparatus as described in the specification and drawings of Japanese Patent Application No. 2002-368134 as a basic means for achieving the above object. The present invention proposes further means based on this basic means.

  The first means includes an image information processing device that inputs image information obtained by photographing an object with a movable imaging device and generates CAD data of the object, and a storage device that stores the generated CAD data. In the visual information processing apparatus, the image information processing apparatus includes an update target determination processing function for determining whether or not the generated new CAD data is an update target of already generated CAD data, and an existing target to be updated. A CAD data accuracy comparison processing function for comparing the accuracy of CAD data and new CAD data, and CAD for updating existing CAD data with new CAD data when the accuracy of new CAD data is higher than the accuracy of existing CAD data to be updated It is characterized by having a data update processing function.

  This first means has a function in the visual information processing device that receives image information obtained by a movable imaging device as an input and processes the input image to generate CAD data of an object existing in the moving space. CAD data of various objects existing in the moving space can be easily generated.

  Further, when the generated new CAD data is an update target determination function for determining whether or not the existing CAD data is already generated, and when the new CAD data is an update target of the existing CAD data, When the accuracy of the CAD data is compared with the accuracy of the existing CAD data generated before the update target and the accuracy is improved, a function for updating the CAD data is also provided. By approaching and obtaining image information of a more detailed portion of the object, it becomes possible to update while generating more detailed CAD data of the object. In addition, since the moving imaging apparatus can input image information of the top, bottom, left, and right sides and the back side of the object, not only the color and shape of the front side of the object, but also the top and bottom, left and right sides, and even the color and shape of the back side, CAD It can be generated as data. The update target determination processing function for determining whether or not the new CAD data is an update target of the existing CAD data is, for example, an update target when there is no previously generated CAD data at the same location after the CAD data is generated. CAD data is added as it is, and if it exists, it is to be updated. By comparing the accuracy of the new CAD data with the accuracy of the existing CAD data to be updated when the new CAD data is an update target of the existing CAD data, and updating the CAD data every time when the accuracy is improved, Even if CAD data of the same part is generated, it is not updated except when the accuracy is improved, and is updated when the accuracy is improved. By repeating this processing, The accuracy of the object existing in the moving space is improved every time the CAD data is updated, and finally, the CAD data with high accuracy can be obtained. As described above, if only the update that improves the accuracy of the CAD data is performed, the CAD data is generated with a certain degree of accuracy from the image information obtained when the imaging apparatus is away from the target. However, according to the image information taken by the imaging device close to the object, the generation accuracy can be improved accordingly.

  Even in operation, if the accuracy of the CAD data is not good, for example, the CAD data can be used for rough positioning control with respect to a distant object. Since it is possible to perform positioning control with higher accuracy by using the CAD data, CAD data can be continuously and effectively used even from the first stage with lower accuracy.

  In addition, with such a CAD data update function, a lot of similar CAD data is not accumulated regardless of how many times CAD data is generated for the same part of a non-moving object. If a large number of objects are generated, it is a case where moving objects are included. Therefore, the movement of the moving target object such as smoke or steam moves by tracking and so on. Tracking of spotlights or moving shadows can be easily performed by referring to the update history because CAD data is sequentially updated.

  The second means is to distinguish the contour line from the actual line in the CAD data in the visual information processing apparatus configured by applying the first means.

  In such a second means, in the visual information processing apparatus configured by applying the first means, the CAD data distinguishes the contour line from the actual line. In the case of capturing a line, since the line detected as the contour line is also a line indicating the shape of the object, it can be stored as shape data. In addition, when a line actually exists like a pattern line, the line is also shape data having absolute position information, and is also pattern information forming the pattern of the object. Therefore, if the generated CAD data is managed by distinguishing between the contour line and the actual line, if you want to handle the shape data, use more information using both the contour line and the actual line. An accurate shape can be recognized and processed.

  On the other hand, when trying to search for patterns that are pattern matching, etc., it is difficult to search without matching because it does not include correct lines because the data of contour lines are handled together. In this case, the pattern can be easily searched for by matching with the pattern to be searched using only the actual line data. Since the contour line and the actual line are distinguished and managed, only the necessary line can be efficiently used when necessary. Accordingly, CAD data can be efficiently generated and searched even in an environment where a patterned object and an unpatterned object are mixed. Also, the spotlight or shadow outline is an outline if it is known as the outline of the surface that the light is on or the outline of the shadow that is not, but it is an object that blocks light from the light source or light source. If it does not move, it will not change if the imaging device is moved, so it has both features that can be handled as a pattern. When it is understood, the distinction contents may be changed from the actual (pattern) line to the contour line. If it is known that the obstructing object moves but does not move at the time interval using CAD data, the shadow and other contour lines are considered as other contours in consideration of the case where the shadow is used as a pattern matching pattern. You may make it manage separately from a line or an actual line.

  Further, the distinction between the contour line and the actual line is that the position of the line segment, specifically, the feature points constituting the line segment is measured from image information obtained by imaging from a plurality of different viewpoints. In the case of contour lines, since the contour lines photographed and extracted from a plurality of different viewpoints are strictly measured three-dimensionally using different parts of the object as the same contour line, the corresponding points are strict in stereo measurement. (There is an error corresponding to the deviation). In the case of an actual line, the deviation is extremely small. Therefore, the degree of deviation of the corresponding point may be obtained by determining the degree of deviation of the corresponding point. Instead of giving it to the CAD data and distinguishing between the actual line, the contour line, and the line type, the line segment may be distinguished by the degree of deviation.

  The third means is to allow the CAD information to have information relating to the time when the CAD data is generated in the visual information processing apparatus configured by selectively applying the first and second means.

  Such a third means is a visual information processing apparatus configured by selectively applying the first to second means, so that information relating to the time generated in the CAD data is provided. When the generated CAD data is referred to, the time when the CAD data is generated can be referred to. Therefore, for example, if there is a difference when comparing the CAD data generated at the same place with the CAD data generated before, the current time and the time registered in the previous CAD data It becomes possible to recognize how many hours ago the state has changed. When referring to CAD data generated while generating CAD data, if the generation time is in the CAD data, it can be confirmed which is the CAD data generated earlier.

  In a visual information processing apparatus configured by selectively applying the first to third means, the fourth means receives image information from a movable imaging device as an input, processes the input image information, and moves A function of automatically generating CAD data of an object existing in space is provided, and the function of automatically generating CAD data is a function of associating feature points in image information of a plurality of imaging devices captured at the same time with image information. The stereo measurement is performed from the characteristic part of the three-dimensional object, and at least three points among the points measured in stereo are tracked to follow the corresponding points in the image information due to the movement of the imaging device so as to relate the three-dimensional. This is realized by executing a method for obtaining coordinate data.

  In the fourth means, in the visual information processing apparatus configured by selectively applying the first to third means, the image information of the movable imaging device is input, the input image information is processed, and the moving space is processed. The function of automatically generating the CAD data of the object existing in the image data, and the function of automatically generating the CAD data are the correspondence of the feature points in the image information of the plurality of imaging devices captured at the same time. Three-dimensional coordinates are obtained by performing stereo measurement from the characteristic part and tracking corresponding points in the image information due to movement of the imaging device by tracking at least three stable points among the points measured in stereo. Since it has been realized by executing a method for obtaining data, it is easy to measure the three-dimensional position of the feature point in the image information first taken by stereo measurement. Since the three-dimensional position of many feature points can be measured at a time, the three stable points with the highest accuracy during measurement are extracted from the image information captured at the next time when the imaging device moves. If there can be, since there is known data of the three-dimensional position of the three points, all the coefficients of the calculation formula for performing the coordinate transformation between the camera coordinate system before the movement and the camera coordinate system after the movement are known. The three-dimensional position information of all feature points in the camera coordinate system after movement measured by the image information can be easily converted into position information in the camera coordinate system before movement. It is also possible to search for corresponding points in the two-dimensional image information when tracking at least three points, but a plurality of corresponding point candidates remain and one of them cannot be automatically specified by the program. Even in such a case, since the three-dimensional positions of all feature points at the moved position can be obtained by stereo measurement in the moved camera coordinate system, coordinates can be selected by selecting three points from corresponding points of a plurality of candidates. Find all the coefficients of the calculation formula that performs the conversion, and use the conversion calculation formula to calculate the three feature points that have the largest number of matching feature points when all the feature points of the camera coordinate system after movement are converted to the coordinate system before movement. Since it is possible to easily determine that the combination is the correct three corresponding points, it becomes possible to easily obtain the correct coordinate transformation calculation formula by confirming the selected combination of all three candidate points, The imaging device Newly by repeating that we incorporate the image information, it is possible to easily generate three-dimensional CAD data of an object existing in the three-dimensional space. Furthermore, when the feature point that seems to be the same point is measured again three-dimensionally, if the measurement accuracy is further improved based on the measured condition, the measured position information of the feature point is updated. By moving the imaging apparatus and approaching the measurement target, it is possible to automatically generate more accurate three-dimensional CAD data. In addition, only the method of measuring the three-dimensional coordinates of the feature points by stereo measurement every time the image pickup apparatus moves may be used. However, if this means is used, the previously captured image information is retained and the above-described tracking is performed. Since the three-dimensional measurement can be performed again using the captured image information when the image information is moved to a position sufficiently away from the position where the image information was taken, the measurement accuracy can be further improved. In this case, since the three-dimensional measurement can be performed again at an interval sufficiently wider than the interval between the imaging devices that perform stereo measurement, the measurement accuracy can be easily improved.

  The fifth means is that the position of the feature point in the image information changes due to the moving object, or the position of the feature point in the image information changes due to the movement of the imaging device, or the image is taken by a plurality of imaging devices. In the corresponding point search method for associating feature points that change the position of feature points in image information obtained from each imaging device, the image information obtained from the imaging devices is composed of color boundary line segments that connect the feature points. In other words, the similarity of feature parts in the region excluding at least the background region divided by the color boundary line segment is compared, and a higher one is narrowed down as a corresponding point candidate.

  In the fifth means, the feature point position in the image information changes due to the moving object, or the feature point position in the image information changes because the image pickup apparatus moves, or the image pickup is performed by a plurality of image pickup apparatuses. In the corresponding point search method for associating feature points that change the position of feature points in the image information obtained from each imaging device, the image information obtained from the imaging device is composed of color boundary line segments connecting the feature points Image data is converted into two-dimensional CAD data, and the similarity of feature parts in the region excluding the background region at least separated by the color boundary line is compared to narrow down the higher ones as corresponding point candidates. It becomes possible to easily search for corresponding points with sufficient accuracy without using local two-dimensional image information including a background that changes depending on the position and angle of the imaging device with respect to the target. . The effect | action in which the ease is implement | achieved concretely is demonstrated. The local feature part of a line drawing composed of two-dimensional CAD data line segments is the position of the feature points of the line segments, the color information on both sides of the line segment drawn as a boundary line, and the connection state of the line segments Etc. Regarding the position, for example, when an image is captured at a minute time interval of the moving camera, the moving range of the feature point is within a minute region near the previously photographed feature point although it is related to the moving speed of the imaging device. is there. The smaller the period for capturing image information, the smaller the micro area, so that the number of candidate points when searching for corresponding points can be narrowed down. On the other hand, when the positional relationship between imaging devices with different shooting positions is known in advance, such as stereo measurement, the corresponding points to be searched for are always on a predetermined straight line related to the positions of the feature points on the other side to be searched. Since it exists, the points on the straight line can be narrowed down as candidate points. Further, the corresponding points can be further narrowed down by checking whether the color information on the left and right of the line segment constituted by the feature points matches among the candidate points. At this time, the screening is not necessarily performed only on the condition that the color information on both sides is matched, but if the values match on one side, the corresponding points are sufficiently candidates. That is, when the line segment is an actual line, the color information on both sides matches, but when the line is a contour line, either one does not necessarily match because it is background color information. This is because the background color information has a different background that appears next to the contour line depending on the angle at which the contour is photographed. However, since one side of the line segment is always the color information of the object in front of which the contour is generated, the color information of the corresponding points match even if the shooting angle changes. Therefore, when the color information on both sides matches, it becomes a corresponding point candidate, and when the color information on one side matches, it becomes a corresponding point candidate. The color information becomes color information on the opposite side of the boundary when comparing the similarity in a local area of one pixel order. However, when the processing is performed with the local area on the order of several pixels, It is also possible to determine the corresponding candidate points where there is a minute region with a high correlation value of the minute region excluding the background minute region by taking a correlation for each minute region formed by the boundary line segment extending from good. As described above, for the candidate points of the corresponding points narrowed down by the color information of the position and both sides or the minute area of the boundary line, when the connection information of the line segment information and the color information of the connected line segments match It will be more narrowed down as a corresponding point candidate. Therefore, when the positional relationship between imaging devices with different shooting positions is known in advance, such as stereo measurement such as stereo measurement, the combination of three points is sequentially performed from the narrowed-down corresponding point candidates, and the coordinate conversion calculation formula Since the combination of the three points where many feature points coincide with each other when the coefficients are obtained is the three points with the correct correspondence, the correct corresponding points can be easily obtained with high accuracy. Here, even if the corresponding points match in stereo measurement, they may match in the illusion, so even if the imaging device continues to move and stereo measurement from different viewpoints, the corresponding points match in the same way. If so, it is better to identify and manage it as a stable point and to preferentially select the three points used for the next tracking.

  On the other hand, in the case of measurement by tracking instead of stereo measurement, it is effective to narrow down the corresponding point candidates by increasing the sampling speed and reducing the search range. However, if it is difficult to increase the processing speed in relation to the processing speed, the position information of a certain range, the color information on both sides of the line segment, and the connection information of the line segment and the similarity of the positional relationship with the surroundings Narrow down further, and finally select three points from the corresponding point candidates that have a higher overall match and obtain the coefficients of the coordinate conversion calculation formula to find all the other corresponding candidate points at the same coordinates. Three points having a large number of corresponding point candidates that match when converted into a system are determined to be the most reliable corresponding points, and the process is continuously continued. In the case of tracking by stereo measurement, the three-dimensional CAD data of the three-dimensional feature points measured by the camera coordinate system before and after tracking movement is obtained by stereo measurement before searching for corresponding points. By making the coefficient of the coordinate transformation calculation formula having the highest matching of the feature points above the correct conversion coefficient, tracking can be continued with high accuracy. In this way, it is possible to automatically and reliably search for corresponding points. Here, when three probable corresponding points are not obtained, an imaging device in a period in which an acceleration sensor or a gyro sensor is provided so as to move together with the imaging device and the feature point cannot be tracked by information of these sensors It is preferable to interpolate so as to obtain the movement information.

  In the sixth means, the similarity of the feature parts compared in the fifth means is the similarity of the vector to the other feature points subtracted from the corresponding point candidates, or the similarity between the vector and the feature point of the vector tip. It is to be combined with a method of narrowing down the corresponding points as the corresponding points using the similarity of the color information.

  In the sixth means, the similarity of the feature parts compared in the fifth means is the similarity of the vector to the other feature point subtracted from the corresponding point candidate or the similarity of the vector (the same length in the same direction). This is combined with the method of narrowing down the corresponding points as the corresponding points using the similarity of the color information of the feature point at the tip of the vector (the degree of coincidence of the color information on the top, bottom, left and right). Matching processing of image information (data) replaced with two-dimensional CAD data can be performed at high speed. Here, in the case of matching with conventional image information of a two-dimensional array, a two-dimensional correlation coefficient is obtained or a long time is required for one matching calculation, but the amount of data is CAD At the time of replacement with data, the two-dimensional image information amount has been reduced to the number of feature points, and the vector similarity can be obtained by comparing two numerical values of direction and distance. Therefore, it becomes simpler than the matching calculation of two-dimensional image information. In addition to the vector direction and distance, even when comparing the color information similarity of the feature point at the tip of the vector, only the color information around one feature point is obtained, and the number of vectors subtracted from one corresponding point candidate Even if the number of tens of hundreds is several hundred, the amount of calculation for obtaining the correlation coefficient in a two-dimensional image can be reduced, so that accurate matching can be performed more quickly and easily. Also in this case, as described in the fifth means, the vector similarity is compared for each of the regions to be divided into the background and non-background color boundary line segments.

  The seventh means is that the color boundary line segment extracting method of the fifth and sixth means performs the connection of the inflection points of the image information such as the luminance signal as the color boundary line segment.

  In this seventh means, the color boundary line segment extraction method of the fifth and sixth means executes the connection of the inflection points of the image information such as the luminance signal as the color boundary line segment. The line segment can be extracted well and stably. The luminance signal here is not limited to a luminance signal of a monochrome image, but may be a luminance signal whose brightness corresponds to distance information, or a luminance signal of color information (R, G, B, etc.) of color image information. Alternatively, a luminance signal obtained by measuring the transmittance inside the object by X-ray CT may be used. When handling a plurality of pieces of luminance information as in the case of color information, feature points obtained by the same process are synthesized and used. The inflection point is a part where the rate of change of the left and right color information changes, so even if the lighting conditions change and the overall brightness changes, or even under conditions where the illumination brightness changes continuously An inflection point occurs where there is an inflection point on the object being photographed, so there is an inflection point that is not affected by a slight change in lighting conditions. The place to do can be detected stably and easily as a line segment. The sensitivity may be constant for the entire captured image, but the highest sensitivity can be obtained anywhere on the entire screen by locally changing the threshold for detecting the inflection point as an inflection point according to the surrounding luminance information. It may be possible to detect. From the viewpoint of processing speed, the sensitivity may be automatically adjusted so that the number of detected line segments in the captured image information is about a predetermined number. By doing so, an appropriate amount of line segments (feature points) can always be obtained, so tracking can be performed stably and continuously, and the amount of line segments can be prevented from increasing beyond the processing speed. Can do. Since feature points need to be extracted stably, the extracted inflection points change slightly depending not only on the lighting but also the shooting viewpoint, so a line segment is generated from the inflection points in order to stabilize. Sometimes, a line segment may be generated by fitting using a least square method or the like, or an intersection of a previous line segment extended by extending the line segment may be used as a feature point. In addition, since a lot of surface data can be defined for a line segment to be generated if a closed loop is generated, color grouping processing of a two-dimensional captured image is performed, and the entire captured image information is divided into closed loop color region boundary line segments. The boundary line segment branch point or sharp bend point may be used as a feature point, but the boundary line segment is further associated with the line segment generated from the inflection point. You may make it produce more closed-loop line segments as a reference.

  The eighth means is a method for extracting feature points in the image information by directly or indirectly taking image information obtained from the imaging device, and for feature points of line segments and feature points of closed-loop figures. Both are extracted as feature points.

  In the eighth means, in the method of directly or indirectly taking the image information obtained from the imaging device and extracting the feature point in the image information, the feature point of the line segment and the feature point of the closed-loop figure are used as the feature points. Since both are extracted as feature points, line segments that do not generate a closed loop are extracted as feature points at the end points, branch points, and sharp bent points of the line segments. Not only is it used for tracking, but it is a boundary line of a closed-loop figure, and there are no end points, no branch points, and no sharp corners, so it is not possible to extract feature points. In such a case, if features such as the center of the basic figure, the orientation of the basic figure, etc. are combined into a feature point of the closed loop figure, a large number of feature points are extracted from the image information that contains the closed loop figure. It is possible to use the measurement and tracking of the three-dimensional coordinates.

  In addition, if a line segment is connected to a closed loop while tracking, such as an end point, is detected as a feature point, the feature point (centroid or orientation information, etc.) of the basic figure of the closed loop is used. It is possible to continue tracking by switching. Again, if the closed loop is broken halfway, an end point appears, so that it is possible to switch to tracking with the end point as a feature point and perform continuous tracking.

  A ninth means is that in the visual information processing apparatus configured by selectively applying the first to fourth means, the photographing position of the camera and the image information are recorded in association with the CAD data. .

  In the ninth means, in the visual information processing apparatus configured by selectively applying the first to fourth means, the shooting position of the camera and the image information are recorded in association with the CAD data. CAD data generated so far using the color information (texture information) of the object to be photographed even when the number of line segment data to be extracted is small or a lot of surface data defined by the line segment data constituting the closed loop is not generated. Can be displayed and used easily. Specifically, when CAD data of line segment data is displayed on the monitor screen, it is sufficient to display the texture information (specifically, two-dimensional image information) obtained by measuring the line segment on the background. Even if such information is not converted into CAD data, the operator who sees the monitor screen can easily understand which line on the image the line segment of the CAD data corresponds to. In addition, the three-dimensional position information of the CAD data and the color information of the image information can be combined and used on the program for controlling the robot and editing the inspection result data. Since the image information that is a texture is recorded in association with the position of the imaging device, there is no sense of incongruity when the image data is converted and displayed at the viewpoint position when displayed on the monitor screen. The background can be displayed. If image data corresponding to each line segment data is assigned, texture (image information) taken at a position closest to the viewpoint position on the monitor screen can be used for display. Basically, image information taken from the normal direction to the surface is recorded, but if the surface has not yet been generated, the image is taken from the normal direction to the estimated surface that can be estimated from multiple line segments. Try to record information. In any case, not all image information taken from the normal direction can be recorded, so while the imaging device moves repeatedly, image information closer to the normal direction and images taken from closer When the information is obtained, the old image information may be deleted, and the texture information may be updated with new image information as needed.

  A tenth means is a visual information processing apparatus and visual information processing system configured by selectively applying the first to ninth means, wherein the image information processing apparatus is a position or orientation of a moving imaging device or its position It is to have a function for obtaining both pieces of information and a function for outputting the obtained position information and / or posture information.

  As such a tenth means, the visual information processing apparatus configured by selectively applying the first to ninth and the visual information processing apparatus having CAD data in advance or those visual information processing apparatuses are applied. In the visual information processing system, the image information processing apparatus has a function for obtaining information on the position and / or orientation of the moving imaging device and a function for outputting the obtained position information and / or orientation information. The position and orientation information of the apparatus can be used on the application system side such as another CPU. When applied as a visual guidance control device for an automatic machine such as a robot, the relative attachment position relationship between the imaging device and the robot hand is known by attaching the imaging device to the tip of a robot hand or the like accessing the target object. Therefore, if the position and orientation of the imaging device are known, the position and orientation of the robot hand can be known, and the robot hand can be guided and controlled to an arbitrary target point in the movement space of the generated CAD data.

  The eleventh means is a visual information processing apparatus configured by selectively applying the first to tenth means and a visual information processing apparatus having CAD data in advance or those visual information processing apparatuses. In the visual information processing system, the image information processing apparatus has a function of searching for CAD data based on a search command signal for CAD data and outputting information of the searched CAD data based on the command signal. That is.

  Such an eleventh means includes a visual information processing apparatus configured by selectively applying the first to tenth means, a visual information processing apparatus having CAD data in advance, or a visual information processing apparatus thereof. In the visual information processing system to which is applied, the image information processing apparatus has a function of searching for CAD data based on the search signal of CAD data, and a function of outputting information of the searched CAD data. Therefore, information required on the application system side such as another CPU, for example, CAD data of the target object to be searched is searched, and the position and orientation information is searched as the CAD data information of the searched target object. For example, on the application system side, for example, the position and appearance of the robot hand holding the target object It is possible to induce controlled. Furthermore, if the CAD data of the robot hand is periodically searched, and the position and orientation of the searched robot hand CAD data is obtained as a search result, the information is fed back to the position and orientation of the robot hand. It can also be used for control, and it is possible to perform guidance control that corrects the deflection of the arm portion of the robot hand or the positional deviation of the base of the robot hand. In addition, since the CAD data of the target object to be searched is periodically searched, even when the target object is moving, the position and orientation information of the target object can be obtained in real time. It is possible to guide and control the movement of the target object to be moved by accessing the robot hand.

  A twelfth means is that the visual information processing apparatus having the eleventh means is provided with an input interface for command information for designating a range for generating the three-dimensional CAD data.

  In the twelfth means, in the visual information processing apparatus having the eleventh means, a command information input interface for designating a range for generating the three-dimensional CAD data is provided, so that command information for designating the range is input from the outside. By doing so, the range in which the three-dimensional CAD data is easily generated can be limited to a necessary range. Therefore, the processing speed can be increased. Although the processing speed depends on the amount of feature points (line segments) to be extracted, it takes the longest processing time when the processing range is the entire input image. On the other hand, if the processing range is limited, for example, if only the periphery of a red moving object is designated as the processing range, the processing range can be made smaller than the processing range of the entire image, so that the processing cycle can be increased accordingly. If the processing is speeded up, it becomes easier to follow even if the moving speed of the imaging apparatus increases. A plurality of commands for specifying the range may be prepared. For example, a part that has changed due to a difference in 2D image information is extracted and only its periphery is set as a processing target range, or it is moving on 2D image information, or specifically moving fast, Specify slowly moving objects together with the moving speed, and set only the periphery of the detected object as the processing range, or if the object that shines by reflection on the 2D image information is detected, set the periphery as the processing range, The shape, color, and size are specifically specified, and the corresponding object is detected on the two-dimensional image information, or the periphery is processed by detecting it on the three-dimensional CAD data converted into the three-dimensional CAD data. For example, when a rough direction such as sound can be detected with a directional microphone, a command is prepared so that the processing range can be directly specified by coordinates and the specified coordinate area can be processed. The There.

  The thirteenth means is a visual information processing apparatus configured by selectively applying the eleventh and twelfth means, and the CAD data search command information includes at least two types of two-dimensional search commands and three-dimensional search commands. The search command is provided.

  In the thirteenth means, in the visual information processing apparatus constructed by selectively applying the eleventh and twelfth means, a two-dimensional search having a high processing speed even if the matching accuracy is insufficient for the CAD data search command information. Since at least two types of search commands, which require a command and processing time but have high matching accuracy, are provided, search candidates are first narrowed down at a high speed with a two-dimensional search command, and then a three-dimensional search is performed. Since it becomes easier to narrow down the search candidates by matching with a command with higher accuracy, the search process can be performed at a higher speed.

  According to a fourteenth aspect, the search function of the above-mentioned first to thirteenth means is such that the similarity of a vector to another feature point subtracted from the corresponding point candidate of the two-dimensional or three-dimensional CAD data to be searched or the similarity of the vector And a method of using the similarity of the color information of the feature point at the tip of the vector and combining a method having a high similarity as a corresponding point.

  In such a fourteenth means, with the search function of the above-mentioned first to thirteenth means, the degree of similarity or vector of a vector up to another feature point subtracted from the corresponding point candidate of the two-dimensional or three-dimensional CAD data to be searched for The degree of similarity is high using the degree of similarity (the number of vectors of the same length in the same direction) and the degree of similarity of the color information of the feature point at the tip of the vector (the degree of coincidence of the color information of the top, bottom, left, right, up, down, left, right Since this method is combined with a method of narrowing down a corresponding point as a corresponding point, matching processing using two-dimensional or three-dimensional CAD data can be performed at high speed. Here, in the case of matching with a conventional line drawing, there is means for performing at high speed using a potential matching method or the like, but a long time is required for one matching calculation. The vector matching here is simpler than the calculation amount of other matching methods because the vector similarity can be obtained by comparing two numerical values of the direction and the distance. In addition to the direction and distance of the vector, the matching accuracy can be further improved by comparing the similarity of the color information of the feature point at the tip of the vector. In addition, only the color information around one feature point, and even if the number of vectors drawn from one corresponding point candidate is several tens to several hundreds, the amount of calculation can be reduced. It will be possible to do this faster and easier.

  A fifteenth means is a visual information processing apparatus configured by selectively applying the above-mentioned means 11 to 14 or a system to which the visual information processing apparatus is applied, and at least a combination condition between the part shape of at least three-dimensional CAD data and each part Is provided as a database, and searching is performed by generating search CAD data by changing the component coupling condition according to a predetermined rule of the component coupling condition from the information of the database at the time of the search.

  In the fifteenth means, in the visual information processing apparatus configured by selectively applying the above-mentioned means 11 to 14 or a system to which the visual information processing apparatus is applied, at least the part shape of the three-dimensional CAD data is combined with each part in advance. Since the conditions are provided as a database and the search is performed by generating search CAD data by changing the component coupling conditions according to a predetermined rule of the component coupling conditions from the information of the database at the time of searching, human limbs and faces Parts that can be moved and how they move even when the shape of the object being searched for changes depending on the open / close state of the car door, the state where the antenna of the mobile phone is put in and out, the operating state of the backhoe of the power shovel, etc. Change all the combination states of and match the shape shown in the current image information Since it is management, also it can be easily realized search objects can take a variety of shapes. Here, if all moving parts of a moving part are combined with a small amount of moving change, the number of combinations increases. It is possible to further narrow down the candidates by changing the portion by a minute amount to further increase the matching accuracy. There is a method to prepare a reference model of the movable state of all parts in advance, but if you try to prepare many types by changing the movable state finely, the amount of data will increase, but by generating it at the time of matching The amount of data in the database can be reduced, and matching can be performed using many detailed movable reference models. It is also possible to specify the state of the movable part of the search target object by searching for a movable state in which the matching degree is the highest by performing the matching process while changing the movable state of the parts in various ways. For example, if the door is opened or closed, or if it is a human face, the opening of the mouth can be obtained as a result of the search process.

  The sixteenth means is a visual information processing apparatus configured by selectively applying the above-mentioned eleventh to fifteenth means or a system to which the visual information processing apparatus is applied, and a command for using the judgment function for the searched object and the judgment function An interface for inputting information and an interface for outputting information of determination results are provided.

  In such a sixteenth means, in the visual information processing apparatus configured by selectively applying the above first to fifteenth means or a system to which the visual information processing apparatus is applied, the judgment function for the searched object and its judgment function are used. An interface for inputting command information for output and an interface for outputting information of determination results, so that a search command is issued, the search result is acquired, the search result is determined, the search command is issued again, and the search is performed again. Since iterative processing can be performed in the visual information processing apparatus, search processing that matches the purpose can be performed at higher speed.

  According to a seventeenth aspect, in the mobile inspection device, a visual information processing device having a function of processing at least image information from the image pickup device and the image pickup device to generate three-dimensional CAD data and obtaining the position of the image pickup device, or three-dimensional in advance. A visual information processing device having a function of obtaining the position of the imaging device in the three-dimensional CAD data while moving with the CAD data is provided, and specified from the position information of the imaging device in the three-dimensional CAD data This means that the inspection result data is recorded in association with the part of the three-dimensional CAD data corresponding to the inspection target part.

  In the seventeenth means, in the mobile inspection device, a visual information processing device having a function of processing at least the image information from the image pickup device and the image pickup device to generate three-dimensional CAD data and obtaining the position of the image pickup device or 3 in advance. A visual information processing device having a function of obtaining the position of the imaging device in the three-dimensional CAD data while moving with the three-dimensional CAD data is provided, and specified from the position information of the imaging device in the three-dimensional CAD data Since the data of the test result is recorded in association with the part of the three-dimensional CAD data corresponding to the test target part, the test result of the test target part to be confirmed can be easily referred to when checking the test result later. As can be done, it is possible to eliminate the manual work of associating the inspection result data with the inspection target part. Further, when the same place is repeatedly inspected, it is possible to compare changes in the inspection results at the same place and to easily check the trend of the change. Time data may be recorded together with the inspection result.

  According to an eighteenth means, in the mobile inspection apparatus using the seventeenth means, a function for setting a planned movement path of the movement inspection apparatus in advance, and an error between the predetermined planned movement path and the actual movement path are set in three-dimensional CAD. A function to obtain from the position information of the imaging device in the data, a function to set a rule for performing movement control of the movement inspection device so as to reduce the error, and a function of the movement inspection device based on the rule of the movement control That is, a control unit for executing movement control is provided.

  In the eighteenth means, in the movement inspection apparatus using the seventeenth means, a function for setting a planned movement path of the movement inspection apparatus in advance, and an error between the preset planned movement path and the actual movement path are three-dimensionally calculated. A function that is obtained from position information of the imaging device in CAD data, a function that sets a rule for performing movement control of the movement inspection apparatus so as to reduce the error, and a movement inspection apparatus based on the rule of the movement control Therefore, it is possible to automatically inspect a necessary range without operating the movement inspection apparatus by the operator. Here, if it is possible to easily set the rules for performing movement control of the movement inspection device so as to reduce the error between the planned movement path and the actual movement path, the control performance can be easily improved. Will be able to.

  The nineteenth means applies a visual information processing device that generates image data of an object existing in a moving space by inputting image information of an imaging device and processing an input image in a robot or a robot control device. Means for generating three-dimensional CAD data of the environment in which the robot operates and specifying the current position of the robot in the CAD data; and means for specifying the current posture and operating state of the robot from various sensor signals from the robot; , 3D CAD data of the environment in which the robot operates, means for reflecting the current position of the robot in the CAD data, the current posture and operating state of the robot in the simulation model of the robot, and simulating the robot operation And means for inputting an operation command signal of the robot based on the movement plan to the simulation means A means for determining whether or not the simulation result has reached a predetermined evaluation level; and a means for correcting an operation command signal to the robot based on the simulation result when the simulation result does not reach the predetermined evaluation level. An operation command signal to the robot having the best evaluation level under a predetermined condition is output to the robot.

  Such a nineteenth means is a visual information processing device for generating CAD data of an object existing in a moving space by inputting image information of an imaging device and processing input image information in a robot or a robot control device. Is used to generate 3D CAD data of the environment in which the robot operates, to identify the current position of the robot in the CAD data, and to determine the current posture and operating state of the robot from various sensor signals from the robot. Means for reflecting the three-dimensional CAD data of the environment in which the robot operates, the current position of the robot in the CAD data, the current posture of the robot, and the operating state in the simulation model of the robot; Means for simulating motion, and motion command signal of robot based on movement plan in the simulation means Means for inputting, means for determining whether or not the simulation result has reached a predetermined evaluation level, and means for correcting an operation command signal to the robot based on the simulation result when the simulation result does not reach the predetermined evaluation level Is set so that the operation command signal to the robot with the best evaluation level under the predetermined condition is output to the robot, and this processing is performed in real time at high speed, so that the robot jumps and jumps. Even in dynamic movement control, you can simulate how the posture changes if you swing your arm or twist the torso while the robot is flying, depending on the posture you plan before landing Simulation of the optimal operation to go (simulation here is a thing (It means dynamic simulation to find the force, reaction force, and speed, acceleration, and displacement), so that it can output a motion command signal that can control the robot as planned. This makes it possible to control the robot dynamically and stably. By applying at least the visual information processing apparatus or the visual information processing system as described in the first to eighteenth means to the control apparatus for the robot that performs such processing, there is currently a robot used for the simulation performed here. As the 3D CAD data of the environment, as-built CAD data is generated in real time, and the current position and posture information of the robot in it can be obtained in real time, so the simulation results are based on the actual environment of the current robot Appropriate control data can be obtained and real-time control can be realized stably.

  The twentieth means includes a manipulator, a mobile robot, a character reader, an object recognition device, a measurement device, a CAD data generation device, a navigation device, an inspection device, a picking device, an autonomous control robot, a monitoring device, an automobile, a ship, an airplane, etc. In other words, the visual information processing apparatus configured by selectively applying the first to nineteenth means or the visual information processing system using the visual information processing system is applied.

  Such a twentieth means is, in a robot or manipulator for positioning a specific part, a visual information processing apparatus configured by selectively applying the first to nineteenth means to the control device, or the same. Since the visual information processing system is applied, for example, a remote operation robot can easily construct a system for automatically positioning the hand. Specifically, for example, if the imaging device is attached to the hand and the position and orientation information of the imaging device is obtained, the position of the hand can be corrected and controlled to a target position based on the information. Further, if the target object to be accessed by the hand is searched from the CAD data and the position and orientation information of the target object is obtained, the position and orientation of both the imaging device and the target object can be known, so that more accurate It is possible to perform guidance control so that the hand is accessed to the target object. In addition, when an image pickup device cannot be attached to the hand, a separate image pickup device for photographing the hand is provided, and the CAD data of the hand is searched using the image information of the image pickup device to directly determine the position and posture of the hand. It can also be detected and used for control.

  Moreover, in the mobile robot, since the visual information processing apparatus configured by selectively applying the first to nineteenth means or the visual information processing system using the same is applied, the positioning of the mobile robot itself, Attitude control can be easily performed. For example, if an imaging device is mounted on a mobile robot, the position and orientation information of the imaging device indicates information on the position and orientation of the mobile robot, so that the position and orientation of the mobile robot can be controlled based on that information. Thus, positioning control, posture control, guidance control, etc. of the mobile robot can be easily performed. If an imaging device is mounted on a ROV that floats in water and this device is applied, for example, the CAD of the surrounding environment is located even if the ROV is located deep in the water and the ROV in the water cannot be seen directly from above the water surface. If the data and the position and orientation of the imaging device are displayed on the display device of the console, the remote operation of the underwater ROV can be easily performed while confirming the position and orientation with the display device. It becomes like this. In addition, if the final target position and posture are input so as to be positioned in front of an arbitrary place on the CAD data, and the underwater ROV is automatically controlled to be the target position and posture, automatic driving can be performed. A control system can also be easily constructed. Of course, an imaging device can be mounted on an airship or an airplane and applied to remote operation and automatic control of a flying robot.

  In the character reading device, a visual information processing device configured by selectively applying the first to nineteenth means or a visual information processing system using the same is applied. It is possible to easily construct a system that makes it easy to recognize predetermined characters by matching processing of three-dimensional CAD data even for characters that are difficult to read by a two-dimensional image processing. Also, since characters that change in time series can be generated in time series, the movement pattern is known in advance, for example, characters that are displayed to flow on an electric bulletin board or signs that are swaying in the wind. It is possible to construct a system that can easily read characters that do not move.

  In the three-dimensional object recognition device, measurement device, and CAD data generation device, a visual information processing device configured by selectively applying the first to nineteenth means or a visual information processing system using the same is applied. Therefore, it is possible to easily realize a three-dimensional object recognition device, a measurement device, a CAD data generation device, and the like necessary for field surveys of plants and the like, car design design, CG characters, and the like. That is, even if a general-purpose imaging device is used to generate a CAD data by capturing an image of the imaging device close to the target object without using a system using an imaging device with high-resolution resolution, Since CAD data with high accuracy can be generated, it is possible to generate CAD data with high accuracy based on image information obtained by photographing with a general-purpose imaging apparatus, and to refer to the CAD data. For example, it is possible to easily construct a highly accurate 3D measurement system using a general-purpose imaging device. When the generated or updated CAD data is taken out, it is converted into a predetermined CAD data format, for example, a DXF format. Can be easily used in the CAD system. If the imaging device is mounted on a traveling vehicle traveling on the ground and a flying object flying in the sky, a more accurate three-dimensional map of the town can be easily created. That is, the visual information processing device of a traveling vehicle traveling on the ground generates detailed CAD data of a range where a road or a house is photographed from below, but on the roof of the house or on a tree growing on the ground. If you are looking up from below, you cannot get enough image information, so it is not possible to generate enough CAD data at the top, but you can use the image information taken from the flying object in the sky. By integrating with the generated CAD data, it becomes possible to easily create an accurate three-dimensional map of the townscape with no generation omission from the bottom to the top.

  Further, in the navigation device, a visual information processing device configured by applying the first to nineteenth means or a visual information processing system to which the processing device is applied is applied. It can be realized. That is, by connecting an imaging device to a moving navigation device and generating and updating CAD data of the surrounding environment by the imaging device, the position and orientation information of the imaging device is extracted in real time, and the environment CAD data and the imaging By displaying the position and orientation information of the apparatus, the system can easily confirm the current position. The environmental CAD data may be given in advance to the navigation device as a certain amount of CAD data. For example, only the two-dimensional map information is provided, and the three-dimensional CAD data generated as the vehicle travels. A two-dimensional map and three-dimensional CAD data may be associated from a road or a building near an intersection. The CAD data to be created in advance may be input with a characteristic building or the like off-line, or the CAD data of the townscape generated in advance by an automobile or flying object is generated and synthesized by the visual information processing apparatus of the present invention. It is also possible to use the one that has been used. Further, the generated as-built CAD data may be exchanged with other navigation devices that are moving and generating CAD data, or generated by a plurality of navigation devices via a central information processing center or the like. CAD data may be exchanged. The attachment relationship between the imaging device and the moving body is known, and by setting the relationship in advance, the position and orientation of the moving body are displayed instead of displaying the position and orientation of the imaging device. You can also. Also, if the destination in the environmental CAD data is input, what route should be taken to move to the destination from the destination in the CAD data and the current position of the moving body, etc. Navigation display may be configured and executed in the same manner as a GPS-equipped navigation device.

  In addition, since the inspection apparatus applies a visual information processing apparatus configured by selectively applying the first to nineteenth means or a visual information processing system to which the visual information processing apparatus is applied, the three-dimensional shape can be easily obtained. An inspection device can be obtained. That is, for example, if an imaging device is provided at the tip of a manipulator and the like and the CAD data to be inspected is generated by imaging the region to be inspected from various angles with this imaging device, Since the three-dimensional uneven state is accurately reproduced, for example, if it is possible to extract the position and orientation information with a predetermined part such as whether or not the predetermined part is properly attached at a predetermined place, it is easy An automatic inspection device can be constructed. If the imaging device is a microscope, the generated three-dimensional CAD data becomes a fine three-dimensional uneven surface. For example, a fine image that can only be seen by changing the angle of a CD or DVD. It is possible to realize an inspection apparatus that can easily and efficiently inspect even scratches.

  In the picking apparatus or system, a visual information processing apparatus configured by selectively applying the first to nineteenth means or a visual information processing system using the same is applied. By preparing at least one unit, it is possible to easily construct a picking system. That is, by attaching an ordinary general-purpose imaging device to the tip of the manipulator to be handled and generating work CAD data based on image information from the imaging device, by first photographing from around the workpiece, the workpiece is built as-is. CAD data can be generated. Here, the image pickup apparatus may be configured to obtain the same image information as when the image pickup apparatus is moved around the workpiece by fixing a plurality of image information signals around the workpiece and switching the image information signal. After the as-built CAD data of the workpiece has been generated, the shape of one part in the workpiece is registered in advance as CAD data for search, and from the CAD data of the workpiece in which many generated parts are stacked. The position and orientation information of the target part to be grasped by the control device of the manipulator is obtained so that the position and orientation information of each part is obtained by the search function and picked by the manipulator for handling in order from the outermost part. If the manipulator is controlled by giving it, a picking system can be easily constructed. In particular, when an imaging device is attached to the tip of a manipulator, the imaging device can accurately generate CAD data by photographing the workpiece by bringing the imaging device close to the workpiece. It is possible to generate CAD data with high accuracy even when shooting with a general-purpose general imaging apparatus instead of the apparatus. Even if the shape of the workpiece stacked with the components removed by picking is changed, the CAD data of the workpiece is updated, so that the picking control can always be handled according to the state at that time.

  Moreover, in the autonomous control type robot, the visual information processing apparatus configured by applying the first to nineteenth means or the visual information processing system applied thereto is applied, so that the autonomous control type robot can be easily constructed. Is possible. That is, if an imaging device is attached to a robot, it is possible to easily generate environmental CAD data by applying a visual information processing device, and the position and orientation information of the imaging device in the environment and the position and orientation of the robot itself. Information can also be obtained easily. Furthermore, since it is possible to search for specific CAD data in the environment, the presence / absence of CAD data prepared for searching in advance, and the position and orientation information of the CAD data, if any, can be obtained. In the case of a target object, the robot itself can be controlled in any way with respect to the work target object by the position and posture of the work target object and the position and posture of itself. For example, when trying to execute a work content to be found and returned by searching for a specific work target object, the robot uses the navigation function to move the robot along the route planned by the navigation, and then the work target object. When the work target object is found, the robot itself is approached at a predetermined position and posture, the work target object is held by the robot, and the navigation function is again performed. Control to return to the starting point using can be easily realized. If there is an obstacle on the way, the CAD data of the environment is generated and updated at any time, so it is easy to recognize this obstacle as an obstacle. For example, if you prepare a program that searches for an avoidance route while avoiding obstacles, you can autonomously execute the target work as planned even if there is an obstacle on the way. Become. Here, what is called an autonomous control type robot may be any of an automobile, an automatic guided vehicle, a manipulator, and a humanoid robot, but is a robot having a function of controlling the operation by the robot itself. The autonomous control type robot may basically be programmed to follow the operator's command. For example, in the case of a pet robot, etc. It may be programmed to prioritize charging and not necessarily follow the command. Also, when applied to watchdog robots, it is easy to build a system that recognizes suspicious clothes and suspicious movements of surrounding people and programs suspicious people to follow them. Can be realized.

  In addition, since the monitoring device applies a visual information processing device configured by applying the first to nineteenth means or a visual information processing system to which the visual information processing device is applied, the monitoring target is recognized and each monitoring target is recognized. Necessary monitoring can be easily performed. For example, if you register the baby shape in advance, you can search for registered babies, detect the baby's position, and easily control the shooting of the baby while tracking it even if the baby moves You can also register your baby's crying facial expression, laughing facial expression, etc., and by detecting which facial expression the current facial expression is close to, the situation such as whether the baby is laughing or crying Since it can be automatically recognized, if it is set to notify the parent with an alarm if it crys in advance, a monitoring system that automatically notifies the parent when the baby cry can be easily constructed. In this example, the monitoring of the baby's facial expression has been described as an example. However, by searching and matching the shape with CAD data, such as the natural state, device operating state, plant state, and suspicious person intrusion monitoring, the voice and temperature Various monitoring systems can be easily realized by combining with other sensor information such as sensors.

  In systems such as automobiles, ships and airplanes, the visual information processing apparatus configured by applying the first to nineteenth means or the visual information processing system using the same is applied. As a result, it is possible to generate environmental as-built CAD data, and the current position and current posture of the imaging device are also calculated during 3D measurement and tracking processing when generating as-built CAD data of the surrounding environment. Since the position and direction of a moving vehicle such as an automobile, ship, or airplane with an imaging device attached is detected, if the target travel route has been set in advance, the current position will be deviated from the target travel route. It can automatically control to calculate and correct the position and direction of moving vehicles such as cars, boats, airplanes, etc. to match the target moving route. It is also possible to detect nearby objects by registering in advance the surroundings that move at this time, people and cars ahead, boats, airplanes, and other objects in advance. If a control program is prepared in advance according to the object, such as bypass control, stopping, or proceeding as it is, the purpose is to correct and change the appropriate movement path according to the detected object. It is also possible to easily control driving of moving vehicles such as automobiles, ships, and airplanes so as to match the travel route.

  In a monitoring service providing system that centrally monitors information sent from one or a plurality of terminals at the center, the twenty-first means searches for a user by a request from the user, tracks the user during monitoring, And continuously monitoring the user's surroundings.

  In such a 21st means, in a monitoring service providing system that centrally monitors information sent from one or a plurality of terminals at the center, the user is searched by a request from the user, and the user is tracked during monitoring. Since the user and the user's surroundings are continuously monitored, a monitoring service system that guarantees the safety of the user can be easily constructed. Here, tracking may be performed by a manual device, or automatic tracking may be performed by image processing or the like. In addition, a user of an autonomous control type robot can use this service for robot monitoring. The image may be recorded during monitoring so that it can be referenced later. In addition, when a person moves around a place where a surveillance camera or the like is not installed or a place where the number of installations is small, it is possible to attach a portable small camera so that the person can receive this service. Further, if a visual information processing device configured by selectively applying the first to nineteenth means to a processing device of a monitoring service providing system or a visual information processing system using the visual information processing device is used, it is based on three-dimensional CAD data. Automatic tracking of users. In addition, since the surrounding situation can be viewed with the three-dimensional CAD data, the field situation can be confirmed more accurately.

  The twenty-second means is a monitoring service providing system that centrally monitors information sent from one or a plurality of terminals, and records predetermined information for a predetermined period in each terminal in association with time. This is a function that allows recording data at a predetermined time of a predetermined terminal to be taken out by a request from a user.

  In such a twenty-second means, in the monitoring service providing system for centrally monitoring information sent from one or a plurality of terminals, each terminal associates predetermined information for a predetermined period with time. Since a recording function is provided so that recorded data of a predetermined terminal at a predetermined time can be extracted by a request from the user, a monitoring service is provided in addition to the monitoring service when a monitoring service request is requested from the user. Even if an incident occurs when it has not been requested, since past monitoring record data can be easily extracted and used for incident investigation, users who are involved in the incident without requesting a monitoring service Will also be able to provide services.

  A twenty-third means is a monitoring service providing system configured by selectively applying the twenty-first and twenty-second means, and a state expression means for notifying a person around the user that the monitoring service is in progress is provided during monitoring. That is.

  In the twenty-third means, a monitoring service providing system configured by centrally monitoring information sent from one or a plurality of terminals at the center and selectively applying the second and second means, Since a state expression means for notifying a person around the user that the monitoring service is being provided is provided, the suspicious person can be restrained before being attacked by the suspicious person.

  In a monitoring service providing system configured by selectively applying the twenty-third to twenty-third means, the twenty-fourth means is configured to notify the user or the user and the surrounding people when a situation to inform the user occurs during monitoring. A means for providing the information is provided.

  In the twenty-fourth means, in the monitoring service providing system configured by selectively applying the twenty-second to twenty-third means, when a situation to inform the user during the monitoring occurs, the user or the user and the surrounding people Since the means for providing the information is provided to the user or the person in the vicinity thereof before being attacked by the suspicious person, the incident can be prevented in advance.

  According to a twenty-fifth means, in a terminal device for acquiring sensor information, sensor installation ports are provided in advance at a plurality of locations, and when assembling the terminal, one set or a plurality of sets of sensors are used as installation ports at predetermined locations. It is a configuration that can be connected.

  In the twenty-fifth means, in a terminal device for acquiring sensor information, sensor installation ports are provided in advance at a plurality of locations, and when assembling the terminal, one set or a plurality of sets of sensors are installed at predetermined locations. Since the sensor can be incorporated only in the necessary direction according to the place where the sensor is installed, the necessary terminal device can be assembled with the minimum necessary sensor.

  According to the present invention, as-built 3D-CAD data can be automatically generated in real time, combined with a CAD data search function, as a visual guidance control device for various automatic machines such as robots, navigation systems, and mobile inspections It is possible to provide a visual information processing apparatus and its system that can be applied as a device, a monitoring service system, and the like.

The present invention relates to an image information processing apparatus that inputs image information obtained by photographing an object with a movable imaging apparatus and generates CAD data of the object, and a vision device that includes a storage device that stores the generated CAD data. In the information processing apparatus, the image information processing apparatus includes an update target determination processing function for determining whether the generated new CAD data is an update target of the already generated CAD data, and a ready CAD that is an update target. CAD data accuracy comparison processing function that compares the accuracy of data and new CAD data, and CAD data that updates new CAD data with new CAD data when the accuracy of new CAD data is higher than the accuracy of existing CAD data to be updated An update processing function is provided,
The function to generate new-generation CAD data is a stable measurement among the points measured in stereo by associating the feature points in the image information of a plurality of imaging devices captured at the same time from the feature parts of the image information. The method is realized by executing a method of obtaining three-dimensional coordinate data so that corresponding points in the image information by movement of the imaging device are related by tracking and tracking at least three points.

  FIG. 1 is a block diagram showing the basic configuration of the first means in the visual information processing apparatus of the present invention.

  The image information processing apparatus 100 generates CAD data by inputting image information obtained by the imaging apparatus 10 directly or indirectly via a recording medium, and stores the generated CAD data in the storage device 200.

  The imaging device 10 is a device such as a moving TV camera, and shoots every object to be converted into CAD data from various angles to generate image information.

  The image information processing apparatus 100 directly or indirectly inputs image information generated by imaging by the imaging apparatus 10 to generate 3D CDA data of any object included in the image information, and stores it in the storage device 200. Remember. In addition, the image information processing apparatus 100 reads the ready-made CAD data stored in the storage device 200 as needed, compares the newly-generated new-CAD data with accuracy, and the accuracy of the new-generation CAD data is the ready-made CAD data. When it is higher than the data, the existing CAD data stored in the storage device 200 is updated with the new CAD data. Since the imaging apparatus 10 can freely move around in the environment, the imaging apparatus 10 can approach or leave an object in the environment. Since the object can be photographed with high accuracy when the imaging device 10 is close to the target object, for example, a portion that was seen as a single line at a distance is closer, or two appear as the viewing angle changes. Therefore, the image information processing apparatus 100 has a CAD data update function for updating the previously generated CAD data so as to be more detailed and so as to correct a part that was previously erroneously recognized and generated. Have.

  As a result, more accurate CAD data can be constructed as the imaging apparatus 10 is moved in various ways to photograph an object in detail. In addition, since the imaging apparatus 10 can move around in various positions, image information can be generated by photographing the back surface, the bottom surface, and the top surface of the object, so that accurate as-built CAD data can be easily generated. Will be able to.

  FIG. 2 is a block diagram showing another example of the basic configuration of the first means in the visual information processing apparatus of the present invention.

  The image information processing apparatus 100 inputs two pieces of image information obtained by photographing with the first imaging device 10 and the second imaging device 20 directly or indirectly through a recording medium. The first imaging device 10 and the second imaging device 20 are fixed at a predetermined interval so that stereo measurement can be performed. With this configuration, the image information processing apparatus 100 can generate CAD data while measuring distance information to an object by stereo measurement using two pieces of image information input from the two imaging apparatuses 10 and 20. It becomes possible. The other configuration is the same as the configuration of the visual information processing apparatus shown in FIG.

  Next, the internal configuration of the image information processing apparatus 100 will be described. The image information processing apparatus 100 basically includes an arithmetic processing device and an information (arithmetic) processing program. Here, the first means is a case of using a monocular imaging device 10 as shown in FIG. 1, and the configuration stage of another form of the first means is a compound eye as shown in FIG. This is a case of using the (stereo) imaging devices 10 and 20, but the basic information processing flow of the image information processing device 100 is the same. After describing the processing flow, the processing in the case of compound eyes (stereo) will be described with reference to FIGS. Processing in the case of a single eye is processing that omits processing that can only be performed with a compound eye (stereo).

  In the basic processing flow in the case of a monocular, when processing is started, image information obtained by the imaging device 10 is taken in, processing such as differentiation processing and noise removal is performed, and a line segment is extracted. By tracking the start point and end point of a minute by tracking, etc., it is possible to obtain clear image information of corresponding points like the start point and end point between multiple pieces of image information taken from different viewpoints, so photo measurement The three-dimensional coordinates of those points are calculated by the method. Then, a line segment connecting the three-dimensional start point and the end point is defined as three-dimensional CAD data, and by defining a plurality of line segments in the same manner, line segments of various parts of the object in the space are defined. CAD data is generated. Here, if the imaging device 10 is further moved closer to the target object, the imaging device 10 can capture a larger image of the target object, so that the resolution, that is, the accuracy is improved. Then, the CAD data generation process is repeated, and if the line that was previously seen as one appears as two lines (CAD data update target discrimination function and CAD data accuracy processing function), the ready-made CAD data The one line segment in is updated to two line segments (CAD data update processing function).

  With such a function, as the imaging apparatus 10 is moved and the target object is observed (photographed) in more detail, it can be replaced with more accurate CAD data.

  By doing this, one end point of the line segment disappears in the middle of tracking, and even if a corresponding point cannot be obtained in a predetermined number of pieces of image information, that point is still If the process of processing only points where a sufficient number of corresponding points are collected without processing and generating three-dimensional CAD data at those points is repeated, the imaging apparatus 10 continues to move. Therefore, by combining with the update processing function of CAD data, the number of points that are gradually measured increases and the CAD data also increases, and the CAD data becomes more accurate while being updated in more detail.

  Here, once the line that appeared to be two lines is converted into CAD data of two line segments, when the imaging device 10 takes a picture again from a distance away from the object, again, You may do the update process to return it to the line segment, but once it looks like two, even if it becomes one after that, two appear as one Therefore, a processing method that does not update the CAD data to one is suitable for the purpose of more accurately creating the CAD data. In this case, for example, when calculating three-dimensional coordinates, information on the accuracy of the calculation result at that time is stored together, and again, the same point around the same place (calculated earlier) It is also possible to determine whether a sphere having a predetermined radius around a certain point is the same place as a point that falls within the sphere, and the assumed radius depends on the accuracy of the previously calculated point. If the accuracy is good, it may be set to be small, and if the accuracy is poor, it may be set to a large value.) When required, the accuracy obtained this time is compared with the accuracy obtained last time, and the higher accuracy is left. It is better to use a processing method.

  The concept of updating CAD data will be described in detail separately with reference to the drawings. Here, by providing a processing function to update, it is possible to generate only the points that can be generated at that time without applying a means to completely solve the problem of reliably tracking corresponding points, which has been considered difficult in the past. By generating only line segments that can be generated and replacing them when they are generated more accurately, that is, by updating, it is sufficient to obtain sufficient CAD data of the required accuracy for the imaging apparatus 10. By moving to, it becomes possible to obtain CAD data with an accuracy sufficient to meet the purpose of use.

  Next, processing in the case of compound eyes (stereo) will be described with reference to FIGS. In the case of compound eyes (stereo), an embodiment will be described in which three-dimensional measurement using stereo and the imaging devices 10 and 20 are moved and combined with photo measurement in the same manner as in the case of a single eye. Means for measuring three-dimensional (3D) coordinates, means for tracking, what is tracked as a feature point, CAD data is generated only for three-dimensional data, or two-dimensional image information based on two-dimensional Various methods of generating CAD data, such as using CAD data together, can be considered. Therefore, it is possible to apply a method suitable for the purpose, such as reducing processing speed or memory capacity. As described above, there are various ways of applying the present invention, but here, it will be described as an embodiment.

  FIG. 3 is a flowchart showing a basic flow of processing in the image information processing apparatus 100 realizing the first means.

  The start 100a may be started when the power is turned on, or may be started with a start button or the like, but is a start of processing. When the processing starts from the start 100a, first, the processing flow of the image information obtained by the imaging devices 10 and 20 is performed. The captured image information may be stored as raw image information in the storage device 200, or may be stored in an image-dedicated memory in the image information processing apparatus 100, but may be used for the next processing. It is necessary.

  Next, processing (1) 100c such as differentiation processing and noise removal of the captured image information is performed.

Here, a specific example of the processing (1) 100c will be described with reference to FIG. Since the image information is stereo, there are two images. By performing differential processing or the like on the left raw image information (raw image) 101a, line image information (line image) such as processed image information 101c is generated. This process does not have to be a differential process, and the same result can be obtained by performing a process of extracting a contour. By processing the right raw image information (raw image) 101b in the same manner, line image information (line image) such as the processed image information 101d is processed.
Convert to Here, what is important is that the start point, end point, etc. of the line segment are followed by tracking, so that the line segment extracted when the lighting state changes slightly or the viewpoints of the imaging devices 10, 20 change slightly. In order to obtain a stable processing result in which the length of the signal does not change drastically, it is necessary to consider a parameter for performing differentiation processing, a parameter for noise removal, a method, and the like. In addition, even if the end point of a line segment is determined using a threshold that may or may not be in contact with other line segments, even if the line segment is extracted in a separated state due to lighting conditions, etc. It is desirable to make corrections such as However, a line segment that cannot be seen if it is in contact with another line segment is left as it is until a visible line segment is extracted. In either case, it is preferable that the images are extracted in the same way even if the viewpoints of the imaging devices 10 and 20 are slightly changed.

  In the next process (2) 100d, the line segments are numbered, or in the case of stereo, the left and right line segments are associated.

Here, details of the process (2) 100d will be described with reference to FIG. Name and manage each line segment of the extracted line image information (line image). In the line image information (line images) 102a and 102b, outlines of rectangular parallelepipeds are extracted. Numbering is performed so that each line segment constituting the rectangular parallelepiped can be managed by giving names L1 to L9. In the case of the start point, end point, line type, and curve of the numbered line segment information on the line image information 102a and 102b, the function of the curve is stored in the internal memory or storage device 200 as a two-dimensional image database. And register. Here, a simple image of the two-dimensional image database will be described below.

これらのデータは、撮像装置１０，２０が移動して撮影シーンが変わったならば、シーン（２），（３）…と増やしていく。

These data are increased to scenes (2), (3), etc. when the imaging devices 10 and 20 move to change the shooting scene.

By installing the left and right imaging devices 10 and 20 as close to the left and right eyes of a person as possible, the left and right image information is a little shifted, but the image information is almost similar (left and right Since the correlation is very good), the line segments and points corresponding to the left and right image data are associated with each other. In the case where the target object is in front and the left and right image information are extremely different, only the points of the portion that can be determined to be sufficiently the same as the partial matching are associated. The point where the correspondence could not be taken can be obtained from photographic measurement with time-series image information when the imaging devices 10 and 20 are moved later, so that processing cannot be performed simply because 100% correspondence cannot be obtained here. is not. The obtained association relationship is also registered in the left and right two-dimensional screen databases. For example, the line names may be the same as in the above example. The line segment name is the record number of the memory. Alternatively, the left and right data may be stored in physically separate memories in the storage device 200 and registered. Here, the coordinates of the start point and end point may be integers because they are addresses on the image memory. When the line type is a straight line, the coefficient of the fitting function may be omitted. If the curve is a quadratic curve, when fitting with a function of Y = aX ² + bX + c, there are three coefficients a, b, and c. When the line type is a circle or ellipse, the necessary number of coefficients necessary for the function are registered. A point may be defined as a short line segment whose start point and end point coincide.

  In the next process (3) 100e, in the case of a compound eye capable of stereo processing, at this time, each line segment is converted into three-dimensional data to generate a three-dimensional shape model.

  Here, details of the process (3) 100e will be described with reference to FIG. Since the line segment data corresponding to the left and right are already obtained as L1 to L9 in the stage of the process (2), the coordinates of the start point and the end point of each line segment in the three-dimensional space in the compound eye stereo measurement. It can be calculated by calculating the coordinates. In this process (3), a three-dimensional database is generated based on the result of calculating the three-dimensional coordinates.

  The concept of this three-dimensional database is that the rectangular parallelepiped line segments L1 to L9 can be converted into three-dimensional line segment data, and if each line segment is connected to form a closed loop, surfaces S1, S2, and S3 are defined. It is a form that can. Further, in a state where there is nothing, the first plane (which may be considered as an infinite plane at infinity) S0 can be defined.

  Specifically, the start point, end point, line type, and line function of the three-dimensional line segments L1 to L9 are registered from the left and right associated image information. Next, the constituent line segments, surface types, and surface functions of the three-dimensional surfaces S1 to S3 forming a closed loop with the three-dimensional line segments are registered. The background is registered as the surface S0 for the time being (at first, since the background surface is not yet determined, there is no data content).

The following is a simple image of this three-dimensional database.

上記の面データは、単純な平面データのみであるので、面を構成する線分名称をデータとしてもつだけでも良い。

Since the above surface data is only simple plane data, it is also possible to have only the names of line segments constituting the surface as data.

  In the next process (4) 100f, if necessary, invisible data is generated. This is effective when CAD data is generated by considering all data as basic figures such as a cylinder, a sphere, a cone, and a rectangular parallelepiped. However, it is not necessary to do so.

  Details of the process (4) 100f will be described with reference to FIG.

  A group of faces that can be estimated to constitute a part of the basic figure is searched for from a group of three-dimensional planes generated based on the image information obtained by the first shooting. If not found, this process is passed. When it is estimated that the planes S1, S2, and S3 constitute a part of the basic figure, it is determined which basic figure is closest to the basic figure based on the coordinate data. Then, if it is determined that the planes S1, S2, and S3 are part of a rectangular parallelepiped, planes that are not visible are generated as estimated planes SS4, SS5, and SS6. At this time, line segment data of the invisible part is also generated as estimated line segments SL10, SL11, SL12. Furthermore, the configuration surface defines estimated solid data SV1, which is S1, S2, S3, SS4, SS5, SS6. These data are registered in the three-dimensional database.

  In this way, when stereo processing is used together, the CAD data before the imaging devices 10 and 20 start moving, although there is an estimation part in part, immediately moves the arm toward the target object, for example, in manipulator handling or the like. Can be used as control information (object search information).

  Here, the estimated data is managed so as to be understood as estimated data. For example, CAD data of the back side portion is generated based on image information obtained by photographing the back side portion of the object. In such a case, the estimated CAD data is replaced with non-estimated CAD data.

  As described above, the two-dimensional screen database is configured so as to perform processing of necessary portions in the processing (1) 100c, processing (2) 100d, processing (3) 100e, and processing (4) 100f in the flowchart shown in FIG. When the imaging devices 10 and 20 start to move by repeatedly registering them in the three-dimensional database and taking in the image information from the imaging devices 10 and 20, a large amount of two-dimensional image data (scenes) is accumulated. . Then, in the process 100g, the accumulation degree of the two-dimensional screen data is monitored, and the coordinates may be obtained by the photo measurement of process (5) 100h when it is accumulated to some extent.

  Details of the process (5) 100h will be described with reference to FIG. Although it may be considered as the raw image information of either of the left and right imaging devices 10 and 20, when the viewpoint of the object of the raw image information 105a changes, the appearance changes as the raw image information 105c and the raw image information 105e.

  Here, the right side is a concept of line image information based on data registered in the three-dimensional database, and cannot be seen depending on the viewpoint angle of the imaging device 10 (20) like the line image information 105b, 105d, and 105f. The estimated line segments SL10 and SL12 generated by estimation from the above are not estimated at the stage of the image information 105d and are at the estimation level, but at the stage of the image information 105f, they are visible in the actually captured image information. Therefore, since these line segments can be generated directly based on the image information, the estimated line segment SL10 can be determined as the line segment L10, and the estimated line segment SL12 can be determined as the line segment L12. it can. In addition, as the two-dimensional image data based on image information taken from different viewpoints increases, photo measurement can be applied, so the spatial coordinates of the start point and end point defining each line segment are X, Y, and Z. Is updated so as to improve the accuracy of the CAD data by rewriting the coordinate data in the portion where the coordinates obtained by the photo measurement are more accurately obtained from the coordinate values obtained by the stereo measurement. At this time, if each coordinate value of the three-dimensional database is obtained by stereo measurement, the measurement accuracy at that time is stored in the database, and if the same point is newly recalculated in stereo measurement or photo measurement Then, the accuracy of the three-dimensional coordinate value calculated under the condition at that time is obtained by the method, and the coordinate value and its accuracy data are updated when the accuracy is better than the accuracy when the previous coordinate value is calculated. Incidentally, the accuracy of the three-dimensional coordinate calculation may be better for photo measurement than stereo measurement for a distant object. As a result, when the accuracy is improved, the CAD data can be automatically updated.

The processing contents when processing (1) 100c to processing (4) 100f in FIG. 3 are repeated will be supplementarily described below.
(1) The imaging devices 10 and 20 that are initially stopped also move. Changes in the position and orientation of the imaging devices 10 and 20 may be calculated in this.
(2) Since two-dimensional screen data is added sequentially, data with little change in the middle is deleted so that the data amount does not explode (memory is insufficient).
(3) When a new line segment or point is generated in the same screen, it is newly registered in the two-dimensional drawing database.
(4) Even if the same line segment or point in the same screen is moved, it is tracked and managed as the same line segment or point. Correspondence as moving image tracking is managed in this two-dimensional screen database.
(5) At the same time, it is sequentially checked whether there is an error in the corresponding points of the left and right images, and if an error is found, the part related to it is immediately recalculated, and the two-dimensional screen database and 3 Modify the dimensional database.
(6) When a new line segment or point is newly generated, a new model is additionally generated in the three-dimensional database in processing (3) 100e and processing (4) 100f.
(7) If the two-dimensional screen data has been accumulated, the process (5) 100h is also performed to newly generate a new model in the three-dimensional database.

  As described above, when image processing is performed on the shooting scene to calculate the three-dimensional coordinates, the lenses of the imaging devices 10 and 20 practically follow the tracking by shooting a wide range using a wide-angle lens. It is desirable to be able to generate and search CAD data for a wide range of environments in a short time while reducing the chance of losing sight of the feature points that are present. Distortion may occur, and accurate 3D calculation may not be performed. Therefore, when using a wide-angle lens, consideration must be given to using a wide-angle lens with less distortion, and consideration must be given to performing 3D calculation while correcting lens distortion by image processing.

  The distortion of the image information is caused by the refraction of the lens. Since the distortion characteristic of the image information is known as the lens characteristic because of the relationship of the refractive index of the lens, the correction according to the distortion characteristic of the lens is performed on the image. By performing it at the pre-processing stage of processing, 3D calculation based on image information without distortion after correction becomes possible. In order to perform such processing at high speed, it is preferable that the processing is performed by hardware such as an LSI circuit. In addition, when using a zoom lens, the lens characteristics (correction parameters) vary depending on the zoom state, so it is desirable to use a correction circuit that takes in the zoom state and performs correction processing.

  FIG. 9 shows a concept of utilization of floating line segments generated as CAD data. In FIG. 9, for example, line segments L11, L12, and L13 are lines attached to each surface of the rectangular parallelepiped, and the coordinate data of those line segment data are coordinates included in a part of the plane of the rectangular parallelepiped. The shape of a rectangular parallelepiped is shown. Further, when a line is actually drawn, it can be used as line segment data indicating the feature of the object as pattern data.

  Therefore, floating line segments that do not constitute a plane generated when contour lines are extracted are registered and managed in the two-dimensional screen database and the three-dimensional database. Each line segment is registered and managed on which side. These floating line segments can express the shape of the surface by referring to the coordinates of the line segments in the surface when creating the surface data, and search for patterns and scratches as reference data. In this case, the data is necessary for matching. If surface data is attached to some surface, the relationship is registered and managed.

  FIG. 10 shows a concept when applied to a cylinder or a sphere. The cylinder has a line segment L7, and the sphere has a line segment L8. These are data indicating a part of the surface shape of a cylinder or a sphere as actual line segment data.

  In FIG. 10, line segments L1, L4, L5, and L7 are registered in the database as curves. The surfaces S2 and S3 are actually curved surfaces, but are unknown at this stage (may be planes). Since the positions of line segments L2, L3, and L5 change when the viewpoint changes in planes S2 and S3, it can be understood that line segments L2, L3, and L5 are boundary lines of curved surfaces. Estimate and define as a sphere, or estimate and define as a curved surface function from the coordinates of the floating line segments L7 and L8.

  FIG. 11 shows an example of the definition of the special curved surface as CAD data. For example, the leaf of a tree 900 and the like cannot be seen for each piece of image information taken from a distance. In that case, for example, it is assumed that line segments L2, L3, L4, L5, and L6 are attached to the special surface S1 formed by the contour line L1. Since each line segment has a three-dimensional coordinate value, the shape of the special curved surface S1 attached to the surface of the line segment can be considered to have a surface at a portion to which the line segment data is attached. By using such a concept of defining a special surface, it is possible to define a complex curved surface of a cloth or a kimono or a surface such as a sand pile.

  Specifically, in the leafy tree 900, the contour lines L1 and L7 can be clearly extracted, but since there are no vertices or the like, it is difficult to define the shape, so the contour lines L1 and L7 are formed. The curved surfaces S1 and S2 are defined, and floating line segments L2, L3, L4, L5, L6, L8, L9, L10, and L11 are defined on the curved surfaces S1 and S2. The leaves sway in the wind, and the position and shape of the floating line segment change or disappear depending on the light condition, but it is sufficient if the data at that moment is considered positive. The reason that such a rough expression is acceptable is that data of a distant object is not directly used for control that requires accuracy. In handling control such as gripping a leaf with a manipulator, if the hand approaches the leaf, the imaging devices 10 and 20 also approach the leaf and obtain detailed image information for each leaf. This is because the required accuracy can be obtained with corresponding details.

  Similarly, the solid data can be defined as an object made of a curved surface on which floating line segments L2, L3, L4, L5, L6, L8, L9, L10, and L11 exist. This solid data can be used when examining a tree of such a shape of this size. If possible, it may be easier to mechanize the process if the curved surface is expressed by a function of a curved surface including those points than the coordinate data of floating line segments.

  Such special curved surfaces are objects such as cloths, kimonos, bags, towels, etc., complicated curved surfaces, human fingers, arms, clothes, sandstones, distant mountains, curtains, etc. This is also the same expression.

  FIG. 12 shows an embodiment when the CAD data is updated in the first means of the present invention.

  (A) is raw image information when an object is photographed by the imaging devices 10 and 20 located far away from the object. When the object is photographed close to the object, (B) is obtained. This object is like a desk, for example.

  When CAD data is generated based on raw image information (A) obtained by photographing this object from a distance, a rectangular parallelepiped composed of line segments L1 to L9 is generated as shown in (a). Therefore, when CAD data is generated based on raw image information (B) obtained by shooting the imaging devices 10 and 20 close to an object, data as shown in (b) is generated. In this case, new line segments L10, L11, L12, L13, and L14 are generated and added, and the coordinate data of the start and end points of the line segments L1 to L9 are also measured nearby, so that the accuracy is higher. Updated to coordinate data.

  Specifically, one outline generated based on raw image information obtained by photographing an object from a distance improves the resolution according to the raw image information obtained by approaching and photographing. There may be two contour lines, which may result in more accurate shape data.

  Here, the description will be made by updating the two-dimensional screen database. Similarly, all the related three-dimensional databases are also updated.

  The line segments L1, L2, L3, and L5 generated based on the raw image information obtained by photographing the object at a position far away from the object are each one, but obtained by photographing at a close position. There are two line segments generated based on the raw image information. The line segments L1, L2, L3, and L5 are not deleted but updated to the newly obtained coordinate values as more accurate data of the nearest line segment. If there is no close line segment, delete it and create a new one. New line segments L10, L11, L12, L13, and L14 are newly numbered and additionally registered.

  In this way, if not only the generation of CAD data but also the update is automatically performed, the generation of CAD data based on the raw image information obtained by photographing by the movable imaging devices 10 and 20 is performed. , 20 moves, the better the accuracy. Also, since CAD data can be generated with a certain degree of accuracy even at the initial stage, this CAD data can be used from an early stage for robot control and the like.

  FIG. 13 is a conceptual diagram showing an embodiment of the second means. Here, it is assumed that the imaging devices 10 and 20 generate CAD data based on image information obtained by shooting while moving around a cylindrical object (cylinder) 901 as a target. Further, the cylinder 901 is an object on which a character 101A “A” is drawn.

  In such a state, in the imaging devices 10 and 20, the side line of the cylinder 901 and the upper or lower disc ridge appear to be an ellipse. However, all of these lines are outlines, and actual lines do not exist there. Therefore, when generating the three-dimensional CAD data, if the image is generated by densely capturing the image information of the ridges of the cylinder 901 and detecting a lot of contour lines around the periphery, the data is generated in the three-dimensional database 200G. In this case, CAD data of a wire frame like a cylinder 902 is generated.

  In addition, the line segment constituting the letter “A” drawn on the surface of the cylinder 901 is an actual line that is a line segment actually present there. For example, it is generated as line segment data constituting the character “A” 902A in the CAD data. If such a contour line and an actual line are properly used when using CAD data, the CAD data can be used efficiently. For example, since the line segment data of the contour line does not actually exist, it becomes an obstacle when searching for the line segment data of the character “A”. In addition, if you want to refer to the shape data of a cylinder, you can get more detailed data by using many contour lines, so it was determined that this is an actual line when generating the data. CAD data is generated so as to be defined as an actual line at a stage. In this case, when the imaging devices 10 and 20 are moved, the outline of the side surface of the cylinder 901 does not change even if the imaging devices 10 and 20 are moved, whereas the letter “A” is When the imaging devices 10 and 20 are moved, the visible place changes. When the positions of the line segments before and after the movement of the imaging devices 10 and 20 are compared after being converted into three-dimensional data, the outline line segment before the movement is not visible as a line segment at the position after the movement. However, the character A, which is an actual line, identifies that the line segment constituting the character “A” is an actual line segment from the characteristics that exist in the same place even when the imaging devices 10 and 20 move. be able to. Further, in the case of a side contour line such as a cylinder 901, the position of the contour viewed by the compound-eye imaging devices 10 and 20 is slightly shifted. However, when generating the contour line by stereo measurement, Since it affects as an error, it is easy to be careful when using it if it is distinguished that the contour line includes such an error.

An example of the concept of a three-dimensional database is shown below. One method for distinguishing among line types is shown below. A line segment L1 indicates an actual line, and a line segment L2 indicates an outline.

次に、第２の手段の他の実施の形態を説明する。直方体の基本図形をソリッドデータＳＶ１として生成する方法を図７を参照して説明したが、撮像装置が移動し、ＣＡＤデータを更新するようにしたシステムでは、撮像装置が遠くから物体を見たときには直方体であっても、近くに寄って見たときには、例えば、サイコロのように直方体の角に丸みがある場合には、それを直方体として定義しておくことが適切ではなくなってくる。また、大きな建物を遠くから見ると直方体に見えるが、近くに寄ればドアがあり、中に入ると最初に直方体と見えていた建物は壁のような平面で囲まれた物体になる。更に、壁に近くに寄って見ると、厚みのある複数の平面の偏平な直方体となるようになることから、このようなシステムの場合には、必ずしもソリッドモデルにまで落とし込む必要はない。そこで、必要最小限のデータ構成を考えると、点と線と面まで上げられるので、必要最小限の面データまでを生成して更新していくシステムとするのが処理速度の観点などから有利である。

Next, another embodiment of the second means will be described. The method for generating the basic figure of the rectangular parallelepiped as the solid data SV1 has been described with reference to FIG. 7, but in a system in which the imaging device moves and updates the CAD data, the imaging device sees an object from a distance. Even if it is a rectangular parallelepiped, when viewed close to it, for example, when the corner of the rectangular parallelepiped is round like a dice, it is not appropriate to define it as a rectangular parallelepiped. Also, when you look at a large building from a distance, it looks like a rectangular parallelepiped, but if you get close, there is a door, and when you go inside, the building that was initially visible as a rectangular parallelepiped becomes an object surrounded by a plane like a wall. Furthermore, when viewed closer to the wall, it becomes a flat rectangular parallelepiped having a plurality of thicknesses. In such a system, it is not always necessary to drop it into a solid model. Therefore, considering the minimum necessary data configuration, it is possible to raise points, lines, and planes, so it is advantageous from the viewpoint of processing speed to create a system that generates and updates the minimum necessary plane data. is there.

Next, another embodiment of the second means will be described. The database already described has a form that expresses line data as a function and a form that also expresses surface data as a function. However, expressing surface data as a function saves in terms of processing complexity and data memory saving efficiency. Is less effective. Therefore, it is proposed that the data amount be greatly reduced by expressing the function of at least line data. For example, when processing is performed for a normal environment, an enormous number of line segments are extracted. If you try to express the shape of these line segments as a straight line, for example, a line segment with a circle or a complicated bent shape will define many points on the line segment, and a straight line between those points will be defined. It must be connected with line segments to express a circle or a bent shape, and a lot of point data is also required, so a huge memory capacity is required. Therefore, the amount of data can be greatly reduced by defining the curve as a function and expressing the curve with appropriate coefficients (a1, b1,... In the above data example).
Another embodiment of the second means will be described. In this embodiment, color information is given to at least surface data in CAD data to be generated and updated. The concept is as follows.

勿論、赤，青…という表現ではなく、色コードで微妙な中間色まで区別できるようにしても良い。このように、ＣＡＤデータに色情報があれば、赤いものを探すというような場合には赤い色情報をもつＣＡＤデータのみを探せば良いので、ＣＡＤデータを取り扱うときの効率が良くなる。また、色情報で色の変わるところを境界線の線分データとして積極的に定義すれば、より多くの形状を示す線分データも得られるし、色の分布状態をより詳細に定義することも可能となる。

Of course, it may be possible to distinguish subtle intermediate colors by color code instead of red, blue, etc. Thus, if there is color information in the CAD data, it is only necessary to search for the CAD data having the red color information when searching for the red one, so that the efficiency in handling the CAD data is improved. In addition, if the point where the color changes in the color information is positively defined as the line segment data of the boundary line, the line segment data showing more shapes can be obtained, and the color distribution state can be defined in more detail. It becomes possible.

  FIG. 14 illustrates the concept of extracting a contour using color information. Reference numeral 110a is raw image information. A small frame of each mesh is a portion corresponding to one pixel. It shows the situation that there are various color surfaces and line segments as image information.

  Considering the creation of a two-dimensional image database on this image data, for example, for each pixel, the entire screen is processed while comparing the color of the surrounding pixels with its own pixel, and the pixel portion where the color changes If the boundary line is defined in the table, the result shown by reference numeral 110b is obtained. If it is further defined symbolically, the surface data 110c, 110f, and 110i define colors in their respective closed loops, and color information is also defined in the surface data. In addition, in the boundary portion, only the contour line data 110e or the line data 110d is defined in the surface 110f when only the line is used, and the line data 110h is also independently defined. The contour line 110e here indicates a boundary line of different colors, so if it is a painted pattern, it may be defined as an actual line. Here, a virtual point 110j is defined in the heel portion of the screen so that the plane data 110i can be defined. The concept of these diagrams is a concept of a two-dimensional image database. A two-dimensional image database may take a method of defining surface data in addition to line segment data, and surface data is defined only in a three-dimensional database. You may do it.

  Here, a supplementary description will be given of the procedure of the processing in FIG.

  The color and brightness of adjacent pixels are compared, and if it changes greatly, it is determined as a contour. Similarly, when searching around, a closed loop contour is created. It is registered as contour line data (colorless at the border) and closed loop surface data. Those without a region (area) are registered as line data.

  Next, another embodiment of the second means will be described. An example of CAD data is shown below.

ここでは、最大の明るさを１００として、背景面Ｓ０は５０の明るさ、赤い面Ｓ１は２０の明るさ、青い面Ｓ２の明るさは１００と例示している。このように定義しておくことによって、例えば、面Ｓ２が青ランプで光る面である場合には、その明るさデータ１００を見れば、その青ランプは点灯していると判断させるような使い方をすることができるようになる。ここで、ＣＡＤデータを生成するときに、撮像装置１０，２０によって得た画像情報の当該面の輝度値を参照してその情報をＣＡＤデータに登録すれば、ＣＡＤデータに明るさの情報を追加することが可能である。

Here, the maximum brightness is set to 100, the background surface S0 is illustrated as 50 brightness, the red surface S1 is illustrated as 20 brightness, and the blue surface S2 is illustrated as 100 brightness. By defining in this way, for example, when the surface S2 is a surface that shines with a blue lamp, a method of using the brightness data 100 to determine that the blue lamp is lit is used. Will be able to. Here, when CAD data is generated, if brightness information of the surface of the image information obtained by the imaging devices 10 and 20 is referred to and the information is registered in the CAD data, brightness information is added to the CAD data. Is possible.

Next, an embodiment of the third means will be described. An example of CAD data is shown below.

以上の例では、面データＳ１，Ｓ２は、時刻13:15に生成されたデータと時刻13:20に生成されたデータの２種類が存在する部分を示している。このように、そのデータを生成した時刻データを登録しておくようにすることで、例えば、面データＳ２は、明るさが１００から２０に変わったので青のランプが消えた、ということを容易に認識することができるようになる。前の時刻に生成されたＣＡＤデータと違いのあるものは残しておくようにすれば、同じＣＡＤデータを調べることで変化の状態や履歴も認識させることが可能となる。また、このようにしてデータを追加していくとメモリ容量が足りなくなるので、例えば、重複するデータは、生成した時刻から１時間とか１０分とかの所定の時間を経過した後には自動的に消去するようにプラグラムすることで、メモリ容量が爆発することを防ぐことができる。それは、例えば、ＣＡＤデータの時刻を常に確認するようにして、所定の時間が経過したものは消去するようにすれば、自動消去処理が可能である。

In the above example, the plane data S1 and S2 indicate portions where two types of data, that is, data generated at time 13:15 and data generated at time 13:20 exist. In this way, by registering the time data at which the data was generated, for example, the surface data S2 can easily indicate that the blue lamp has disappeared because the brightness has changed from 100 to 20. To be able to recognize. If the data different from the CAD data generated at the previous time is left, it is possible to recognize the change state and history by examining the same CAD data. In addition, since the memory capacity becomes insufficient when data is added in this way, for example, duplicate data is automatically deleted after a predetermined time of 1 hour or 10 minutes from the generation time. By programming in such a way, it is possible to prevent the memory capacity from exploding. For example, automatic erasure processing can be performed by constantly checking the time of CAD data and erasing data after a predetermined time has elapsed.

  15, FIG. 16, FIG. 17, and FIG. 18 are block diagrams showing embodiments of the tenth to eleventh means.

  FIG. 15 shows another CPU 300 connected to the image information processing apparatus 100, and the position and orientation information of the imaging devices 10 and 20, that is, the cameras calculated in the image information processing apparatus 100 each time are displayed in another CPU 300. This is a device that can be taken out with Here, for example, by outputting the camera position and orientation information from the image information processing apparatus 100 at regular intervals, the CPU 300 can obtain the camera position and orientation information at that interval. With this configuration, the position and orientation information of the camera can be used in an application system configured using the CPU 300. For example, in the case of a robot arm control system, if the CPU 300 controls the robot arm, the CPU 300 can be attached to the tip of the robot arm so that the CPU 300 can detect the position of the tip of the robot arm. And attitude information can be obtained, and the result can be applied to control.

  FIG. 16 shows a configuration in which the CPU 300 of the application system inputs a command signal from the CPU 300 to the image information processing apparatus 100 in order to obtain the position and orientation information of the camera, and the position and orientation information of the camera is displayed as image information in response. The apparatus is configured to be acquired from the processing apparatus 100. Therefore, the image information processing apparatus 100 can output the camera position and orientation information to the CPU 300 only when the CPU 300 requires it.

  FIG. 17 is configured so that a CAD data search command signal can also be input from the CPU 300 of the application system to the image information processing apparatus 100, and as a response, information (shape or shape of CAD data searched by the image information processing apparatus 100 may be used. , Position and orientation information) may be acquired by the CPU 300. As a result, the CPU 300 side can easily obtain both the position and orientation information of the object to be controlled and the position and orientation information of the other party that is necessary for the control, so that information is used for the control. Will be able to.

  In FIG. 18, not only the search command signal but also the command signal for outputting the camera position and orientation information is output from the CPU 300, and the search result and the camera position and orientation information are acquired from the image information processing apparatus 100. It is the apparatus comprised in. In the apparatus configured as described above, for example, when a cup is held by an arm, a camera is provided at the tip of the arm, and the image information processing apparatus 100 is made to search for CAD data of the cup to be held by the arm. The position and orientation information of the cup can be obtained from the CAD data search result, and the position and orientation information of the tip of the robot hand can be obtained as camera position and orientation information and used for control. . Since the camera position and orientation information is information calculated when CAD data is generated, there is an advantage that the camera position and orientation information can be obtained in a short time because it does not require another time for calculation processing. Of course, the position and orientation of the target cup can be searched using CAD data, and the robot hand can also be searched from the shape data of the hand to obtain the position and orientation information of the hand.

  FIG. 19 is a flowchart showing an embodiment of processing of the image information processing apparatus 100 when the tenth means to the eleventh means are implemented as a visual information processing apparatus dedicated to robot control.

  The processing of the image information processing apparatus 100 starts from start 123a and performs command input processing 123b from the CPU 300 or the like in the application system. Here, in the case of a command called CAD data search, information on what kind of CAD data is input.

  Next, the processing proceeds to image information processing, and first, image information capturing processing 123c is performed. Subsequently, three-dimensional measurement, 3D-CAD data generation processing, camera position and orientation calculation processing 123d are performed. Then, a CAD data search process 123e is performed.

  And the process 123f which outputs the position and attitude | position information of CAD data searched with the camera to CPU300 is performed.

  Thereafter, the process repeats from the command input process 123b. By making this repetitive cycle be performed at a high speed of several tens of milliseconds, the CPU 300 that controls the robot or the like in real time can easily acquire information used for controlling the robot.

  Specifically, as shown in FIG. 19, the camera 123g provided in the robot hand unit and one of the objects in the environment, for example, so that the position and orientation information of the cylinder 123h can be known in real time. Therefore, it is possible to easily realize control for guiding the robot hand to hold the cylinder 123h.

  With reference to FIG. 20 and FIG. 21, another embodiment of the eleventh means will be described.

  FIG. 20 shows a scene where a car is running on a road in a city with traffic lights and buildings, and an example of searching for a signboard of the HITACHI building 904 in the scene will be described.

  As a method of searching for CAD data, a method of searching for a thing close to the basic figure based on the basic figure is adopted. In this case, if the size of the signboard is known, a rectangular body having a predetermined dimension is prepared as the reference CAD data 904a as a basic figure, and the CAD data that is close to it is searched from the generated CAD data. By doing so, the signboard of the HITACHI building 904 can be easily searched.

  Further, in the example shown in FIG. 21, only a limited shape can be searched by using only basic figures. Therefore, a part of CAD data generated in advance is extracted and input as reference CAD data 904b for search. To do. In this case, since the CAD data itself is generated, there is always a portion that matches with high accuracy, so that the search can be performed more reliably and accurately. In addition, it becomes possible to search a part of a block of CAD data having any shape not included in the basic figure.

  Here, the method of reading the search data in advance is an example of the eleventh means. For example, a cuboid that can be changed to an arbitrary size is displayed in the generated CAD data, and the cuboid of the cuboid is displayed. There is a method of reading out CAD data as CAD data for search in an area. According to this method, a necessary part can be easily taken out. If the area 904A is designated with a size that encloses the HITACHI building 904, the CAD data for search including the signboard portion can be read out. By searching with this data, the character data of the signboard is also used as characteristic information when searching, so that the target search target can be found more reliably. Of course, if various locations are read out, it is possible to construct a system in which the locations can be searched and confirmed immediately when necessary. The area may not be a rectangular parallelepiped but may be a sphere that can be set to an arbitrary size. Since it is not necessary to specify the posture if it is a sphere, the operation becomes easy.

  The eleventh means is a method of reading a large area including the sign as search CAD data after first searching for the sign with the basic figure. With this method, it is possible to search using only the basic figure at first, and then to use the part of the signboard including characters of the signboard for searching. Of course, even if it is not a basic figure, if a similar place is searched and the vicinity of the similar place is read as search CAD data, the search CAD data suitable for searching for the place is used. Can be obtained.

  By using the eleventh means in this way, even if there is only a basic figure at first, it is possible to search for a similar place gradually and use it as new search CAD data. It is also possible to create a data collection.

  In the eleventh means, when searching with a basic figure, for example, when the size of a rectangular parallelepiped of a signboard is unknown, an approximate ratio of vertical and horizontal length that indicates a long rectangular parallelepiped is designated. This is a technique for searching for a plurality of rectangular parallelepiped objects close to the ratio. This method is effective when it is desired to search from a large size to a small size if the shapes are the same. This is useful when searching for valves with the same shape and various sizes, such as valves and pipes. In addition, even when searching from children to adults in a human-like form, by allowing a certain degree of matching, the same search data can be specified in a size-free manner to search for large and small similar items. Will be able to. Since the search data need not have the same number as the size, memory can be saved.

  With reference to FIG. 22, another embodiment of the eleventh means will be described. The imaging apparatus 10 enters the room 905a together with the object 905 to be imaged, images the object 905 from various angles, up and down, left and right, generates image information, and generates CAD data. In this case, the object 905 to be imaged is a flower.

  Since a flower in a general environment is shaded by various things, it is difficult to generate CAD data of the entire shape of the flower, but by using this method, accurate CAD data 905c of the object 905 to be photographed is used. Can be easily obtained. Further, if no other object is present in the dedicated room 905a, the other object will not be mixed even if the target object 905 is photographed from any angle. The interior of the room 905a may be white or blue on the floor, wall, and ceiling, or a grid such as a grid is drawn on the floor, wall, or ceiling of the room, and the grid is generated when CAD data is generated. May be used.

  An embodiment of the eleventh means will be described with reference to FIGS. 23, 24, 25 and 26. FIG. 23 to 26 are for explaining an example of the reference CAD data search process 123e in the basic process described with reference to FIG.

  FIG. 23 is a summary of the search example conditions. The image information (video) of the current imaging device (camera) includes four rectangular parallelepipeds, and three of the rectangular parallelepipeds are stacked in three stages. It is assumed that CAD data has been performed to some extent, and CAD data having a similar environment has already been generated. Specifically, it is assumed that solid data SV1, SV2, SV3, V4 corresponding to each rectangular parallelepiped has been generated, and data with solids SV1, V4 on the plane S0 has been generated. In this state, the reference CAD data to be searched is a two-layered rectangular block.

  An example of the process (1) in the search process 123e will be described with reference to FIG. This process is a process of calculating a feature amount of search CAD data (reference CAD data). There are various features that can be considered depending on CAD data. Here, volume, slenderness ratio, vertices, line segments, number of faces, number of irregularities, number of basic shapes (two cuboids), line segments, These include surface characteristics (functions for curved surfaces), color information, and the like. These calculation processes are performed only once when new search CAD data is input.

  An example of the process (2) in the search process 123e will be described with reference to FIG. Specifically, it is a process of searching for CAD data having the same feature quantity as the feature quantity extracted in the process (1) from CAD data in the field of view that is currently visible.

  In this example, since there are four cuboid data, each of the four cuboids (candidates 1 to 4) and the stacked cuboids have two combinations of two (candidate 5 and candidate 6) and three more stages. Since there are a total of seven candidates for the combination of the overlaps (candidate 7), if the feature amounts of the seven candidates are compared with the feature amounts of the reference CAD data to be searched, candidate 5 is the best match. In a simple environment in which a basic figure can be searched, the candidate CAD data with the most approximate basic shape is the search result, and as a result, the center position and orientation information of this candidate CAD data is output. You can do that. This may be performed all at once in the basic search process 123e of FIG. 19, or may be performed in several steps. That is, the search result may be output about once every 10 generation cycles of CAD data, or may be output every time.

  FIG. 26 is a processing method that is particularly effective when the environment is not only a basic figure. Reference CAD data and candidates after selecting one closest candidate in process (2) in search process 123e, or in some cases, selecting a plurality of candidates. This is a process (3) of searching for a more appropriate candidate by performing pattern matching of CAD data.

  This example is a 3D potential matching method, in which reference CAD data is placed at the most probable position and orientation of the closest candidate to perform final matching. If the degree of matching is equal to or greater than a predetermined value, the object is specified as an object to be searched, and if it is less, the search is continued as no.

  In the matching processing, the search speed can be increased by extending the pattern potential matching method, which has already been established by the two-dimensional processing, to three dimensions, or using the eigenvalue space matching method. Here, the potential matching method is used to extend the two-dimensional potential matching processing described in Japanese Patent Laid-Open No. 02-137072 to three-dimensional processing. The eigenvalue space matching method is described in the literature in IEICE Transactions on Electronic Communication D-II Vol. J80-D-II No. 5 PP. The method described in 1136 to 1143 is applied. In any case, it is preferable to first use a variety of matching methods for final determination by narrowing down candidates by feature amount. The matching method used here is not limited to the above. A matching method that is performed in combination with a method capable of high-speed and high-precision matching may be used. Further, a method of using a combination of a plurality of matching methods may be used.

  Here, the search CAD data may be three-dimensional or two-dimensional. The two-dimensional CAD data is a pattern, a picture, a signboard character, or the like. That is, they are given as CAD data in which various surface data and line segment data are attached to one surface data. Color information may be attached to the surface. Even if such CAD data for search is given in two dimensions, the object to be searched is the space of the three-dimensional CAD data, so that the most matching portion of the two-dimensional plate is searched from the three-dimensional space. Become. In that case, the two-dimensional plate searches for a matching place by changing its orientation and posture in various directions. As in the case of three-dimensional search CAD data, it is preferable that the scale factor is variable so that the same plate, that is, a pattern or a mark, can be searched from large to small.

  Here, a supplementary explanation of the matching search will be given. In order to reduce the amount of data generated in the CAD data of the generated environment and the input CAD data for search, the line data is expressed by the start point, the end point, the line type, the function parameter of the line, etc. It is expressed by the name of the line segment to be configured, etc., but at the time of the search, as for the portion to be matched, if it is a line segment, it will be a plurality of point data on the line, and if it is a plane data, it will be Conversion to point data may be performed to perform matching processing between the point data, that is, processing for obtaining correlation. The point data to be replaced from the surface data having the color information is also provided with the color information, and matching is performed in consideration of the color information. When the matching process is finished, the replaced point data can be deleted. By replacing the data with point data each time, the amount of data as CAD data that should normally be kept can be reduced.

  Moreover, although it is the density at the time of replacing with a point, it is good to change appropriately according to matching accuracy. Basically, replacement is performed at equal pitches and equal intervals, but it is preferable to increase the density of locations of complex shapes and complex color patterns and to coarsen the locations of color data of simple shapes and simple patterns. In addition, if solid CAD data is used for management and input / output, the solid data is also replaced with point data in the solid space with a constant density or by providing dense and coarse portions. It is better to match the point data. By replacing with point data, a general-purpose matching process can be used in common. Here, the method of replacing with point data has been described, but it may be used as long as it can be matched on a function basis.

  Another embodiment of the eleventh means will be described with reference to FIG. When searching for reference CAD data from among a large amount of CAD data generated extensively, in order to increase the search speed, the entire space where the CAD data is generated is divided into block addresses, and the partition addresses are designated. The search range should be narrowed down.

  FIG. 27 shows the concept of searching by dividing a space into blocks of the same size. The camera which is the imaging device 10 exists among them, and the camera 10 has shown the condition which image | photographs the range of the visual field 10V.

  In the case of searching, there are a method of searching a range that is visible with the camera 10 and a method of searching a range that is not visible. Of course, there is a method of searching while CAD data is being generated, and there is also a method of performing only a search process offline using the generated CAD data. In any method, it takes a very long time to search the entire area. For example, if the target space is all over Japan and the current target area is Tokyo, only the block address in Tokyo is searched. Try to narrow down by the method.

  In another embodiment of the eleventh means, a block in the direction of the direction of the camera 10 is searched. That is, a block in the visual field 10V of the camera 10 is a search target. Here, the CAD data to be generated is classified and managed by block addresses at the time of generation, so that the target CAD data can be automatically narrowed down if a block is designated. If processing is performed by specifying a block, the search can be performed efficiently.

  FIG. 28 is a block diagram of a visual information processing apparatus which is another embodiment of the eleventh means. This visual information processing apparatus has a configuration in which the image information processing apparatus 100 is provided with two systems for inputting a search command signal from another CPU 300 and two systems for outputting a search result as an I / F. Of course, the number may be three or more, and may be configured to be connected to a plurality of CPUs 300 instead of a single CPU 300.

  As described above, by providing the image information processing apparatus 100 with a plurality of I / Fs, even if the processing speed of the image information processing apparatus 100 is increased, the data transfer speed with the CPU 300 is slow, and the improvement of the processing capability of the system is limited. Such a situation can be avoided. The CAD data may be composed of many points and line segment data, or a lot of candidate data is searched by searching, and color or shape information is read in addition to the position and orientation information. Takes a lot of time. Therefore, by providing a plurality of I / F entrances in the image information processing apparatus 100, it is possible to efficiently perform a search process for a lot of CAD data.

  With reference to FIG. 29 and FIG. 30, another embodiment of the eleventh means will be described.

  FIG. 29 shows the concept of the generated CAD data. Actually, the CAD data is composed of a set of fine line segments and surface data. Here, it is assumed that the search CAD data is registered in advance as one set. They are specifically desks, vases, flowers, and cups.

  Therefore, since a similar set of CAD data exists when the CAD data in the room is generated, the search is performed using CAD data generated in advance only for the object for searching. For example, if a search is performed using flower data, a line segment or a block of surface data constituting the flower 905 is searched, and this is defined as ID = 1. ID = 1 is a flower in the process. Next, when the vase 906a is searched, the data of that portion is searched, and it is defined as ID = 2. Similarly, the data portion of the cup 906b is defined as ID = 3, and when searching using the data of the desk 907, a group of line segments and surface data which are considered to be a desk are searched, and this is defined as ID = 4. To do. When this ID is registered, the CAD data of the line segments and faces constituting the ID is grouped. In the CAD data defining the ID, the position of the representative point of the ID and the posture information of the object as a cluster may be registered. If the matched portions are defined and managed as a lump in this way, the next search can be conveniently performed by specifying an ID and performing a search.

  If there is a lump of the portion 908 that has not been matched with the CAD data for search prepared in advance, the lump portion is an unrecognized object. In such a case, the application system may move the camera up and down, left and right around the unconfirmed object 908 to photograph the unrecognized object 908 and generate CAD data with higher accuracy. If the unidentified object 908 is managed by attaching a new ID number that has not been taken so far, even if there is an unconfirmed object, it is observed closely and detailed CAD data is generated to create a new one. It can be recognized as an object. Here, the fact that an ID is attached means that the system recognizes the object by the ID number.

  The ID may have a type and an individual number. For example, when there are two cups, ID = (1 of 3) may be managed first, and ID = (2 of 3) may be managed second. This means that they are managed with types and individual numbers.

In the eleventh means, a human word may be associated with the ID. The concept of CAD data in this case is shown below.

このように、ＩＤ＝１は、人間の言葉で「花」であることを対応させておくことにより、「花」を探索させることが可能となり、システムと人間とのＩ／Ｆを、操作（指示）し易いものに構築することができる。また、探索を行なうときに、例えば、「カップ（ＩＤ＝３）とか花瓶（ＩＤ＝２）は、机（ＩＤ＝４）の上にある場合が多い。」という知識があれば、探索時には、先ずは机（ＩＤ＝４）のＣＡＤデータの上に付いているＣＡＤデータを対象に探索するようにすることができる。このように範囲を特定して探索することで探索効率を上げることが可能となる。

In this way, ID = 1 can be searched for “flower” by making it correspond to “flower” in human language, and the I / F between the system and human can be operated ( It can be constructed to be easy to instruct). Further, when searching, for example, if there is knowledge that “a cup (ID = 3) or a vase (ID = 2) is often on a desk (ID = 4)”, at the time of searching, First, the CAD data attached on the CAD data of the desk (ID = 4) can be searched for. Thus, it becomes possible to raise search efficiency by specifying and searching a range.

  FIG. 30 shows an embodiment for inputting knowledge data. In the search described with reference to FIG. 30, when a block of CAD data of an unrecognized object 908 occurs, the system The display screen of the LCD when the question “What is this object?” Is output to a human being and the human being inputs that it is defined as ID = 5 is illustrated.

  At this time, if knowledge data is also input together, a system can be constructed that eliminates input leakage. Knowledge data can be easily input by inputting it in human language. For example, after inputting ID = 5 = “parcel”, if knowledge 1 = “parcel” is entered “above” of “desk”, a necessary database is automatically created. If the words of all IDs are registered first, the knowledge database can replace the human words with the ID numbers and store the ID numbers in a predetermined place. Here, the word “above” may be stored by replacing it with some other symbol. However, “being over” means that the object on the next ID (space) It is premised that a search program is described so as to correspond to searching for CAD data at a higher position.

  In consideration of the case of registering search CAD data and registering each ID, knowledge data is also prepared in advance in a separate file offline and the knowledge data is collectively read into the system. Yes, knowledge data can be registered efficiently.

  It should be noted that the concept of the CAD data database is that the data is simply recorded in the record No. in the embodiment described above. Although it is described so that it is enumerated every time, when actually realizing the system, it is necessary to make the database easy to search, shorten the time to access various data, and finally the visual information processing device or It is advisable to increase the processing speed of the system so as to improve the real-time property.

  Another embodiment of the eleventh means will be described with reference to FIG. Originally, the block of individual CAD data shown in FIG. 29 is actually a complex collection of incompletely generated line segment data and surface data. In addition, since the number of data is large, when the matching degree matches to some extent after searching, the portion is replaced with a basic graphic defined by a basic graphic or a collection of basic graphic. By replacing all replaceable CAD data with basic graphics in this way, the amount of data can be greatly reduced.

  Next, another embodiment of the first to third means and the tenth and eleventh means will be described with reference to FIG. This embodiment is one of basic product forms of a visual information processing apparatus suitable for implementing the present invention.

  In this embodiment, the image information processing apparatus 100 is configured as a visual guidance chip in which the means (arithmetic processing apparatus) for realizing the image information processing function as described above is implemented as an LSI. Using a CCD camera, the storage device 200 uses a memory such as a smart card, a memory stick, an IC card, and a hard disk alone or in combination, and another CPU 300 uses a CPU on the application side such as a robot controller, An acceleration sensor 30 and a gyro sensor 40 are prepared for expansion.

  The visual guidance chip, which is the image information processing apparatus 100, has an input I / F for inputting image information, commands (commands), search CAD data, sensor signals, and the like, responses to the commands, imaging devices 10, 20 and search. An output I / F that outputs CAD data position and orientation information, a memory I / F that connects the storage device 200, and the like are provided.

  The visual guidance chip (image information processing device) 100 outputs the raw image information output from the CCD cameras (imaging devices) 10 and 20 and the acceleration sensor 30 and the gyro sensor 40 in accordance with a command from the CPU 300 on the application side. Processing to generate CAD data by fetching the sensor signal to be performed, execute image information processing for searching for reference CAD data, report the search result to the CPU 300 on the application side, and store the generated CAD data in the storage device 200. Execute.

  Here, the visual guidance chip that realizes the functions of the image information processing apparatus 100 may be divided into a hard chip unit for CAD data generation and update functions and a hard chip unit for CAD data search functions. In that case, the I / F to the storage device 200 is provided in both the hard chip portion of the CAD data generation and update function and the hard chip portion of the CAD data search function, and is controlled not to read / write the storage device 200 at the same time. With such a visual guidance chip configuration, it is sufficient to use only the hard chip unit for an application that requires only the CAD data generation and update functions. When the CAD data search function is required, the hard chip is used. Therefore, a rational application system can be constructed economically.

  Next, with reference to FIGS. 32 to 40, specific image information processing (CAD data generation and update function) executed by the visual guidance chip (image information processing apparatus) 100 of the embodiment shown in FIG. Will be described in detail.

  32 and 33 are flowcharts of main processing executed by the visual guidance chip 100. FIG. Here, an example will be described in which the four simplest commands that are the minimum necessary are prepared as commands from the CPU 300 on the application side. It is assumed that the command = 0 has a function of clearing the data area by resetting the origin, the command = 1 has a camera position request, the command = 2 has a search command, and the command = 3 has a data registration (filing) function.

  The main process of the visual guidance chip 100 is started by a reset signal after the system is turned on. First, in process (1), the current position and orientation data (Xc, Yc, Zc, αc, βc, γc) of the imaging device (camera) are reset to zero to define the world coordinate system. In the world coordinate system, the camera coordinate system at the start of power-on reset is defined as the world coordinate system. Then, the data storage area of the memory 200 that stores the generated CAD data is cleared to zero.

  In the next process (2), the command input from the CPU 300 on the application side is read. Here, each command is distinguished by four numbers of 0, 1, 2, and 3.

  In the next process (3), it is determined whether or not the command = 0. If Yes, the origin of the process corresponding to the command = 0 in the process (4) is reset, and the CPU 300 on the application side is processed in the process (5). Execute the process to output a response to the origin reset completion.

  If the command is not 0 in the process (3), the process jumps to the process (6). In this process (6), it is determined whether or not the command = 1, and in the case of Yes, a response process for outputting the current position and orientation information of the imaging devices 10 and 20 to the CPU 300 on the application side in the process (7). Execute.

  If the command is not 1 in the process (6), the process jumps to the process (8). In the process (8), it is determined whether or not the command = 3. If Yes, the CAD data generated so far in the process (9) is attached with a file name and the generated CAD data storage area of the memory 200 Register in a separate storage area. Then, in a process (10), a process for outputting a data registration completion response to the CPU 300 on the application side is executed.

  If the command is not 3 in the process (8), the process jumps to the process (11). Process (11) is capturing of raw image information from the imaging devices 10 and 20. Raw image information (1) and (2) obtained by the imaging device 10 and the imaging device 20 is taken into the visual guidance chip 100. When the imaging device 20 is not connected, monocular processing is performed. When the imaging device 20 is connected, stereo processing is also possible.

  In the next processing (12), 3D coordinate calculation and CAD data generation processing are performed. Details of this process (12) will be described later with reference to FIG.

  In the next process (13), it is determined whether or not the command = 2. If Yes, the search process for 3D-CAD data is executed in the process (14).

  Then, in the process (15), it is determined whether or not the search has been made. In the case of Yes, the number of candidates, the coordinate posture data of each candidate object, and the candidates are determined as the search results in the process (16). A process of outputting a matching degree and the like to the CPU 300 on the application side is executed, and the process returns to process (2).

  If it is determined in the process (13) that the command is not 2 or if the search cannot be performed in the process (15), the process returns to the process (2).

  By enabling the visual guidance chip 100 to execute such processing, various applications can be easily realized.

  FIG. 34 is a detailed flowchart of 3D coordinate calculation and CAD data generation processing, which is processing (12) in the main processing of the visual guidance chip 100 shown in FIG.

  In this process, in process (1), it is determined whether it is stereo or monocular, and in the case of stereo, the process branches to process (2).

  In the process (2), a plurality of corresponding feature points are extracted from the left and right image data of the stereo view.

  In the process (3), corresponding feature points of the extracted left and right image data are converted into 3D coordinates.

  In process (4), 3D-CAD data generation processing is performed. Since the method of calculating 3D coordinates in stereo view is common, detailed description is omitted here, but details of the process (4) after the 3D coordinates are calculated are separately shown in FIGS. Reference will be made later. Thereafter, the process (5) is executed by joining with the case of a single eye.

  The process (5) is a process that branches in the case of a single eye in the determination of the process (1). In the case of stereo, after executing the processes (2) to (4), the process merges with the process (5). In this process (5), a plurality of feature points or feature objects of the captured image data are extracted.

  Next, in process (6), the feature points that can be associated with the feature points of the previous captured image data are given the same names as the previous ones, the new feature points are given new names, Memorize temporarily. Processes (5) and (6) are feature point tracking processes. In the case of stereo, processing may be performed on at least one of the left and right image information.

  Next, in the process (7), it is identified whether or not a plurality of pieces of image data stored while tracking are prepared.

  In the case of Yes in the process (7), 3D coordinate calculation in the photo measurement of the process (8) is executed. Specifically, a plurality of corresponding feature points are converted into 3D coordinates. At this time, since the position and orientation information of the camera that captured each image is also calculated, the information is also temporarily stored. The camera position and orientation information of the latest captured image is set to variables Xc, Yc, Zc, αc, βc, and γc.

  Then, in the process (9), the 3D-CAD data generation process is executed, and the process returns with RETURN. Process (9) is the same as process (4).

  If a plurality of pieces of image data are not found in the determination of the process (7), the process returns as it is with RETURN. Although the minimum number of image data required for photo measurement is two, it is preferable to measure the photo using a larger number. In the calculation of 3D coordinates, after the start, 3D coordinates are calculated using the camera coordinate system that captured the first captured image as the world coordinate system.

  35 to 38 are detailed flowcharts of the 3D-CAD data generation processing of processing (4) and processing (9) in the processing shown in FIG. Here, the details of the update function in the most basic first means will also be described.

  If this process is entered, M (MAX) feature points Q1 to QM (MAX) converted into 3D coordinates in the process (1) are converted into feature points P1 to P1 of N CAD data that have already been generated. It is determined whether or not the PN is a feature point at a position sufficiently separated from the line segments L1 to LL of the L CAD data already generated. When there are M feature points that are sufficiently separated, the M feature points are added to the already generated CAD data. That is, a process of adding M points to N CAD data is performed. Here, the determination as to whether or not the distance is sufficient is made by determining whether the distance between the Q point and the P point or the shortest distance between the Q point and the L line segment is the maximum error that may occur when the Q point is calculated and the P error. Compare the sum of the maximum possible errors when the point or the point close to the Q point (LP point) of the L line segment is calculated, and if it is farther than that, the near P point or LP Judged to be a new point that was sufficiently far from the point. If determined in this way, the same feature point is not erroneously determined as another feature point. Here, it is assumed that the maximum possible error is simply assumed, but strictly speaking, the error that occurs varies depending on the direction, so more accurate determination can be made by considering the direction and the magnitude of the error. So you should do that. This process (1) is a process for determining whether or not the generated CAD data is an update target of the already generated CAD data, and M feature points that are not the update target are newly added. is there. The remaining M (MAX) -M feature points become the feature points to be updated, and the update process is performed after the next process (2).

  In the process (2), the feature point that is not sufficiently separated from the existing point P or the line segment and is approaching again is redefined as Q1 to QM for the next process. Here, the number M is M (MAX) -M.

  In the next process (3), KN P points and LP points generated before the feature points redefined as Q1 to QM in process (2) are defined as KP1 to KP (NK). Here, since there may be a plurality of Q points close to one P point, M = KN is not always satisfied, and therefore, the number is defined as a separate number. At this time, since the relationship between the Q point and the KP point that are close to each other is already known, it is noted that if the feature points are close to which feature points, the subsequent processing becomes easier.

  Process (A) from process (4) to process (15) shown in FIG. 36 is a point data update process.

  In the process (4), i = 1 is set, and in the process (14), i = i + 1 in the process (15) until i = KN, and the process after the next process (5) is repeated. That is, the same processing is performed for KN points KP1 to KP (KN).

  In the process (5), KM feature points among the existing points Q1 to QM close to KPi are set as KQ1 to KQ (KM), and in the process (6), TEN = 0 and j = 1 are processed (12 ) Until all the combinations of KQ1 to KQ (KM) are checked with j = j + 1 in the process (13) until j becomes KM.

  In the process (7), the accuracy when KPi and KQj are calculated is compared. If the precision of KQj is UP (improved), the update process is performed in the process (10) or the process (11). If the accuracy is not up, the process jumps to the process (12).

  If the accuracy is increased, for the same KPi point, in the case of the first process, the coordinates of KPi are simply updated to the coordinates of KQj in process (10). For the same KPi feature point, after the second time, the division update process is performed in the process (11). This is because one feature point called KPi is divided into two or more before, so that P point is defined in addition to the first existing KPi point. Therefore, the process of N = N + 1 is also performed here.

  Whether or not KPi is a point (LP point) on the line segment in the process (9) is entered in the determination logic if the feature point on the line segment is not updated in the coordinate value of the process (10). This is because a new feature point needs to be added and divided and updated in the process (11).

  If the above processing is performed for all KPi points, the point data update processing is performed.

  The process (B) from the process (16) to the process (27) shown in FIG. 37 is a process for updating and adding line segment data.

  In the first process (16), when updating the line segment data, a process of picking up all point data in the target range is performed. That is, all point data in the field of view of the image data referred to when converting to 3D coordinates are extracted and defined as KN KP1 to KP (KN) points. Here, the P point added in the previous process and the P point with updated coordinate values are also included.

  Next, i = 1 is set in the process (17), and the processes after the process (18) are repeated while i = i + 1 in the process (27) until i = KN in the process (26). That is, the same processing is performed for all target point data, KN points KP1 to KP (KN).

  In process (18), j = i + 1 is set, and in process (24), j = K + 1 is set and j = j + 1 is set, and all combinations of KP (i + 1) to KP (KN) are checked. J is set from i + 1 to KN so that the same combination need not be processed twice.

  In the process (19), the next process is divided into a process (20) and a process (22) depending on whether or not the line segment data connecting the points KPi and KPj already exists.

  If the line segment data already exists, it is checked in process (20) whether either of the points KPi or KPj has been updated or added, and if neither has been updated or added, the line segment is The process jumps to the process (24) without performing the update process. Since there is no change in either the start point or the end point of the line segment, it is determined that there is no change to review the line segment in the middle, and the process is omitted. Of course, even if there is no change in the start point and the end point, the line segment in the middle may be better imaged. Therefore, the processing (20) may not be provided, and all the processing may proceed to the processing (21).

  The process (21) is a process of updating a line segment, and is a process of revising the line segment data connecting the KPi point and the KPj point while referring to the image data that has become a line drawing by contour extraction or the like. Since the KPi point and the KPj point calculated in the 3D coordinates are associated with each other as points on the two-dimensional image data used for calculating the KPi point in the image data, A line segment connecting the point and the KPj point is extracted, and the line segment data is three-dimensional line segment data, that is, three-dimensional point sequence data connecting the KPi point and the KPj point, or connecting the KPi point and the KPj point. The function of a three-dimensional straight line or curve is obtained and the line segment data is reviewed and redefined. Here, when either KPi point or KPj point is added or updated in the process (20), it means that the accuracy of the starting point and the ending point has been improved, so if the line segment data is also reviewed. The processing is expected to improve accuracy. In addition, even if the existing line segment data is a straight line, if it is impossible to fit to the straight line by reviewing when the accuracy condition is improved, the line segment data of the straight line at that time will be Change to line segment data.

  On the other hand, when there is no line segment data connecting the KPi point and the KPj point, the process proceeds to the process (22), and there is a line segment connecting the KPi point and the KPj point with reference to the image data as a line drawing. If there is no connecting line segment, the process jumps to the process (24).

  If there is a line segment to be connected, line data addition processing is performed in step (23). That is, a line segment connecting the KPi point and the KPj point in the image data is extracted, and the line segment data is converted into three-dimensional line segment data, that is, three-dimensional point sequence data connecting the KPi point and the KPj point, or A line segment data is newly defined by obtaining a function of a three-dimensional straight line or curve connecting the KPi point and the KPj point.

  If the above processing is performed for all combinations of the points KP1 to KP (KN), the line segment data updating process is performed. Here, following the processing (21) and processing (23), the surface data may be updated immediately, but in this embodiment, the surface data is updated together in the following processing. ing.

  Process (C) from process (28) to process (40) shown in FIG. 38 is a process of updating and adding surface data.

  In the first process (28), in updating the surface data, a process of picking up all the line segment data updated in the previous process is performed. That is, all the line segment data updated or added in the process (B) is set as KL1 to KL (KN) as KN line segment data.

  Next, i = 1 is set in the process (29), and the processes after the process (30) are repeated while i = i + 1 in the process (40) until i = KN in the process (39).

  In the process (30), all the surface data whose components are the line segment data KLi are extracted and set to KS1 to KS (KM) as KM surface data. Here, the surface data to be extracted is all. All the surface data including the updated or added line segment data KLi as components are extracted from all the surface data. As a specific extraction method, the line segment data connected to the line segment data KLi is searched on the CAD data, and further, the line segment data is searched such as the line segment connected to the line segment to obtain the first line segment data. A process of extracting all the routes returning to KLi, that is, the closed loop created by the line segment data as a surface, is executed.

  In the process (31), j = 1 is set, and in the process (37), j = + 1 is set in the process (38), and the processes after the process (32) are repeated, and all the extracted surface data KS1. The same process is repeated for ˜KS (MK).

  In the process (32), it is determined whether or not the surface data corresponding to the surface data KSj is already defined as CAD data. Usually, the CAD data may be managed as the back and front of the same surface with respect to the so-called back and front of the surface composed of the same line segment connection, but here the table is a table with a more detailed shape in the future. Considering that the back side is divided and generated on the back side, the back side and the front side are preferably extracted as separate closed loops (different surface data). In the process (30), all the surface data related to the newly added / updated line segment data (connection of closed loop line segment data) is extracted, so that the surface data for which the surface data is already defined is also extracted. New surface data that has not been defined is also extracted. Therefore, in the process (32), whether the extracted surface data KSj has already been defined as surface data, or has not been defined so far and has been extracted as surface data for the first time, that is, new line segment data is newly created. It is determined whether or not a newly defined closed loop of line segment data has occurred due to the addition or update. Here, the determination is made by comparing the surface KSj with all the previously generated surface data existing in the vicinity thereof, the maximum error assumed for the line segment at the line segment level constituting the surface, and The maximum error assumed for the line segment constituting the surface KSj is considered, and the determination is made based on whether or not they match within the error. A simple comparison based only on the connection relationship of the line segments that make up the surface shows that when one line segment is updated to two line segments that are divided in the middle, the same surface is shown. However, since the constituent line segments are divided and defined in more detail, it may not be possible to distinguish them from the same plane. In addition, since one surface may be divided by generating new line segment data that divides the surface, all or part of the surface data KSj matches all or part of the existing surface data. It is determined whether or not it matches the existing surface. Therefore, it is determined whether there is the same surface data that overlaps within the error. If the surface data already exists as a result of the determination, the process proceeds to the updating process (33), and the surface data corresponding to the surface data KSj is not included in the CAD data created previously. In the process (35), the process proceeds to a process of adding and registering new surface data (line segment connection closed loop).

  In the update of the plane data in the process (33), the point data and line segment data have already been updated. Therefore, the line segment data constituting the plane data may be reviewed while referring to the updated contents. Although it is good, it is also possible to refer to the point data and line segment data before update, which are constituted by the surface data before update. When updating the surface, it is premised that there is existing surface data that matches the surface data KSj. However, when the surface data KSj and the existing surface data are entirely consistent within an error range. Can be defined by replacing each line segment data with new line segment data generated with higher accuracy. If the surface data KSj is smaller than the existing surface data and matches a part of the existing surface data, the surface data of the matching part is newly redefined and the remaining part is replaced with another one. Define and divide as one plane data. Further, although there is a portion where the surface data KSj matches the existing surface data, when there is a portion larger than the existing surface data, the surface data KSj is not updated. This occurs because the extracted surface data includes a case where a large closed loop is extracted, but since a small closed loop surface is also extracted, the update process is performed for small surface data. If this is done, the surface data of that part will be updated. Since CAD data is updated when it becomes more accurate, processing that does not greatly integrate a small closed loop is avoided.

  Whether the attribute of the surface data is updated after the line segment constituting the surface data is updated in the process (33). Whether there is sufficient image information to update the attribute of the surface data in the process (34). If not, jump to step (37). If there is updateable image information, the process jumps to processing (36).

  Since the process (35) is the case where there is no surface data generated before overlapping the surface data KSj, the surface data KSj is additionally defined in the CAD data as new surface data. The definition here is the definition of the line segment data constituting the surface data.

  Then, in the process (36), the attributes of the plane data are defined. The attribute of the surface data refers to the image data. Originally, surface data is defined by converting 2D image data into 3D coordinates, so 3D surface data composed of 3D coordinate point data and line segment data is included in 2D image information. Since correspondence with which point corresponds is taken, attributes of surface data such as color and brightness of the portion corresponding to the surface are read from the image information and registered as attribute data of the surface data. When jumping from the process (34), it is an update of the plane data, and therefore the attribute of the plane data originally defined is reviewed and updated. Assuming that it was originally updated in the process (33), the line segment data of the surface outline has been updated more accurately, so that the image information this time represents the correct color and brightness of the surface. In the processing (34), it is judged simply by the size or brightness of the area of the image information of the surface, that is, the outline is updated with high accuracy, but when viewed from an extremely oblique angle, etc. , It may be possible to judge only whether or not the scene is shaded and the outline is clearly visible, but the line connecting the camera and the face at the time of shooting depends on the normal of the face. In the case of being close, that is, in a condition where the surface is viewed from the front, it may be determined more strictly so as to be updated. In addition, when the transmittance and reflectance are defined as the attributes of a surface, the brightness of the object reflected in the surface or the object reflected and reflected in the surface is the position of the object. Since it is obtained after conversion into transmittance and reflectance using information such as relationships, it is not always necessary to define it in the first process. For example, when there is an object that can be seen through, the brightness of the surface data of the object when directly looking at the object, the brightness when viewed through the surface, and the distance relationship between them are all obtained. Sometimes, it becomes possible to estimate and calculate the transmittance of the surface, and it is preferable to update it at that time.

  The above processing is performed for all the surface data related to KLi line segments of KS1 to KS (KM), and further to all line segment data KL1 to KL (KN) that have been updated and added. To.

  FIG. 39 shows the concept when the point data and the line segment data are updated in the 3D-CAD data generation processing described with reference to FIGS.

  The diagram shown in step (1) is the concept of CAD data before update, and is data called a line segment L1 connecting the points P1 and P2. The dotted circles around the points P1 and P2 are circles of the maximum error assumed when the respective points are measured in 3D, and it is assumed that there is an error circle as a sphere on 3D. An error circle on each line segment is defined in the line segment L1, and a sphere having a radius as the maximum error is defined on 3D. Considering this in all points on the line segment L1, the envelope of the sphere is defined. It can be regarded as a face. That is the dotted line around the line segment L1. Here, the size of the error radius in the middle of the line segment L1 is assumed to be a size that is obtained proportionally by estimation from the error radii at the locations of P1 and P2 at the ends of the line segment L1.

  Step (2) shows a concept in which the newly obtained 3D-converted points Q1 to Q4 are drawn on the CAD data before update. A sphere whose radius is the error when each point is calculated is also drawn with dotted lines at the points Q1 to Q4. Considering the error radius, the points Q1 and Q3 may exist within the error radius of the point P1. That is, the point that was P1 before the update may be Q1, or may be Q3. Similarly, it can be seen that the point Q2 may be the point P2, and the point Q4 may be the point LP1 on the line segment L1.

  Here, the update process according to the present invention will be described. In step (3), the maximum assumed when each of the pre-update point data and the newly calculated point data which may be the same is calculated. Compare which of the errors is large (here, compare which of the spheres whose radius is the error is small). If it is small, it can be considered that the measurement is more accurate. Update it. Step (3) shows the concept of the update. That is, when the error circle at the point P1 is compared with the error circle at the point Q1, the error circle at the point Q1 is smaller, so the coordinate value of the point Q1 is replaced with the point data of the point P1. Similarly, the error circle of the point Q3 is also replaced because it is smaller than the error of the point P1, but since the coordinate value of the point P1 has already been updated to the coordinate value of the point Q1, here a new point P3 is defined and the point Enter the coordinates of Q3. This means that the point P1 before the update is divided into the points P1 and P3 after the update. Similarly, the point P2 is updated by replacing the coordinate of the point P2 with the coordinate of the point Q2, since the error radius of the point Q2 is smaller. Similarly, since the error radius of the point LP1 and the error radius of the point Q4 are also smaller at the point Q4, the coordinate of the point Q4 is the coordinate of the point LP1, but the point LP1 is a virtual point on the line segment L1. For this reason, a new point P4 is defined as the CAD data, and the update process is performed by putting the coordinate value of the point Q4 into the coordinate.

  As a result, as shown in step (4), P1, P2, P3, and P4 are updated as the point data as the updated CAD data. Also, as for the line segment update, the line segment data L1 connecting the point P1 and the point P2 of the CAD data before the update shown in step (1) is the line segment L2 and the point P4 connecting the point P2 and the point P4. And the line segment L3 connecting the point P1 and the line L1 connecting the point P4 and the point P3 are updated, and these processes are realized by executing the processes shown in FIGS. .

  FIG. 40 is a diagram in which the concept of new data generation and update in the 3D-CAD data generation process described with reference to FIGS. 35 to 38 is applied to the general graphic generation and update concept. It is.

  (1) shows that the line segment L1 connecting the point P1 and the point P2 has a length of 1 m at first, but the positions of the points P1 and P2 are updated more accurately by the next measurement. The concept of updating the length of the line segment L1 to 0.97 m is also shown. This is an update process with increased accuracy.

  (2) shows a situation in which the line segment K1 is further generated while the line segment L1 is generated and the line segment K1 is additionally generated. This is because the line segment L2 that was initially hidden behind the object and cannot be seen becomes visible when the imaging devices 10 and 20 are moved, and points P3 and P4 are newly generated. L2 also indicates the generated situation. This corresponds to the concept of generating a new line segment.

  In (3), the line segment L1 such as the side surface of the container is visible, and the line segment L1 is also generated in the CAD data. However, when the imaging devices 10 and 20 capture an object from near, a new line segment is generated. It can be seen that the line segment forms a rectangle S1 and is attached to the first line segment L1. In this case, the line segment of the rectangle S1 is newly generated by making it possible to see what has not been seen so far, and thus corresponds to the concept of generation but is in contact with the line segment L1 of the rectangle S1. Since the line segment L1 is divided into the line segment L1, the line segment L2, and the line segment L3, the concept is that the line segment L1 is updated.

  In (4), the line segment L1 is generated at first, but when viewed closer, the point P1 appears to be divided into points P1 and P3, and the point P2 appears to be divided into points P2 and P4. Since the line segment L1 is divided and generated into the line segment L1 and the line segment L2, in this case, the accuracy is also improved, and this is the concept of division update.

  The last (5) is the situation where the side is like the side of the container consisting of the line segments L1, L2, L3, L4 and the one like the rectangle S1 and the line segment L5. Also in this case, when the imaging devices 10 and 20 are close to the object and photographed in detail, the left side of the rectangle S1 has a shape of a rectangle S2, and the line segment L5 also becomes the rectangle S3. Even in this case, the update process is performed.

  These cases (1) to (5) are typical examples. However, when 3D-CAD data generation processing is performed based on the processing flow shown in FIGS. Will be done.

  A basic embodiment of the fourth, fifth, and sixth means of the present invention will be described with reference to FIGS.

  FIG. 41 shows the concept of a basic embodiment based on stereo measurement by two cameras, and is a so-called stereo measurement configuration in which the imaging object 901 is imaged by the two cameras of the imaging devices 10 and 20. The two imaging devices 10 and 20 are fixed in a known relationship with respect to each other's position and posture, and move together when moving. The image of the image information obtained from the imaging device 10 is (A), and the image of the image information obtained from the imaging device 20 is (B). (A) and (B) may be understood based on the concept of an image memory of a visual information processing apparatus incorporating image information.

  Here, as the image information processing, first, extraction of feature points and line segments in the image information from the imaging devices 10 and 20 is extracted. The line segment is a line connecting feature points. There are various methods for extracting the line segment, such as a method of performing a thinning process after differentiating the two-dimensional image data. It is preferable to adopt a stable method that can extract the characteristic part of the imaging target 901 in the same manner even if the image changes.

  Assume that feature points 1a, 1c, and 1e are extracted from the image (A), and line segments 1b and 1d that connect these feature points are extracted as line segments. Here, for the feature points and line segments to be extracted, it is preferable to extract a boundary line where luminance information changes for a monochrome image and color information (R, G, B luminance signals) changes for a color image. Here, such information is referred to as color information without distinction between black and white and color. Since the extracted line segment shows the color boundary on both sides of the line segment, each extracted feature point and / or line segment (feature point and line segment) has color information. To manage the data as feature points and line segments.

  In the extracted line segment in the image (A) of this embodiment, the right side of the line segments 1b and 1d is color 1, the left side of the line segments 1b and 1d is color 2, and the line segments 1b and 1d are the boundary lines. Indicates the extracted situation. Each line segment is managed with left and right color information (color 1, color 2). Many other feature points and line segments may be extracted from the image image (A), but now, the description will be focused on the extracted line segments described here. This is a stereo measurement technique. For the feature point 1a extracted from the image image (A), it is necessary to search for a point in the image image (B) corresponding to the point 1a. Since the positional relationship between the imaging devices 10 and 20 is known, the point in the image image (B) corresponding to the image 1a theoretically exists on the predetermined line segment 1f. . Actually, since there are various error factors, it is considered that they are present in a region sandwiched between the line segment 1h and the line segment 1g having a predetermined width rather than on the line 1f.

A method for obtaining a line segment where the corresponding point of the feature point 1a exists will be described with reference to FIG. The imaging surface of the imaging device 10 (which may be considered as the surface corresponding to the image image (A) in FIG. 41) is the XY plane of the coordinate system that defines the viewpoint O1, and the imaging surface of the imaging device 20 (the image image in FIG. 41). If the point P1 on the XY plane of the coordinate system of the viewpoint O2 is the feature point 1a in the image image (A) in FIG. The actual point P1 in the space is present on a line segment connecting the point P1 on the XY plane of the coordinate system defining the viewpoint O2 and the viewpoint O2, and the line segment is a coordinate defining the viewpoint O1. Photographed so that it exists on the line P0P3 on the XY plane of the system. The method for obtaining the line segment P0P3 is, for example, the intersection of the plane formed by three points, the XY plane of the coordinate system defining the viewpoint O1, and the P1 point on the XY plane of the coordinate system defining the viewpoint O1, the viewpoint O2, and the point O2. It is easily obtained by asking for. Since the actual search range here is a range photographed by the imaging device, the search may be performed on the line segment P0P3 .

  Returning to the description with reference to FIG. When all the points between the line segments 1h and 1g in the image image (B) are searched for the points corresponding to the feature point 1a in the image image (A) in the image image (B), the image image ( 6 shows a situation in which six points (a), (b), (c), (d), (e), and (f) in B) exist as corresponding candidate points. Here, in order to make sure that feature points are extracted as line segments, it is preferable that noise removal processing that does not treat one feature point not connected to the line segment is included in the process. Assume that each corresponding candidate point constitutes a line segment connected to that point.

  As the next processing, a point corresponding to the feature point 1a is found from the six candidate points. The procedure will be described below. The image image (C) in FIG. 41 is an image image in which the feature points 1a of the image image (A) are superimposed and displayed on each of the six feature points searched for in the image image (B). As actual processing, at this stage, since the image data is replaced with feature points and line segments, matching processing is performed using CAD data.

  As the first processing of matching, the degree of coincidence of the color information on the top, bottom, left and right of the feature point or the degree of coincidence of the left and right of the line segment is obtained.

  An example of each color matching will be described with reference to FIG. The feature point 1a is superposed on each searched for candidate point of the image (B) and examined. In the example of (a) of FIG. 41D, the feature point 1a is matched with the candidate point 1n. Now, the right and left colors of the line of the feature point 1a to be examined for the corresponding points are color 1 on the right side and color 2 on the left side. Further, the colors on both sides of the line segment on the candidate point side to be overlapped in (a) of (D) of FIG. 41 are color 3, color 5, and color 4. Three line segments are connected on the candidate point side, and each line segment has a boundary as color information. In this case, there is color information of three regions. Here, if the degree of coincidence between color 1 and color 3 and color 2 and color 5 is high, the corresponding point of feature point 1a is known as 1n. At this time, if the line is an actual line, the left and right colors match and become a very stable point, but if only one side matches, it becomes a candidate point. In some cases, it is more probable that one side is more closely matched than a similar part, although the left and right colors are slightly inferior. That is when one color is the background.

  Although details will be described later, here, a combination in which at least one of them matches is found and the number of candidates is further narrowed down from the six corresponding point candidates. As a process, a threshold value is set for the color matching degree, and when both sides do not match, the process is excluded from the corresponding point candidates. If both sides match, the process is left as a candidate that may be an actual line. Here, the expression “possible” is used because it is not possible to judge from this scene alone. Actually, the imaging device moves and changes the viewpoint, so there is an opportunity to measure the same point. Get higher. However, even if it is determined that it may be an actual line in a certain scene, it may happen that the background is just reflected in the same way. In some cases, it is determined that the outline is a contour line. As processing, when it is determined as such, data may be rewritten from an actual line candidate to a contour line candidate.

  Returning to the description with reference to FIG. 41, if the number of candidates becomes one at the stage of matching color information, further narrowing down is unnecessary. Actually, the width of the line segments 1h and 1g can be narrowed by correcting the distortion of the image information with high accuracy and accurately obtaining the positional relationship between the two units of stereo measurement. It is possible to narrow down efficiently at the point candidate stage.

When a plurality of corresponding point candidates still remain in the matching using the next color information, other matching is also performed. Various matching methods are conceivable. For example, referring to (a) of FIG. 41D, the line segment 1n1q, the line segment 1n1s, and the line segment 1n1r are candidates for the line segment 1b. Needless to say, there are line segments that are candidates for correspondence in (b), (c), (d), and (e) of FIG. First, if the left and right color information is different from the left and right, they are excluded from the candidates, but there may still be a plurality of remaining information. Therefore, a simple method will be described in (a). In this case, the line segment 1b corresponds to the line segment 1n1q . The rotation angle δα1, the rotation angle δα2 when the line segment 1b is made to correspond to the line segment 1n1s, and the rotation angle δα3 when the line segment 1b is made to correspond to the line segment 1n1r are obtained. There is also a method of narrowing down a group of rotation angles having the same tendency to candidate points when narrowing down and further performing similar narrowing down on other feature points other than the feature point 1a.

An example of the most accurate matching method will be described with reference to (D) (f) of FIG. In (f) of FIG. 41D, the feature point 1a is superimposed on the corresponding point candidate 1i of the image image (B), and then the point 1c of the image scene (A) is matched with the point 1j of the image scene (B). A conversion coefficient to be converted to is obtained, and the other point 1e is also converted using the same conversion coefficient and mapped onto the image image (B). In addition, if the line segments are complicatedly connected, all the line segments are mapped with the same conversion formula. Then, the line segment extracted from the mapping result and the image image (B), in this case, the line segment 1i1j , the line segment 1j1k , and the line segment 1k1m are matched and 1i is left as a corresponding point candidate depending on the degree of coincidence. , To determine whether to exclude. If the length of the mapped line segment is not the same as that of the extracted line segment, the matching degree is obtained by excluding an excessively long part in either line segment. This is because the boundary line segment is extracted on the one hand due to the influence of the shooting direction, but the boundary line segment may not be extracted from the other shooting direction. There are various methods for obtaining the line segment and the matching degree of the line segment. For example, the correlation coefficient may be obtained by returning to the line segment of the two-dimensional image data having the same thickness.

Here, one simple method will be described with reference to FIG. 41D (f). A feature coefficient 1a is superimposed on the corresponding point candidate 1i, and a conversion coefficient for superimposing the line segment 1b on the line segment 1i1j in the image (B) is obtained . The conversion coefficient may be considered as a rotation angle for perfectly matching the point 1a and the point 1i and then aligning the point 1c with the point 1j, or a combination of the rotation angle and a case factor (magnification). May be. Alternatively, as another method, the image image (A) is compared with a transparent glass plate, and the line segment 1b and the line segment 1d are written on the surface, and the glass plate is placed on the image image (B). Tilt angle when the glass plate is tilted back and forth, left and right, rotated clockwise or counterclockwise so that the line segment 1b coincides with the line segment 1i1j , or moved close to or away from the screen to change its size , Rotation angle, separation distance, distance moved forward and backward, left and right, etc. are used as conversion coefficients. In this case, conversion with the same conversion coefficient is considered to be the same conversion for the line segment 1d described on the same glass. The degree of matching is evaluated by how much the mapping of the line segment 1d matches the line segment 1j1k . Even in such a method, the evaluation is performed after mapping according to some rule, but the index of matching may be a correlation coefficient converted into two-dimensional image data, or (D) in FIG. In the case of f), the length deviation δr and the angle deviation δθ for matching the mapping of the line segment 1d to the line segment 1j1k are obtained, and the degree of matching may be processed as the deviation becomes smaller. Here, δr may be used without using δθ. In addition, when there are a plurality of complicated line segments and both line segments to be matched are similarly complex and there are a plurality of line segments, the mapping point of the feature point 1e is set to the point 1k and the point is moved forward. If there are connected line segments, δr and δθ are obtained in the same way, and δr and δθ of the deviation amount are obtained in the same way, and matching is performed according to the sum of δr and the sum of δθ. Good as a degree. In addition, when a branch is made in the middle, matching is performed with the same possible combination, and the corresponding line segment having the highest degree of matching is obtained.

  When calculating the conversion coefficient, the conversion coefficient may be calculated so that one line segment 1b is first aligned. However, since the point 1c does not always correspond to the point 1j, the connected line segments It is better to obtain a conversion coefficient to be converted so that the whole matches best.

  Thus, the corresponding point candidates for the feature point 1a are considerably narrowed down. What should be noted here is that, for example, even if the corresponding point candidate of the feature point 1a is known as 1n, the corresponding point of the point 1c is not necessarily the point 1q, so the same processing is performed in the image (A). If there are many other feature points 1c, 1e and others, the process is performed for all the points. As a result, when more than one candidate remains, the tendency matches by comparing the tendency of the transformation coefficient of the feature point having a high degree of matching with the absence of the plurality of candidates and the tendency of each of the remaining candidate transformation coefficients. It is better to narrow down further than the degree.

  Various other matching methods can be considered as a method for narrowing down the corresponding point candidates, but priorities are assigned and the corresponding point candidates are narrowed down based on the respective rules. Therefore, there may be a case where a plurality of candidates are inevitably left at the end, but that is not a problem if the following processing is performed. That is, in this scene, the feature point passes without being converted to a three-dimensional point. Later, when the imaging devices 10 and 20 move to change the viewpoint of shooting, a plurality of candidate remaining feature points can be narrowed down to one candidate.

  In addition, a plurality of candidate feature points may be generated depending on a subject to be photographed that is not a corresponding point but is erroneously three-dimensionally converted. This is because there is actually a positional relationship that looks like that from the shooting position at that time, and the illusion point may be detected in the same manner as a human being in the image information processing system. This correspondence does not mean that a point that is stably correlated and measured at one point of view is still a point that constitutes a stable actual line, but the point of view moves next and the same in another scene. Since it is more likely that a feature point or line segment is a feature point or line segment that is measured by measuring points and the corresponding points exist stably multiple times, the point of view It is better to manage the possibility of being an actual line as an index by using the number of times that can be measured stably as an index.

  These can be achieved by generating 3D CAD data while moving the imaging device and generating additional data, or if there is a function to update CAD data when the accuracy is high when shooting from a closer location, By applying the first to third means for increasing the accuracy of CAD data by approaching 901 from various angles or closer, it can be realized more effectively.

  Next, an example of a stereo measurement processing method will be described with reference to FIGS. Here, FIG. 43 is viewed as the O1 viewpoint coordinate system and the O2 viewpoint coordinate system of each of the two imaging devices 10 and 20 that perform stereo measurement. Although it has been described that the positional relationship between the two imaging devices is first obtained with high accuracy, as one method therefor, a plurality of corresponding points, here, three points P1, P2, and P3 will be described as an example. Then, the operator specifies in advance the correspondence between the point viewed from the O1 viewpoint coordinate system and the point viewed from the O2 coordinate system, and the relational expression of the coordinate conversion between the O1 viewpoint coordinate system and the O2 viewpoint coordinate system is obtained from the conditions. It may be possible to obtain the relational expression with a certain degree of accuracy using a known positional relationship condition, and if the operator specifies at least 8 corresponding points instead of 3 points, the positional relationship may not be known at all. An expression can be obtained. Regarding the 8-point method, there are documents such as "Information Update Society Research Report Computer Vision and Image Media, No. 131-022, 2001," Updating Covariance Matrix by Geometric Constraints ". Detailed description is omitted. Regardless of which method is used, the specified corresponding points, such as the points P1, P2, and P3, are determined so that the corresponding points correspond to each other. Therefore, the O1 viewpoint coordinate system and the O2 viewpoint coordinate system completely match each other. The correspondence is taken. Considering, for example, the case where the point P4 not used when obtaining the relational expression is obtained by stereo measurement from the relational expression obtained in advance in this way, the points P1, P2, and P3 are respectively extended from the O1 viewpoint. Although the war and the extension line from the O2 viewpoint intersect, the point P4 does not necessarily intersect. This is because there are various error factors.

  FIG. 44 is an enlarged view of the vicinity of P4 in FIG. 43, but the intersection of the line segment L1 and the line segment L2 does not exist due to an error. In this case, the sphere in contact with both line segments is considered, the sphere having the smallest diameter is obtained, and the center point of the sphere is obtained as the coordinates of P4. Then, for example, the radius r of the sphere is considered as the error radius of the P4 point.

  44, the sphere P4 is in contact with the line segment L2 and P4atL2, and the line segment L1 is in contact with P4atL1. Here, the error radius r is caused by the influence of the distortion of the image and the accuracy of the calculation formula of the coordinate conversion indicating the positional relationship between the two cameras that perform stereo measurement. The accuracy of the calculation formula for the coordinate conversion indicating the positional relationship of the camera can be made as close as possible to zero as long as it is corrected. Other errors include the resolution of the camera, which is a measurement error that cannot be avoided from the distance between the object 901 and the imaging devices 10 and 20 and the positional relationship between the two. Furthermore, as another error, this is one of the important points in the present invention, but includes an error that affects the correspondence accuracy of corresponding points. If you are looking at the boundary line that is actually painted on the subject, it will be an extremely stable actual line in normal operation without microscope photography, but the line segment that detected the contour of the object. In this case, even if the matching point candidate that most closely matches is selected due to the change in the shooting direction, the same shooting target site is not strictly observed, so the deviation is slightly deviated from the true corresponding point. This error radius r occurs. The image distortion error and stereo measurement coordinate conversion error are sufficiently reduced to within one pixel, and the errors related to the shooting distance and resolution are determined from the imaging device specifications and the approximate distance information of the P4 point. Error radius due to the line. This error also increases when measuring a sphere or a cylindrical surface, and the error of a side such as a rectangular parallelepiped composed of acute sides is reduced. Therefore, this error is managed as an index of the stability of the point. For the distinction between the contour line and the actual line, the degree of which line belongs to the error radius may be expressed. Whether or not to update the CAD data is preferably determined comprehensively including whether or not the error relating to the shooting distance and resolution is reduced and the stability is improved. Furthermore, since conversion errors are also added when the measurement coordinates are further converted to the original coordinate system by performing tracking, which will be described later, it is determined whether or not to update by these comprehensive determinations. Good to do. When tracking is performed, at least three corresponding points are required. Even when these three points are selected, if three points with good stability are selected, errors due to coordinate conversion by tracking can be suppressed to a small level. Is possible.

  Incidentally, in stereo measurement, a point with a small error radius r can be said to be a more stable point, and such points are selected as three points during tracking. A sharp contour point has a smaller error radius r, and an actual radius point has the smallest error radius.

  FIG. 45 shows the concept of error P4 (r1) related to the shooting distance and resolution and error radius P4 (r2) due to the contour line. There are errors related to the shooting distance, the resolution of the imaging device, and the shooting relationship (interval) of the imaging device at the point P4atL1 and the point P4atL2 in FIG. 44. The center of one sphere P4 (r2) is represented by a sphere P4 (r1) having an error radius r1 due to the outline. In this case, when the error becomes small, the CAD data is updated. However, in order to simplify the processing, a sphere P4 (r3) that envelops the sphere P4 (r1) and the two spheres P4 (r2) is considered. If the sphere P4 (r3) of the newly measured point is in the current sphere P4 (r3) and becomes smaller, the CAD data of that point may be updated.

  As shown in FIG. 46, the error sphere P4 (r2) related to the shooting distance, the resolution of the imaging device, and the shooting relationship (interval) of the imaging device has a narrow interval between the imaging devices as in the case of stereo measurement. When shooting, the error in the depth direction is larger than the error in the vertical and horizontal directions relative to the shooting direction. The shape P4 (δ) of the portion where the cone or pyramid intersects corresponding to one pixel subtracted from O1 and O2 may be managed as an error. In this case, the position of the viewpoints O1 and O2 is changed by the movement of the imaging device, so that the portion where the depth direction has been measured can be measured in the vertical and horizontal directions, or tracking is performed. This is effective because the distance between the viewpoints O1 and O2 may be further increased and the measurement may be performed with higher accuracy. The shape of the error shape P4 (δ) is not reduced as a whole, but a part is updated according to the direction and direction, and the direction in which the previous measurement part is highly accurate uses the error of the previous measurement part. When the volume of the error shape and the longest part are reduced, the error shape may be managed while being updated as a new error shape by updating the combination of the previous and current error shapes. For example, if the previously measured error in the depth direction is reduced by measuring from the side, the previously measured error in the depth direction is reduced to be the newly measured left and right errors, and the left and right direction is The direction measured with high accuracy in this case is now the depth direction, so that the direction is not updated using the previous error shape. In this way, the shape of the error that is generated first is a shape such as P4 (δ), but the error is made into a shape by reducing the shape for each part where the accuracy is improved by the update process. The CAD data may be strictly updated while updating.

  FIG. 47 shows an error shape P4 (δ) of the point P4 that is first generated by the stereo camera from the two imaging devices of the viewpoints O1 and O2. Since the error in the depth direction is large, writing a circumscribed circle results in a sphere of P4 (r2).

  Next, when the viewpoints O1 and O2, which are stereo cameras, are moved and the same point P4 is measured as shown in FIG. 48, the shooting distance from the viewpoints O1 and O2 to the point P4 is also close. Although the error spread is reduced, the error shape P4 (δ) in FIG. 47 is taken from an oblique direction, so that the direction with poor accuracy in the previous depth direction can be measured with the accuracy in the vertical and horizontal directions this time. This error shape P4 (δ) can be managed with a smaller error shape P4 ′ (δ). The circumscribed circle is also reduced to P4 '(r2). When a new error shape is obtained, the current error shape and the newly measured error shape may be overlapped and extracted. If the overlapped portion is the entire previous error shape, the measurement coordinate value at that point is not updated because the current measurement error is large. On the other hand, when all newly measured error shapes are included, the measured coordinate values are updated to the newly measured. Further, when there are overlapping portions in the previous and current error shapes, the overlapping portion is updated to a new error shape. In that case, the measurement coordinate value is also updated, but the update method is to update the center of the newly obtained error shape (the center of gravity when the density of the solid figure is uniform) as the point measurement coordinate value. Also good. In this case, the coordinate value of the center of the error shape may be used as the measurement coordinate value from the first measurement stage. In this way, when managing the error magnitude of the measured point while strictly determining the error shape in a shape that also considers the direction, the error due to the contour line and the error caused by tracking etc. are also considered the directionality You may make it manage with the exact | strict shape. In addition, when the error shape no longer depends on the direction to some extent, it may be managed as a sphere for simplification, and may be managed again with the error shape when there is a difference in directionality. . Further, the error may be simplified to some extent by replacing the directional error with a basic figure such as a rectangular parallelepiped or an elliptical sphere.

  As a basic figure, as shown in FIG. 49, an error can be expressed by approximating a sphere having an error radius δR in the direction connecting the PC point to P4 so that it is between P4 and δL. The information on the shape error to be attached to the 3D point P4 can be expressed by five variables of the radius δR, the length δL, and the coordinates x, y, and z of the Pc point. Is. When the error shape is updated in accordance with the update of the coordinates of the point P4, the radius δR, the length δL, and Pc are approximated to the radius of the sphere and the length in which the spheres are arranged without using a complicated shape. What is necessary is just to update five variables of the coordinate x, y, z of a point. Of course, in the approximation, the radius δR and the length δL are determined so that the approximate error shape is not too smaller than the exact error shape.

  Next, an example of a processing method when the imaging apparatus moves and the viewpoint changes will be described with reference to FIG. In FIG. 43, the O1 viewpoint coordinate system indicates the coordinate system of any one of the imaging devices 10 and 20, and shows a situation in which the imaging device has moved from this position to the O2 viewpoint position. And Here, it is assumed that the imaging device is the left imaging device 10. When the left imaging device 10 is at the O1 viewpoint, if the stereo measurement is performed with the two cameras using the method described above, the feature points in the photographing target are measured as three-dimensional point coordinates in the O1 viewpoint coordinate system. Has been. Actually, many feature points are measured, but the corresponding points are the same among them, so that the measurement conditions are good, the most stable, and the most accurate, select three points. To do. In addition, when selecting three points, selecting three points that are close to each other because the accuracy is good also deteriorates the conversion accuracy, so select points that are stable and accurately measured at moderately separated points. Like that. If there are only 3 feature points that can be measured, you can only select those 3 points, but if at least 3 points are measured and taken from the O2 viewpoint, tracking by image information will continue if those 3 points have moved. it can. In other words, if three points cannot be obtained, tracking by image information cannot be performed. In this case, wait until at least the corresponding three points move, or another sensor such as an acceleration sensor or gyroscope. A sensor, a magnetic sensor, or the like may be attached to the imaging apparatus, and the camera position when tracking cannot be performed by image information may be interpolated. If feature point information on three or more image images extracted at the beginning is recorded and saved without being erased, the imaging device returns to the vicinity where it was previously taken, even if tracking by image information is not possible during the process. In such a case, the feature point information on the previously saved image image may be used to resume the tracking process from there.

  An example of basic tracking processing will be described. If tracking is performed so that points P1, P2, and P3 measured in stereo from the O1 viewpoint are not lost in high-speed processing on a two-dimensional image image, the imaging apparatus 10a is in the O2 viewpoint. When moving to a position, the points P1, P2, and P3 can be obtained as coordinate data of the O2 viewpoint coordinate system by stereo measurement. Since the three-dimensional coordinates in the respective coordinate systems of the corresponding three points P1, P2, and P3 in the two coordinate systems are obtained, if a relational expression of coordinate transformation between the O1 viewpoint coordinate system and the O2 viewpoint coordinate system is obtained, 2 If, for example, the point P4 that could not be tracked on the three-dimensional image or newly entered the field of view of the imaging apparatus is seen, the coordinates of the O2 viewpoint coordinate system are obtained by stereo measurement, and the coordinates are obtained from the O2 viewpoint coordinate system. It can be converted into coordinate values in the viewpoint coordinate system. In this way, if tracking is continued and at least stable tracking of three points can be performed, three-dimensional CAD data converted into the same coordinate system at the first start can be obtained for all measurement points.

  Next, with reference to FIG. 50, the same camera movement when there are three known points in the first viewpoint O1 coordinate system of the single-eye camera system and the compound-eye stereo camera system. A method of obtaining a calculation formula for coordinate transformation between the O1 coordinate system and the O2 coordinate system when the later viewpoint O2 coordinate system is unknown, that is, a tracking method for determining the position and orientation of the O2 coordinate system in the O1 coordinate system will be described. . The condition here is that the three-dimensional coordinates of P1, P2, and P3 in the O1 coordinate system are known, and the points P1, P2, and P3 appearing in the camera image at the O1 viewpoint position by continuous processing on the two-dimensional image image. And points P1, P2, and P3 appearing in the image of the camera at the position of the O2 viewpoint are in correspondence.

  Since the constant of the calculation formula for the coordinate conversion between the O1 coordinate system and the O2 coordinate system is unknown, it is replaced with an unknown variable, and all the points of the O2 viewpoint coordinate system are replaced with the O1 viewpoint coordinate system. The extension line of the line connecting the point P1 on the xy plane (two-dimensional plane corresponding to the image) of the viewpoint O1 and the viewpoint O1 coordinate system, and the xy plane of the viewpoint O2 and viewpoint O2 coordinate system (two-dimensional is the image) The condition that the extension line of the line connecting the point P1 on the corresponding plane) intersects at the point P1 in the three-dimensional space of the O1 coordinate system, the xy plane of the viewpoint O1 and the viewpoint O1 coordinate system (two-dimensional corresponds to the image) The extension line of the line connecting the point P2 on the plane and the extension line of the line connecting the point P2 on the xy plane (two-dimensional plane corresponding to the image) of the viewpoint O2 and the viewpoint O2 coordinate system is O1. The condition that the point P2 in the three-dimensional space of the coordinate system intersects, the extension line of the line connecting the point O3 and the point P3 on the xy plane (two-dimensional plane corresponding to the image) of the viewpoint O1 coordinate system, and the viewpoint O2 The point P3 on the xy plane (two-dimensional plane corresponding to the image) of the viewpoint O2 coordinate system is connected. By solving the simultaneous equations from the condition that the extension line of the minute intersects with the point P3 in the three-dimensional space of the O1 coordinate system, the constants of the calculation formulas for the coordinate transformation of the O1 coordinate system and the O2 coordinate system that were initially set as unknown Can be sought. As a result, the positional relationship between the O1 coordinate system and the O2 coordinate system can be obtained. Therefore, in the case of a stereo measurement system, other three-dimensional point data that can be measured in the O2 coordinate system can be converted into three-dimensional point data in the O1 coordinate system. Also, in stereo measurement systems, the distance between the left and right cameras is narrow, so the measurement accuracy when measuring far away in stereo does not improve, so either or both of the left and right cameras of the stereo camera system support at least three points. If the tracking point can be obtained by the tracking process, the relationship between the two camera positions separated by the method explained here can be obtained, so when using the condition, that is, when shooting at the position before the camera moves greatly Compared to the case where the left and right cameras are used for stereo measurement, using the extracted feature points and the corresponding feature point candidates extracted from the currently captured image information where the camera has moved to a position far away from the camera. It is possible to obtain CAD data with better measurement accuracy of a far object. Further, in the case of a monocular system, the calculation formula for coordinate conversion is always calculated from the corresponding points of 3 points from the image information of the first camera position and the image information from the camera at a distant position after the camera has sufficiently moved. 3D CAD data can be generated by a method similar to the stereo measurement described above with reference to the positional relationship information between the cameras.

  Here, in the case of a monocular camera system, three known points are required first, and a method in that case will be described with reference to FIG.

  The imaging device 10 is initially installed on a base that serves as a reference. At this time, it is preferable to attach metal fittings such as stamps on the base side and the imaging device side so that the imaging device 10 can be accurately reproduced regardless of how many times it is attached to and detached from the base. Then, the mark 6a is installed so that the relative position relationship with the base does not change steadily so as to enter the field of view at the start. The reference mark 6a may be a cross as shown in the figure, but at least three points, in this case, for example, when the imaging device 10 is placed on the base of the three points P1, P2, P3 at the corners of the reference mark 6a. It is also possible to obtain the coordinates with high precision in the camera coordinate system, and to start tracking, 3D measurement, and CAD data generation by the three-point method using the coordinate values. Every time you start from the same base. When this is applied to inspection / inspection equipment, etc., the same part is regularly inspected. Therefore, the reference base and the reference mark are installed in advance, or the reference base and the reference mark are integrated. The camera stand jig may be installed and operated at the place where the camera stand jig has been entered and marked on the site. When permanent installation is difficult, it is an effective means. Even if the accuracy of installation at the site marking site is somewhat poor, the generated CAD data is only relatively different from the previous implementation, so that level of installation is sufficient for operation as inspection work. .

  In addition, since there is a method that does not use a reference mark in a monocular system, contents related to the method will be described here.

  In the above-described 8-point method, if the corresponding points can be tracked on the two-dimensional image image with respect to 8 points or more, the calculation formula for the coordinate transformation of the viewpoint coordinate system of the imaging device far away is determined using the corresponding 8 points. Therefore, if the 8-point method is used, it is possible to construct a system that can start even with a monocular system or without first capturing a reference mark. When the 8-point method is used, it is important how to select stable 8 points. Therefore, when a plurality of corresponding points can be tracked by two-dimensional image processing, a plurality of those points are selected. Select 8 points from the point group that can be tracked, obtain a calculation formula for coordinate conversion using the 8 points, and use as an index the number of points that match accurately when other points other than 8 points are converted using the calculation formula. In the case of all other 8 point combinations, the index is obtained in the same way, and the 8 points used when obtaining the calculation formula when there are many other points with the most accurate match are the most stable and accurate. By taking the points, the coordinate transformation formula obtained at the 8 points is used. In this way, there is also a method of measuring by the 8-point method while always tracking, but it is necessary to obtain calculation formulas for the number of combinations of 8 points and repeat the evaluation, so the calculation speed is sufficiently high. The system may be configured to operate by switching to the above-described three-point method when the stable three points are obtained by measuring by the 8-point method only at the beginning. In this case, the operation may be devised so that the imaging apparatus is moved slowly at first in relation to the processing speed.

  In order to always keep 8 corresponding points on the screen, if the field of view in the image image is changed greatly, the corresponding 8 points cannot be obtained or concentrated on only one side of the screen (concentration calculation on one side is calculated) It is better to switch to the three-point method because there is a merit that it is easier to secure the corresponding points.

  As with stereo measurement or monocular system, tracking can be continued by image processing if at least 3 points can be tracked. However, if the stable point falls below 2 points, the acceleration sensor Alternatively, the camera movement position may be interpolated by a signal from a gyro sensor or the like. When resuming, in the case of a stereo camera system, a 3D point can be measured from the beginning. The position of the imaging device that has been lost due to matching with the group can be found and resumed continuously. At that time, if there is movement information of the imaging device in the period that has been missed by the acceleration sensor or gyro sensor, the approximate position can be estimated. It is easy to find the position accurately and resume. In the case of a monocular system, when resuming after losing 3 points, it is necessary to start again with the 8-point method.

  Next, a method of narrowing down corresponding feature point candidates using color information and other rules on the two-dimensional image information in the case of stereo measurement will be described with reference to FIG. In FIG. 52, (a) and (b) are left and right image images being stereo-measured. The feature point 7a of (b) exists as the corresponding point candidate 7a on the straight line L1 of the image image of (a). Here, it is determined whether or not the corresponding point candidate is correct based on the color information around the feature point 7a and the corresponding point candidate 7a. In this case, the color information of the feature point 7a and the corresponding point candidate 7a is good. We use it as a corresponding point candidate. There may be a case where a more appropriate corresponding point exists outside the image image (a). In this case, the point corresponding to the feature point 7a when the imaging device moves and is measured from another viewpoint. Such an illusion point can be easily deleted by confirming that measurement can be performed again from another viewpoint and then converting it to normal CAD data.

  Specifically, a flag of the first time is provided in the measured point data at the first one measurement stage, and it may be determined strictly from another viewpoint, or in some cases from three viewpoints. Good, but reset the flag if you can measure more than once. Of course, it is possible to make it possible to confirm and display the point of only one measurement, but to manage the data separately. In addition, a measurement error due to the deviation of the corresponding point is managed as a contour line, and a measurement error with few errors is managed as an actual line. By photographing the contour line and the actual line from various viewpoints and repeating the measurement, it may have been the contour line if you first looked at the stable points that seemed to be the actual line. Instead of distinguishing between 0 and 1, ranking may be made based on the magnitude of the measurement error due to the shift of the corresponding points. If the actual line is measured from many different points of view with few errors in the corresponding points, there is a high possibility that the actual line is a stable line segment, so information such as the number of measurements is also managed as point data. Is good. The points used for tracking etc. should be real lines that are as stable as possible, so it is better to manage whether the lines or feature points can be easily identified. .

  (C) and (d) in FIG. 52 are examples of a scene in which the same left and right image images are added, but one rectangular parallelepiped is added to the shooting target. A feature point candidate in the image image (c) corresponding to the feature point 7b in the image image (d) exists on the L2 line and there is a corresponding point candidate 7b. Here, when the color information around the feature point 7b and the corresponding point candidate 7b is compared, there are three color boundary lines connected from the point 7b, although the color 1 and the color 2 are different. Since there are no line segments with different colors, it can be considered as a corresponding point candidate.

  Next, in the cases of (e) and (f) in FIG. 7, in this case, the point in the image image (e) corresponding to the feature point 7c of the image image (f) is on the line segment L3. There are two corresponding point candidates 7c and 7d. In the process in such a case, the left and right color information of the line segment connected from the point is similarly compared. When compared in this case, three line segments 7i, 7j, and 7k are connected from the feature point 7c of (f), and line segments 7g and 7h are connected point candidates from the corresponding point candidate 7c of (e). Line segments 7e and 7f are connected to 7d. When comparing color information, compare the right and the left toward the point of the color boundary line connected to the point, and see whether the right matches the right and the left matches the left. . Then, the line segment 7k and the line segment 7h coincide on both sides, the line segment 7j and the line segment 7g coincide on one side, and the line segment 7i and the line segment 7e coincide on one side. Lines 7i and 7g, and lines 7f and 7k have the same color on one side, but these have a large angle that must be rotated when they are overlapped. In addition, processing that excludes a predetermined rotation angle that cannot be considered in normal stereo measurement may be included. Even if the determination is made on that basis, the two corresponding point candidates 7c and 7d are connected to the line segments having the very same color information on at least one side. Here, since these points cannot be narrowed down, the processing may not be performed. In this case, if the corresponding point really exists, it can be measured at that time when the imaging device moves next and measures from another viewpoint. Further, as a further narrowing-down process, the corresponding point candidate 7d is one line segment in the color information comparison result of the line segments at the corresponding point candidates 7c and 7d, but the corresponding point candidate 7c includes two lines 7h and 7g. Since the corresponding color boundary line segment is used, the number of the line segments may be used as an index for determination, and in this case, the corresponding point candidates may be narrowed down to 7c. In the case of the corresponding point candidate 7c, it remains as a candidate here, and can be stereo-measured and converted into CAD data of a three-dimensional point. It can be identified that such a point is in a state where the first flag is not erased until is moved and measured in the same manner. If the point that has been measured once does not exist even though it should have been measured around the measured point, the point data may be deleted.

  Actually, if a complete corresponding point determination is to be performed, various determinations are made and the processing time becomes longer. Therefore, a simplified determination process may be performed. In addition, since it takes a long time to process 100% of the corresponding points so that there is no mistake, if 3D measurement is performed at the wrong corresponding points, there is an error that does not actually exist in the three-dimensional space. 3D point data is generated, but it is a 3D point measured at an incorrect corresponding point by checking whether the same point is generated at the same place continuously while the viewpoint moves. Since this can be discriminated, a process for canceling a 3D measurement point that has been stereo-measured at an incorrect corresponding point may be inserted using the relationship.

  As another method of discrimination, if the corresponding points of the left and right images subjected to stereo measurement are tracked in the left and right two-dimensional images, and stereo measurement is performed again from the same corresponding points from different viewpoint positions, For example, the corresponding point of the left image does not ride on the line segment L3 on the left image due to the corresponding point of the right image such as L3 in FIG. Since it is possible to determine that the corresponding point is not a corresponding point by changing the viewpoint, a process of canceling a 3D measurement point that has been stereo-measured at an incorrect corresponding point using the relationship It is also good.

  Here, the color information of the boundary line segment is compared. However, when the sensitivity is lowered and the amount of the extracted line segment is reduced, a macro technique is also described. That is, instead of comparing the left and right color information in the line segment, the color boundary line segment is gathered radially at the feature points, so a two-dimensional image matching (correlation) is assumed assuming a slightly larger area. But it ’s okay. In this case, since the region spreads radially from the feature point, the region becomes a fan shape, a triangle, or a quadrangle. The radius may be determined in advance as a predetermined minute area. However, since the opening angle of the radiation of the corresponding fan is different from the opening angle of the color boundary line of the corresponding candidate, the fan, triangle, and minute are processed after deforming to the same shape and making it the same shape and size. Two-dimensional correlation matching is performed in a square region or the like.

  Next, an example of a method for narrowing down corresponding point candidates in the case where tracking is performed using color information of color boundary lines on a two-dimensional image will be described with reference to FIGS.

  53A shows an image image before movement of the same camera, and FIG. 53B shows an image image after movement. Here, focusing on the feature point 8c in the image (A), the corresponding point candidate is searched and narrowed down in the image (B). Color boundary lines 8b and 8d are connected to the feature point 8c. Assuming that the image image (A) is taken and then the image image (B) is taken at a predetermined time interval, the image pickup apparatus moves during the time interval, so that the corresponding point candidate of the feature point 8c is obtained. The amount of movement is estimated. Considering the shooting time interval determined from the relationship of processing speed and the maximum moving speed, rotation speed, etc. on the application system of the imaging device, the amount of movement in the image image within a predetermined time can be estimated. The inside of the movement amount is considered as the search range for corresponding point candidates. In this embodiment, a circle having a radius of d / 2 may be considered around the feature point 8c, and the search may be performed in the circle, or the area may be a square having one side length d instead of a circle. The length d is the size of the search range determined in consideration of the moving speed and processing speed of the imaging device. Naturally, the higher the processing speed, the smaller the search area, so the processing speed should be as fast as possible. Further, in a special case, the minute area becomes large, and the maximum can be implemented by the method described here even when the corresponding point candidates are narrowed down from the entire screen area.

  Here, an example is shown in which three points 8f, 8g, and 8h exist as feature point candidate points in the search area in the image (B). If a candidate in the area is searched, the next process is to leave as a corresponding point candidate that the color information on both sides of the color boundary line segment matches at least one side with high accuracy. This is because, in the case of stereo measurement, instead of searching for candidates from the line, tracking is performed by searching from within a very small area. .

  In FIG. 53 (a), the feature point 8c is superimposed on the corresponding point candidate 8f, and the left and right color information of the color boundary line segments connected to these points are compared. In this case, three line segments are connected to the corresponding point candidate 8f and two line segments are connected to the feature point 8c. Since minutes are not always extracted, there is no need to worry about this process.

  Similarly, in (b), the feature point 8c is superimposed on the corresponding point candidate 8h, and the colors on both sides of the color boundary line segment connected from there are compared.

  Similarly, in (c), the feature point 8c is superimposed on the corresponding point candidate 8g, and the color information on both sides of the color boundary line connected from there is compared. When at least one side of the left and right color information matches with the same color with high accuracy, the line segment is a corresponding line segment candidate. Then, the next narrowing process is performed.

The following processing is the same as that applied to stereo measurement, but when one color boundary line segment is combined, it is narrowed down by an index indicating how much the same conversion result of other points connected to that line segment is shifted. You may do it. When a conversion equation is obtained to match the feature point 8a of FIG. 54A with the corresponding point candidate 9b and the line segment 8a8c with the line segment 9b9c , and the same conversion is performed on the line segment 8c8e , the mapping line segment is obtained. And the line segment 9c9d are measured by using an index such as an error in the rotation amount of δθ and an error in the length of δr.

(B) in FIG. 54, so as to match the characteristic points 8a on the corresponding point 9f, when deformed to match likewise the segment 8a8c the line 9F9g, and mapping segment 8c8e and the line segment 9g9h In this example, an error in the rotation amount of δθ and an error in the length of δr are obtained. The index is narrowed down according to the size of the index, and the next narrowing-down process is performed by selecting a line having a high matching property as a line matching degree. Then, in the case of tracking, the rotation also becomes large, but the whole is basically deformed in the same way, so that the difference in the degree of coincidence of the whole tendency is excluded depending on whether the conversion formula is similar or not. Process. In particular, processing is performed such that a corresponding point candidate having a conversion formula close to a conversion formula having a high matching degree is left as a corresponding point candidate.

  As described above, various methods of narrowing down are conceivable. However, it is only necessary to determine rules (processing contents and priority) in advance and narrow down until narrowing down. When a plurality of candidates remain without being able to be narrowed down at the end, in the case of tracking, the following processing is also possible. Even if there are a plurality of remaining points, if three corresponding points that match well are found, tracking may be performed using the three points. If there are more than 3 points, select 3 points from multiple corresponding point candidates, find the calculation formula for the coordinate conversion of those 3 points, and map other points using the calculation formula other than 3 points The number of points that match with accuracy is used as an index, and tracking is performed using the three best points of the index. At this time, a plurality of corresponding point candidates of one feature point cannot be narrowed down and remain. In this case, a plurality of corresponding point candidates that remain may be included in the combination selection of three points, and the index may be examined. This is especially true when there are few feature points to be extracted, and when there are few corresponding point candidates as a whole, there is a case where there is a real highly accurate corresponding point while one corresponding point cannot be narrowed down. Because there is, it becomes effective processing.

  In addition, using FIG. 55 to FIG. 57, not only the color information of the color boundary line segment but also the color information of the surface constituting the color boundary line segment is actively used to narrow down corresponding point candidates or corresponding surface candidates. An example of the method to perform is described. Such a narrowing-down method is on the line segment in the case of stereo measurement, and in the estimation area in the case of tracking search, but can be applied in the same manner. Also, in narrowing down only the color boundary line segment, the subtractive color boundary line is increased by reducing the line segment extraction sensitivity and extracting only large change points and searching for corresponding points and increasing the line segment extraction sensitivity. It is also possible to perform processing by combining micro-matching that tries to narrow down the corresponding points of the line segment, but the method described here is, for example, an object to be photographed that has multiple similar patterns This method is also effective when you want to associate which pattern in the image taken from a different viewpoint with the corresponding pattern in the image. This is an example in the case where color boundary line matching is performed for microscopic matching. In this method, the color boundary line segment constitutes surface data, and even when the color boundary line segment is closed loop, i.e., does not constitute a surface, all color boundary line segments constitute surface data, or By managing CAD data using the relationship that is attached to surface data, it is possible to refer to which line segment is attached to which surface at any time during processing. From the macro matching by the micro to the micro matching by the line segment, it is possible to easily use the macro narrowing information to connect to the micro matching processing.

FIG. 55 shows a part of a line segment obtained as a result of extracting a color boundary line segment of a certain image. Color boundary lines are extracted from many feature points. For example, the points 9i and 9j are feature points, and the line segment 9i9j is one of color boundary line segments.

  The line segment extraction method in FIG. 55 will be described in detail later with reference to FIG. 68. However, a method for extracting an inflection point of a color that can extract the color boundary line segment most accurately and with high sensitivity is used. Use. This is because micro-matching obtains corresponding points with higher accuracy, and the line segments used for the matching should be extracted more accurately. However, if this method is used, line segments that do not constitute a closed loop are also extracted because of high sensitivity and accuracy. The portion between the end points 9p and 9q is a portion where the color is gradually changing like gradient, and is not extracted as a color inflection point. In order to perform macro matching by surface, it is necessary to extract the surface.

  56 (a) to 56 (d) are diagrams for explaining some examples of color surface extraction methods. The example of FIG. 56A is an example in which the surface 9A which is a closed loop is extracted by a method of extracting contour lines of, for example, the average brightness level of the end points 9p and 9q of the color line segment which is not in the closed loop. It is. When the luminance of the end point 9p and the end point 9q does not match the average luminance, a surface 9A that is a closed loop that does not pass through the end point 9p and the end point 9q is extracted from the contour line.

FIG. 56B shows an example in which the surface 9A that is a closed loop is extracted by a method of extracting contour lines from the luminance values of the color boundary of the color boundary line segment 9n9p . In this case, the contour line of the surface 9A is an example that coincides with the color boundary line segment 9n9p . However, this is an example in which the other color boundary line segment extracted as a contour line which is a closed loop but inferior in strictness does not match the other color boundary line segment which is strictly extracted.

The example of (c) of FIG. 56 is an example in the case of processing so as to connect the end points 9p and 9q that are not in the closed loop by making use of the strictly extracted color boundary line as it is. In order to distinguish the line segment 9n9p that overlaps the end points 9p and 9q from the color boundary line segment that is strictly extracted, the line segment data is managed by distinguishing the type as a virtual line segment for closed loop. . There are various methods for connecting the virtual line segments in order to generate a closed loop. Here, an example will be described. Since there are many combinations of how a closed loop can be constructed, an appropriate virtual line segment is obtained while determining whether or not a nearby feature point can be connected from each feature point. For example, a closed loop can be generated by connecting the feature points 9m and 9n. In this case, for example, the left and right normal directions of the virtual line segment 9m9n are examined, and the average color until the nearest boundary line segment is obtained on the left and right sides. , if the average color of the left and right are similar since it is found that it is about Musubo virtual line 9m9n in the plane of the same color, in this case is a method such as to prevent tied virtual line 9m9n . In this way, by performing various combinations and drawing the most different left and right average color portions as virtual line segments, virtual boundary line segments can be prevented from being drawn in a closed loop of the same color. With this method, it is possible to divide all areas into plane data so that the exact color boundary line segment and the boundary line segment of the plane data coincide.

  FIG. 56D shows another example in which a surface is extracted by simply drawing a contour line of a similar color portion at a predetermined luminance level. The exact color boundary and the boundary of the surface data do not match. In this case, since all the extracted color boundary lines exist in any one of the surfaces, the surface of the same color and the same shape is first selected by macro matching on the surface using the relationship. After narrowing down the corresponding surface in this way, it is possible to narrow down the corresponding line segment and corresponding point of the color boundary line segment that is strictly extracted by micro matching in the limited area of the corresponding surface. can do. This is a method for obtaining a precise visual information processing system by extracting strict color boundary lines. Of course, a simpler visual information processing system has a strictly extracted color boundary line. Instead of using minutes, all line segments may be extracted with contour lines of color boundaries, and processing may be performed only with the line segments constituting the closed loop surface. In this case, the accuracy of the generated three-dimensional CAD data is also appropriate.

  Next, with reference to FIGS. 57A to 57F, an example of a method for narrowing down corresponding surfaces, corresponding line segments, and corresponding points when a plurality of the same patterns exist in one image image will be described. .

  FIG. 57A shows a part of an image image obtained by extracting a closed loop of color boundaries of image information taken from a certain viewpoint. Here, the extraction method shows a state in which closed-loop surface data is generated using a strict color boundary line by the method of FIG. 56C described above. FIG. 57 (c) shows a corresponding part of an image image obtained by performing a similar process by moving the viewpoint a little, but the image image is deformed as seen from an angle because the viewpoint has moved a little. Is shown.

  Here, let us consider a case where corresponding points of the color boundary feature line segment and feature point of the first processing result (a) are searched from the processing result (c) to narrow down correspondence candidates. This will be described in the case of searching for a corresponding surface of the surface 9A as macro matching. First, in step 1, all the surfaces 9A, 9B, 9C,..., 9L,... That are the graphic data of the processing result (c) while transforming the graphic data of the surface 9A in the processing result (a). If matching is performed with the one with the highest matching and the coefficient of the deformation processing in the matching of the other surface is the same or not, the surfaces 9A and 9G in the processing result (c) are Matching results are obtained that match very well to the same extent. Here, the transformation process is to place a figure on a transparent flat glass, hold the glass plate, tilt it back and forth, left and right, move it away, bring it closer, or rotate it so that they match. This is equivalent to adjusting the six parameters of the x, y, and z parallel movement amounts and the α, β, and γ direction cosines in a normal coordinate conversion process. When looking at the tendency of these coefficients, the tendency varies depending on the shooting distance, so if they are in the same vicinity, they tend to be the same. Note that it is not always a coefficient. Also, in order to simplify the processing, when the degree of deformation is small, that is, when the change in the two shooting viewpoints is slight, the accuracy of the corresponding point search is somewhat lowered, and the number of corresponding points obtained is reduced. In some cases, an omitted method may be used. In this step 1, two corresponding surface candidates of the surfaces 9A and 9G of the processing result (c) are obtained. This is not limited to two when a scene where a plurality of similar patterns are arranged, but a plurality of corresponding surface candidates remain.

  Therefore, step 2 is executed as the next processing. In step 1, matching processing is performed only on the surface 9A of the processing result (a), but here, matching processing is performed on a surface including the surface existing around the boundary of the surface 9A of the processing result (a). In the case of this example, there are three surfaces 9B, 9C, and 9D around the surface 9A of the processing result (a). Therefore, the matching processing is performed by combining one surface each.

  Specifically, as shown in (d), (e), and (f) of FIG. 57, the deformation process is performed and the matching process is performed. As a result, the deformation processes (d) and (f) have the same pattern configuration in the processing result (a), and thus the degree of coincidence is high. However, in the result of the deformation process (e), the processing result ( On the surface 9A side of c), the degree of matching is high, but on the surface 9G side, the surface 9C of the processing result (a) and the surface 9J of the processing result (c) have different colors and shapes. As a result, it becomes possible to narrow down the fact that the face 9A is more suitable among the face 9A and the face 9G of the two corresponding face candidates remaining in the processing result (a). . However, it should be noted here that the deformation process (d) and the deformation process (f) match each other, and strictly speaking, the surface 9G may still be a corresponding surface candidate. This is because if the boundary between the surface 9A and the surface 9C is a contour line with the background, even the corresponding surface may not match. Therefore, as a result of performing deformation processing by combining the surrounding surfaces one by one as a process, if there is even one result with a high matching degree, leave it as a corresponding point candidate, and as the next process, Matching is performed while increasing the number of surfaces to which similar processing is combined by adding adjacent neighboring surfaces. When the same pattern is densely packed, the surfaces around one surface (here, the surface 9A) are combined to perform matching processing for a plurality of wide surfaces. If there is a different pattern at the end of the image scene, it is possible to narrow down the matching surface candidates to be matched and those that are not, if the matching process is expanded to that extent. However, if all the captured image scenes are filled with the same pattern, the corresponding surface cannot be narrowed down even if this method is applied. Since this is image information that cannot theoretically be narrowed down, in such a case, the processing may not be performed. In this way, even when a plurality of the same patterns exist, the corresponding surface can be narrowed down by macro matching expanded to the non-identical range. Once the corresponding surface has been narrowed down, by performing micro-matching processing for each line segment attached to the surface within the corresponding surface, the exact color boundary line segment that extracts the detailed portion, closed loop It is possible to narrow down corresponding line segments and corresponding points even for color boundary line segments that are not formed. Here, the concept of data management that a line segment is attached to the surface is the case where 3D CAD data is generated using this method or when the accuracy is improved. This is an effective technique for making it easy to refer to the processing target surface and line segment data.

  First, before the first CAD data is generated, a virtual surface called the background is defined, and the generated line segment data and surface data are taken from the idea that the virtual surface called the background is attached to the data. Try to manage. Since the line segment becomes surface data when it becomes a closed loop, this surface data is also managed as surface data attached to the virtual surface called the background. Further, when the line segment in the surface becomes a closed loop and the surface data is generated, the surface is attached to the surface data attached to the virtual surface data called the background. In this way, the virtual background data of the background is the vertex, line data and surface data exist below it, and the line data and surface data similarly exist below each surface data. By performing the data management, it is possible to easily refer to the CAD data to be processed to be referred to. Here, the virtual line segment drawn for macro matching is a virtual line segment, and the surface data that is closed loop by the virtual line segment is still virtual surface data at that stage, so the actual line segment data and surface The CAD data corresponding to the actual scene is such that management is performed separately from the data so that the virtual line segment and the virtual plane are not displayed when the monitor is actually displayed. However, the shape data may be used if it can be used effectively in the processing of the virtual line segment or the virtual plane, so both data are managed, but it is preferable to manage them separately. In addition, as in this method, virtual line segments and virtual plane data are generated and all objects in each image scene are represented as closed-loop figures. If the processing is performed only with a line segment, the line segment is changed from a non-closed loop to a closed-loop figure, and the process when changing to a non-closed-loop line again (details will be described later with reference to FIG. 14) is not performed. May be. At this time, the processing speed of matching etc. may be slow if each surface data consists only of the line segments that make up the surface, so the centroid, area, maximum vertical / horizontal length of each closed loop figure, The matching of the closed-loop basic figure may be performed in a short time by obtaining the length, the concavo-convex direction vector, and the like.

  It is better to perform macro matching and micro matching in this way, but even if the sensitivity is increased and the color boundary line segment for micro matching is extracted, a subtle line segment is extracted, so a slight viewpoint If the moves, matching may not be possible even at the micro-matching stage. This is because the influence (noise) of the viewpoint change is larger than the color boundary line segment to be detected. In such a case, a laser beam may be projected to detect irregularities with delicate shapes by actively creating stable color boundary lines. Macro matching is equivalent to processing that lowers sensitivity, that is, rough processing at the expense of strictness, so the processing speed can be increased, so sensitivity can be changed remotely (for example, by command commands to visual information processing devices, etc.) By making it possible to specify whether the change and matching is also macro or micro, it is usually possible to generate coarse CAD data by rough macro matching and necessary when necessary It is also possible to increase the sensitivity at various points and generate detailed CAD data by micro matching. The order of macro matching and micro matching may be reversed. In other words, macro matching is not always performed from the beginning, but when multiple candidates cannot be narrowed down first by micro matching, the processing speed is increased when macro matching is unnecessary by narrowing down by macro matching. May be.

  In addition, an example of a basic method for performing a narrowing down using a relationship with surrounding feature points will be described as another macro matching method using FIGS. 58 to 66. Such a narrowing-down method can be applied in the same way for searching in the estimation area in the case of stereo measurement and in the estimation area in the case of tracking search. This method can also be applied when finding a highly correlated target portion by pattern matching of a two-dimensional pattern or a three-dimensional shape after conversion to.

  FIG. 58 shows an image obtained by converting image information input from the imaging apparatus into two-dimensional CAD data. Here, in order to facilitate understanding, an image obtained by processing extremely simple image information will be described. The entire image information is divided into eight graphics 9M, 9N, 9P, 9Q, 9R, 9S, 9T, and 9U having the color information of each closed loop by color boundary line segments. Here, the closed loop graphic is generated only by the exact color boundary line segment, but of course, a virtual line segment or a virtual plane may be defined. In addition, for the frame portion of the image information, the color boundary line segment of the surface actually outside it is still unknown, so the virtual surface is defined using the virtual line segment. For example, the surface 9M is composed of line segments L1, L2, L3, and L4, and the points 9r, 9s, and 9u are virtual points on the frame frame of the image information, and the line segments L1, L2 , L3, and L4, since at least one side of both end points is a virtual point, it is still a virtual line segment at this stage, and the surface 9M is also a virtual surface. The planes 9Q, 9R, 9S, 9T, and 9U are portions that are generated as actual plane data because all of them are included in the image frame unless virtual line segments are used in the closed loop graphic. The CAD data is managed such that the surface 9N has the surfaces 9Q, 9R, 9S, 9U and the line segment L7. Further, the surface 9T is attached to the surface 9S, and the line segment L8 is attached to the surface 9T. In addition, this relationship is basically the same when three-dimensional measurement is performed for each feature point and three-dimensional CAD data is generated as a line segment connected to the feature point or a surface formed by the line segment. In this case, as described above, the virtual surface called the background is considered to be the last. By adopting such a data structure, it becomes easy to express CAD data of soft and wrinkled objects such as flowers, plants and kimonos. Also, in the case of 3D data, it should be noted that whether a line segment or surface data is attached to a predetermined surface depends on the position information etc. of those data, or in a 3D space. It is necessary to determine whether or not the line and the surface are attached to a certain surface. In 2D CAD data, the line and surface data in front of the subsequent surface may only appear to be attached in the image, so that it is attached on the 2D CAD data. What is there is not necessarily the same relationship on the three-dimensional data, so it is better to examine the relationship when generating the three-dimensional data. For two-dimensional data, for convenience of processing, the data structure of being attached may be taken. For two-dimensional data, such data management is not necessarily required. In the case of three-dimensional data, since the processing is a vast three-dimensional space, in order to make it easy to narrow down the processing target, it is possible to perform processing more efficiently by adopting such data configuration. .

  Next, when the viewpoint of the imaging device is changed in this image processing result image (FIG. 58) and the same part is shot in the same way from another viewpoint, so-called stereo measurement and tracking are performed, the same processing is performed. Is also performed on an image newly input from the imaging apparatus. Here, another method for narrowing down the corresponding color boundary line segments and feature points will be described. Here, the corresponding point of the feature point 9v of the image image shown in FIG. 58 is searched from another image image whose viewpoint has changed. The viewpoint of another image also changes, but it is assumed that the field of view includes the same range as the similar image (FIG. 58). Then, if matching is performed only with the color boundary line segments L5 and L6 connected to the feature point 9v by micro matching, there are a plurality of similar patterns in the same scene, so the other feature points 9w and 9x have the same feature. Therefore, it is difficult to narrow down only by micro matching. Therefore, macro-matching is also performed by actively using feature point information at positions distant from the macro, but as an efficient method, there are vectors having the direction and length of other feature points and beyond. A method for performing matching using color information of feature points will be described.

  FIG. 59 (a) shows the concept of subtracting vectors V1, V2, and V3 to neighboring feature points for the narrower region created by the color boundary line segment L5 and the line segment L6 of the feature point 9v for which a corresponding point is to be searched. Show. This vector is subjected to the shape deformation process (b) corresponding to the change of the viewpoint, and the vector is similarly drawn for all the corresponding point candidates in the processing result (c) of the image photographed from the new viewpoint. Then, narrowing down is performed based on the degree of coincidence between the vector and the color information of the feature point ahead. The deformation process may use a method omitted when it is simplified.

  FIG. 60 and FIG. 61 show the same for the feature points 9w and 9x that could not be narrowed down by micro-matching as corresponding point candidates in the processing result of the image information taken from the new viewpoint (FIG. 59 (c)). Show the concept. Here, in the case of FIG. 60, since there are a surface 9T and a line segment L8 attached to the surface 9S, a large number of vectors V1 to V6 are drawn to the feature points constituting them. At this stage, the vectors V1 and V3 in FIG. 59 (a) coincide with the vectors V1 and V6 in FIG. 60. However, in the direction corresponding to the vector V2 in FIG. 59 (a), the vectors V2 to V5 in FIG. Since it exists, it turns out that the possibility that the point 9w is not a corresponding point is high.

  Further, FIG. 62 shows the corresponding point candidates 9v, 9v, and vice versa by drawing the vector in the same manner for the wide area created by the color boundary line segments L5, L6 from the feature point 9v, and using the image information after the viewpoint is changed. A concept in which a vector is similarly subtracted from each of 9w and 9x is shown. When the vectors are drawn and compared in this way, the corresponding point candidate 9v can narrow down the true corresponding feature points because the vectors drawn from the corresponding point candidates 9w and 9x match.

An example of specific data comparison of vectors will be shown below. For simplicity, a description will be given by comparing four vectors and six vectors. Each is composed of the vector direction, length, and color information of the previous feature point. The direction may be expressed by a clockwise angle of a vector from a reference line segment (for example, a line segment L5 or a line segment L6 if the feature point is 9v).

<Example of vector data of feature point P for which a corresponding point is to be searched>
Vector Direction Length Color information PV1 of previous feature point Angle 1 Distance 1 Left color L1 and right color R1 at the vertical boundary
PV2 Angle 2 Distance 2 Left color L2 and right color R2 at the horizontal boundary
PV3 Angle 3 Distance 3 Left color L3, right color R3 at the vertical boundary
PV4 Angle 4 Distance 4 Horizontal color left color L4, right color R4
<An example of vector data of one corresponding point candidate Q>
Vector Direction Length Color information QV1 of previous feature point Angle 1 Distance 1 Left color L1 and right color R1 at the vertical boundary
QV2 Angle 2 Distance 2 Left color L2 and right color R2 at the horizontal boundary
QV3 Angle 3 Distance 3 Left color L3, right color R3 at the vertical boundary
QV4 Angle 4 Distance 4 Horizontal color left color L4, right color R4
QV5 Angle 5 Distance 5 Left color L5, right color R5 at the vertical boundary
QV6 Angle 6 Distance 6 Horizontal color left color L6, right color R6
Here, there are four vectors PV1 to PV4 subtracted from the feature point P, and six vectors QV1 to QV6 are subtracted from the corresponding point candidate Q. However, the number of feature points is stably extracted at the time of shooting. As a result, the number of vectors does not match even at corresponding points. In this example, for example, the vectors PV2 and QV2 and the vectors PV4 and QV5 may match very well, but they can still be candidates for corresponding points. This is because feature points are extracted based on the number of matching vectors, but not on either side, so it is not determined whether all the subtracted vectors match. It is information on how many vectors match the color information of the previous feature point, and in comparison with other candidate points, it is possible to narrow down which one has the higher degree of matching good. In this process, the comparison is performed for each region formed by the color boundary lines L5 and L6 connected to the feature point 9v for which a corresponding point is to be searched. Since the comparison result may be a boundary line segment (contour line) with the background that first entered the field of view from another viewpoint, it does not necessarily match in all areas, so compare for each area, The corresponding points are narrowed down if there are many regions that match or have a high degree of matching. In this example, since there are two color boundary line segments L5 and L6, the degree of coincidence is examined by drawing a vector in each region where 360 degrees around feature point 9v is divided into two, but there are three color boundary line segments. If so, the area is divided into a large number of areas according to the number of color boundary lines so that the number of areas becomes three. In the case of the end point of one color boundary line, it is examined in one area in the range of 360 degrees. Also, in this example, when a vector is drawn in the area, if the vector reaches another color boundary line, no further feature points are drawn. Since the presence / absence of a region having a high tendency is examined, the comparison may be performed by drawing a vector to the feature point on the other side of the other color boundary line segment.

  Here, an example of the discrimination logic will be described with reference to FIGS. 63 (a), (b), and (c) are cases where a vector is drawn for each of corresponding point candidates having similar shapes as in the surfaces 9R, 9S, and 9U in FIG. A region is illustrated. In either case, the narrower angle becomes the region (1) and the larger angle becomes the region (2). In this example, there are two regions, but there may be three or four regions. In order to simplify the process, when the color boundary line is detected in the horizontal and vertical directions, the region may be divided in the detected direction. When color boundary lines are extracted both horizontally and vertically, there are four areas. This simplified method is effective when searching for corresponding points during stereo measurement in which a rotation component is not included in the change in viewpoint. In addition, the region may not always be clearly extracted, so the degree of coincidence of the vectors is examined in order in one direction, and the processing is performed by regarding the boundary between the region with a large proportion of matching vectors and the region with a small region as a region later. You may make it do. This method is effective when searching for corresponding points at the time of tracking, in which a rotation component is generated when the viewpoint changes.

  FIG. 63 shows an example in which the regions (1) and (2) are identified as “◯” when the frequency of matching vectors in the regions is sufficiently high compared to others, and “x” when not. Shown in (d) and (e). (D) is an example in which the corresponding point candidate (a) has a high degree of matching between the region (1) and the region (2), and the matching point candidate (c) has a high degree of matching. However, in such a case, the corresponding point candidate (b) can be excluded, but the corresponding point candidate (a) and the corresponding point candidate (c) cannot be narrowed down only from this result. This is because the corresponding point candidate (c) may happen to have a low degree of coincidence due to the background of the region (2). Of course, if the corresponding point candidate (c) is X in both the region (1) and the region (2), the corresponding point candidate (a) is both in the regions (1) and (2), and narrowing down. Can do. In (d), only the region (1) of the corresponding point candidate (a) is ◯, but even if one of the regions is ◯ and there is no other, the corresponding point candidate is It becomes possible to narrow down. When at least one region has two or more corresponding point candidates, the result is that the corresponding point candidates cannot be narrowed down to just one by the matching process alone. In such a case, the allowable error range of the angle and length for determining that the vectors match is narrowed, the allowable range of the color information at the tip of the vector is reduced, and the range for subtracting the vector is the color boundary line. The corresponding point candidates may be narrowed down more precisely by limiting the area to be drawn so as not to exceed or by increasing the matching accuracy of the deformation process. In addition, even if the number of vectors to be drawn is not a comparatively large number from the beginning, the processing speed may be increased by initially drawing the vectors by focusing on feature points existing within a predetermined distance. As a result, if narrowing down cannot be sufficiently performed, the feature points for which the vector is drawn may be expanded to a wider area, and the vector may be drawn and compared using information on feature points farther away. Also, in the example here, the vector is drawn to other feature points, but the feature points here are end points, branch points, large bent vertices in the middle of the line segment, etc. The only data that can be divided as a line segment is the start point and end point. However, when performing vector matching, a vector is also drawn for matching between the start point and end point of the line segment. May be. This is because, for example, when the number of line segments is small, it is difficult to narrow down the corresponding point candidates because the number of vectors to be compared is small when vector matching is performed using only feature points such as end points of the line segments. Even in such a case, the vector up to the point in the middle of the line segment becomes a vector corresponding to the direction and shape of the line segment, so that the corresponding point candidates can be narrowed down by the degree of matching of the direction and shape of the line segment. it can. On the other hand, if the number of line segments is large and the line segments are crowded in one place, many similar vectors exist when the vector is drawn to the middle point of the line segment. On the other hand, it may be difficult to narrow down the corresponding point candidates, so as a vector matching method, use vectors up to the feature points, and if you cannot narrow down, perform vector matching using points in the middle of the line segment. Narrow down or reverse the procedure, but narrow down the corresponding point candidates by using both the vector matching of only feature points and the method of vector matching including vectors up to the middle of the line segment You may do it. As a method for narrowing down the matching, it can be applied to both stereo measurement and tracking, but when extracting the color boundary line for the first time, the sensitivity is lowered and only the clear color boundary line segment is selected. Extract and narrow down corresponding point candidates by matching, then increase the sensitivity of the color boundary line, extract more color boundary line segments, and perform the matching process. The corresponding point candidates are narrowed down by the newly extracted line segment except for the color boundary line segment in which the corresponding point is clear. Further, in the second and subsequent matching processes, it is possible to narrow down the corresponding point candidates currently being processed by actively utilizing the relationship between the corresponding points obtained first. If a lot of color boundary line segments are extracted with high sensitivity from the beginning, many color boundary line segments are extracted in the adjacent parts, so it is not easy to narrow down the corresponding point candidates, and a combination of matching processing Since the number of times increases, processing time also takes a lot. Therefore, by extracting the color boundary line segment in several steps according to the procedure described here, it becomes possible to narrow down the corresponding point candidates with high accuracy and efficiently for many color boundary line segments. .

  Further, as shown in FIG. 63 (f), even when two figures (a) and (b) are in front of the background (Z), both are attached to a common plane (A). In this case, by drawing the vector up to the feature point of the common plane (A), different parts of the vectors of the figures (a) and (b) are generated. Even narrowing down is possible.

  Further, even when there is no common plane as shown in FIG. 63 (g), when the relative positional relationship between the figures (a) and (b) does not change, the respective figures (a) and (b). It is possible to identify and narrow down by subtracting a vector. However, in the case where the figures (a) and (b) move in front of the background (Z) independently from each other in FIG. 63 (g), a stereo that simultaneously captures images from two viewpoints. Although the corresponding point search at the time of measurement is possible, the corresponding point search at the time of tracking has a limit of application. In such a case, it is also necessary to combine with a method of associating corresponding points from a drain of a change in three-dimensional position information obtained by stereo measurement. If such a vector matching method is used, it becomes possible to easily narrow down the corresponding point candidates 9v from the corresponding point candidates 9v, 9w, 9x that cannot be narrowed down by the micro matching alone at the corresponding points of the feature point 9v. Since this method only performs processing using the coordinates and color information of feature points that have been CG-converted in two dimensions, it is easier and faster than matching processing as surface data even with local micro-region image data or CG. This is a possible technique.

Also, FIGS. 64A and 64B show an example in which the present method is extended to three dimensions. FIG. 64A shows a concept in which points P1 to P8 are generated as three-dimensional point data generated at a certain time, and a line segment P1P3 and the like are also generated as three-dimensional data. FIG. 64B shows a similar concept when CAD data of the same part is newly generated at another time. Here, it is an example of the case where the present invention is applied to search for the point corresponding to the point P1 of the generated CAD data (a) from the CAD data (this) generated at another time. This can be applied to an alignment process or the like in the case of losing sight by the search function of CAD data or two-dimensional image tracking and continuing tracking again.

  First, a vector from the point P1 to a point existing within a predetermined distance around is drawn. Similarly, in the CAD data generated at another time, all the points that are candidates for the point P1 are similarly drawn to vectors that exist within a predetermined distance. (B) shows an example when a vector is subtracted from the point P3. When the vector from the point P1 in (a) and the vector from the point P3 in (b) are not matched, it can be immediately understood that the point P3 is not a candidate for the European point of the point P1. Similarly, in (b), for other points, if the vector is similarly drawn and examined, it matches very well when the vector is subtracted from the point P1 which is a true corresponding point candidate, so the corresponding point of the point P1 Can narrow down candidates. Similarly, if three corresponding points are found, the entire CAD data can be dealt with, and this method can be effectively applied to such a case.

  The above-described example is a case where the points P1 to P8 are feature points extracted from the color boundary lines that actually exist. Therefore, the corresponding corresponding points are found by processing as they are. An example in the case where point data does not exist will be described with reference to FIGS.

  65 (a) and 65 (b) show, as CAD data, feature points 9y, 9z, etc., of actual color boundary lines that are characteristic of the weld line by irradiating a laser beam such as a weld line bead with a red laser beam. In addition to being generated, a concept is shown in which there are CAD data of contour lines RedL1 to RedL3 and RedL4 to RedL6 generated by a laser beam. The difference between (a) and (b) is that the CAD data of the same weld bead portion is generated, but the position and direction of the laser beam irradiated at that time are different, so the virtual line indicating the shape generated by the laser beam A description will be given of a case where the line segment data and the point data constituting the line segment do not correspond to each other in FIGS. Color boundary line segments detected as actual bead patterns are generated at the same position, but the number of generated line segments is very small. In this example, matching processing is also performed using the virtual line segment data.

  As shown in (c) and (d) of FIG. 65, the points RedP1, RedP2, and RedP3 of the line segment generated by the laser beam RedL2 are the points RedP4, RedP5, and RedP6 of the line segment generated by the laser beam RedL5. Is not supported, for example, even if the vectors V1, V2, V3, V4 are subtracted from the feature point 9z in (c), the vectors V1, V2, V2 are similarly derived from the feature point 9z in (d). Even if V3 is subtracted, since the virtual point originally generated by the laser beam is not a corresponding point, the vector subtracted from the virtual point is not the same in the case of the corresponding point. That is, it is difficult to narrow down the corresponding points.

  In such a case, as shown in (e) and (f), the virtual points Dp1 (x) and Dp2 (x) of (e) that smoothly interpolate between the original virtual points. ), Dp3 (x)... And (f) are created as virtual points Dp1 (y), Dp2 (y), Dp3 (y), a vector is subtracted from the feature point 9z, and the degree of coincidence of the vectors is examined. In this way, even when there is no corresponding point, the corresponding point can be searched and narrowed down from the entire shape information. In such three-dimensional vector processing, it is also possible to narrow down more accurately by using color information of feature points ahead of the vector and line segment information connected thereto.

  66 (a) to 66 (e) show an example in which this vector matching method is used for the search function. There are two possible assumptions for the search. If the search target is known and does not move, if the search target is known but where it is unknown, or if the search target moves (position and orientation change) It is.

  With reference to FIGS. 66A and 66B, a basic search method when a search target is known and does not move will be described. (A) shows the concept of the known search object 901 and the existing three-dimensional CAD data around the known environment. It is assumed that the search object 901 and other CAD data are prepared as CAD data in advance in what position and how in the world coordinate system O1. The CAD data may be generated in advance by moving the imaging apparatus using the method of the present invention, or may be generated by another means. Therefore, at the time of search, image information from the imaging device 10 is actually obtained and processed to generate two-dimensional CAD data, or three-dimensional CAD data is generated by performing stereo measurement or tracking measurement. However, here, two-dimensional CAD data (feature points extracted from color boundary lines, color boundary line segments, surfaces formed by them, etc.) generated from image data taken for efficient search Will be described in the case where matching is performed in a two-dimensional state.

  Specifically, if the current position and orientation of the imaging device 10 are known in the world coordinate system O1, the appearance when the search target 901 is captured from the position can be generated from three-dimensional CAD data. 66A is a concept of three-dimensional CAD data. However, assuming the imaging device 10 in the three-dimensional CAD data, the three-dimensional CAD imaged on the imaging device 10 in that case. If data is generated, two-dimensional CAD data to be referred to as shown in (b) is obtained. In this case, since the object 901 does not move and the position and orientation in the environment are fixed in advance, the reference CAD data is generated by projecting both the object 901 and other environment CAD data to two dimensions. To do. Here, when searching for a corresponding point in the captured image information corresponding to, for example, the point P1 in (b) which is the reference CAD data, the vector subtracted from the P1 point of the reference CAD data as described above. Similarly, a vector is subtracted from all feature points in the scene obtained by generating similar two-dimensional CAD data for the image information obtained by photographing, If a feature point having a matching region is extracted and searched, it is possible to perform a search by two-dimensional vector matching at a higher speed than three-dimensional matching.

  A basic search method when the search target 901 moves or when the position and orientation in the world coordinate system O1 are unknown will be described with reference to FIGS. 66 (c) to 66 (e). In this case, two-dimensional reference CAD data is generated using three-dimensional CAD data only for the search target. (C) is three-dimensional CAD data of only the object 901, and when performing a two-dimensional matching process in which the object 901 is searched in an unknown three-dimensional environment, the object 901 has some basic shape appearance. Two-dimensional CAD data projected onto the imaging surface of the imaging apparatus when viewed from different directions is generated for the number of viewpoints, or prepared as two-dimensional CAD data in advance. In the example of FIG. 66 (c), the position of the imaging device is the three positions and postures of the imaging devices 10 (1), 10 (2), and 10 (3), and each is projected two-dimensionally from there onto the imaging plane. (D), (e), and (f) show the situation in which the CAD data is generated. Here, typical two-dimensional CAD data having three different appearances are shown, but in addition, if a large number of reference CAD data is generated with the appearance when photographing from another viewpoint, it is prepared in advance. You may do it. If the three-dimensional CAD data is stored as data in advance, the necessary two-dimensional CAD data can be generated when necessary. In the case of three-dimensional CAD data, first, after narrowing down the direction in which the outline is viewed based on a rough difference in view from a representative viewpoint, the next is slightly different from that direction. It is possible to increase the matching accuracy by generating two-dimensional CAD data viewed from the viewpoint and repeating it. As CAD data, it has both 3D CAD data and 2D CAD data viewed from a certain viewpoint, and it is a system that can roughly search by omitting the time to generate 2D CAD data. Also good. When the target moves or the position is unknown, the search is performed from CAD data of only the search target object 901. There are CAD data for other environments, even if the search target is shown, in which two-dimensional CAD data is generated from actually captured image information, and there may be many items that do not relate to feature points. If a vector is subtracted from all feature points using the same method and compared, the vector subtracted from the feature point of the relevant part is subtracted from the point of the reference CAD data when the search target is actually shown. Search for a vector that may match.

  Even if the search is performed in this way, if the matching process is performed using two-dimensional CAD data, a vector is similarly extracted by subtracting the vector from the corresponding point to be searched, and subtracting the vector from each feature point in the input image information. Although it is possible to narrow down and find matching points, in this case, since it is unknown which of the prepared two-dimensional CAD data is seen, it is the same for all the prepared two-dimensional reference CAD data. To narrow down differences in viewpoints. Specifically, when three pieces of two-dimensional reference CAD data (d), (e), and (f) are prepared as in this example, all visible points in (d) are displayed. The corresponding CAD is searched in the same manner in (e) and (f), and the reference CAD having a high matching degree with the two-dimensional CAD data generated from the currently captured image information of the three two-dimensional reference CAD data. Select data. Of course, since there is a case where the search target is not shown at all, when matching is performed at a predetermined threshold value or more, the matching data is selected from the three pieces of reference CAD data. When matching is achieved to some extent, two-dimensional reference CAD data is generated by slightly changing the viewpoint, and the search target is searched more accurately. In addition, when generating the reference two-dimensional CAD data, the positional relationship between the current imaging device and the search target (for example, searching for an object along the road from the opposite side of the road, or on the bridge) If approximate distance information can be narrowed down), if the approximate shooting distance is known, two-dimensional CAD data may be generated in consideration of this, and the orientation is determined depending on the object. In such a case (for example, the placed state can only take a stable posture), the orientation is narrowed down in advance to generate the reference CAD data, and the number of the two-dimensional reference CAD data is reduced as much as possible. To increase the search speed.

  Further, when performing two-dimensional or three-dimensional matching, grouping of CAD data may be performed in advance, and only a specific group may be used as CAD data to be searched to increase search speed and improve matching accuracy. Specifically, for example, CAD data of a target object in which a black character “ABC” is printed on a cover of a book with a blue or yellow pattern as a background is generated. When searching for “ABC”, CAD data of the background pattern and CAD data of the characters “ABC” are respectively generated. At this time, both are closed-loop surfaces formed by the outline of the book. Since the black character “ABC” is the one to be searched for, the line data related to the black color boundary line or the black closed-loop surface data is first extracted to obtain one group. The matching search processing is related to the black characters of group A, with the color boundary line segment data constituting the background blue and yellow patterns or the blue and yellow closed loop surface data group B. CAD day By to take the search matching from among, it becomes possible to accurately match faster.

  Of course, when searching for characters in a book, matching with character data of various title candidates and recognizing the title of the book, even if narrowing matching processing is performed on the color group of the title character, Similarly, narrowing down the search target by color group is effective even when a matching search is performed by grouping title portions of a book with a title “ABC” of black characters from a plurality of candidate books. In addition to grouping by color, the length of a line segment, the size of a surface, the shape of a straight line, curve, arc, etc. of a line segment, the shape of a surface, etc. are grouped in advance if they are valid according to the characteristics of the search target. If the search process is performed after narrowing down the search target, the search matching process is performed by matching while changing the various directions and sizes, etc. Since it takes time, first narrowing down the number of candidates for matching by grouping is effective in shortening the search time. In addition, since the group features are first narrowed down, finding a candidate with a high degree of matching among them can eliminate candidates that are similar to another group from the matching target at the stage of grouping. Matching accuracy is also improved.

  Next, the concept of error at the time of coordinate conversion by tracking will be described with reference to FIG. FIG. 67 (a) shows a case where a shooting target 901 is tracked from the viewpoint O1 to the viewpoint O2 and may be a monocular camera system, but a stereo camera system performs 3D measurement in a sufficiently separated state at that location. It is also good. (B) shows a situation where the next sequence is changed from the viewpoint O2 to the viewpoint O3, and (c) shows a situation where the viewpoint O3 is changed to the viewpoint O4. Here, (d) shows a situation in which CAD data is converted into the same coordinate system so that the viewpoints O2 and O3 overlap each other. In this case, an error of coordinate conversion by tracking is included in the CAD data.

  FIG. 67 (e) shows an example in which the CAD data is matched with the viewpoint instead of the viewpoint. In this case, an error due to coordinate conversion of tracking is included in the position of the viewpoint, that is, the position and orientation information of the imaging apparatus. In the case of an application system that generates CAD data, it is preferable to use the method (e) in which there is no error on the CAD data side. In this case, since the position of the imaging device is obtained based on the relative positional relationship with the CAD data in the measured sequence, even if the absolute position error from the first O1 coordinate system becomes large, the imaging with respect to the current imaging target is performed. Since the position of the device is obtained with high accuracy, there is no problem in application to movement control avoiding an obstacle, navigation for accessing a corresponding object, and autonomous control. There is also a method of converting both CAD data so that the viewpoint position information of the image pickup apparatus is also appropriate. What is necessary is just to select those most suitable according to the system to apply.

  68 to 70 show the basic structure of an embodiment of the seventh means of the present invention.

  In FIG. 68, the X axis represents the horizontal coordinate of the image data, and the B axis represents the luminance value. If it is a black and white image, it is brightness, but if it is a color image, it may be considered as the brightness values of R, G, and B, and even a color image may be processed with the brightness value of brightness. Think of it as a luminance value. In the case of an imaging device that can obtain a distance image, the brightness value corresponding to the distance information is applied. Here, I1, I2, I3, and I4 on the X axis indicate pixel addresses, and X1, X2, X3, X4, and X5 correspond to coordinate values on the X axis (corresponding to X coordinate values in the camera coordinate system). Coordinate value). The respective luminance values are 11a, 11b, 11c, and 11d. The actual luminance distribution of the object to be imaged is shown by a solid line 11f, but since the image is captured by an imaging device having a finite pixel, the value obtained as sensor information is the luminance value at each address. Become. Here, since the brightness value changes means that the brightness and color information have changed, the inflection point is found and the left and right color information is given to the inflection point information. . There are various methods for estimating the position of the inflection point, but the simplest method is to increase the amount of change from the luminance 11b to the luminance 11c from the amount of change from the luminance 11a to the luminance 11b, and then to the luminance 11b. Since the amount of change from the luminance 11c to the luminance 11d has decreased from the amount of change from the luminance 11c to the luminance 11c, the inflection point 11e is the place where the increase / decrease tendency of the amount of change is reversed between the pixel addresses I2 and I3 in this example. Extraction should be performed. Further, as a method for obtaining the position of the inflection point with high accuracy, if the inflection point is found as described above, the luminance values of the four points before and after the inflection point are used, and for example, a cubic curve is used by the least square method or the like. A position closer to the true inflection store 11g may be detected by fitting to obtain the inflection point of the cubic curve. And the left and right color information of the extracted inflection point may be the average color of the color up to the next inflection point on the left and right of the detected inflection point, Color may be used as color information. When averaging, the color information near the inflection point may be removed because it is the color of the changing process. Alternatively, the left and right color information of the inflection point may be simply given without worrying about the adjacent inflection point. The inflection point indicates the boundary where the color information changes, so the averaged color has some meaning, but when using the nearest primary colors, the difference may be small, but color matching is strictly the primary color There is an advantage that can be done. The above processing is performed not only on horizontal pixels but also on vertical pixels. Further, if the image is rotated in a 45-degree oblique direction and then the horizontal process is performed again, the oblique line segment can be extracted with higher sensitivity.

  Next, an example of a method for creating a color boundary line segment from the inflection points extracted in this way will be described with reference to FIG. 69 (a) and 69 (b) describe an image of 4 × 4 pixels as image data. Here, it is assumed that there are two colors of color information, and there are pixels of color 1 and pixels of color 2. Here, an inflection point is obtained by the method shown in FIG. 68, and points such as both ends 12a and 11b on one side of the pixel are connected.

  FIG. 69B shows a method in which a point 12c is defined at the center of one side of a pixel extracted as an inflection point and the points are connected.

  The method (a) is faithful to the pixels. For the inspection apparatus, the method (a) is suitable. The method (b) feels like connecting the line segments smoothly.

  Next, even if line segments are extracted in this way, since they are extracted from the pixel level, jagged characteristics remain in units of pixels, and processing for removing them may be performed.

  In FIG. 69 (c), a point in the middle of a straight line can be omitted, so the point in the middle of the line segment CAD data may be omitted. Also, as in the example of (d), the direction of 90 degrees changes after a straight line of a predetermined length, and then there is a straight line of one pixel length, and then the direction of 90 degrees changes in the opposite direction and is the same as the first. When the line segments of the length are connected, the points that change the direction by 90 degrees may be omitted and directly connected. In the example of (e), a portion where the same processing is zigzag in units of one pixel can be omitted. Further, in the example of (f), in a place where a line segment is not necessarily sufficiently extracted due to illumination conditions or the like, the line segment estimated as a straight line is defined as a straight line fitted by the least square method or the like. A process for defining the intersection of straight lines as a new feature point may be included. In this case, the error of the feature point obtained in this way includes the error of the measurement point from which the straight line that created the point was found and the error fitting with the straight line. It is managed as a data error, and it is determined whether or not such a point is to be updated depending on whether the error considering such a fact becomes larger or smaller.

  Further, in a line segment estimated to be a non-straight curve, the extracted feature points may be connected with a smooth curve such as a B-spline instead of connecting with a straight line. The feature point and feature line segment extraction method described with reference to FIG. 68 can extract the boundary portion with high accuracy. However, since the accuracy is good at the pixel level, the linear portion is a straight line as shown in FIG. It may not be extracted cleanly. In addition, the arc may not be extracted as an arc. In addition, there are many straight lines, circular arcs, and smooth curves in the outline and color boundary of an object in a general environment. Therefore, extraction of feature points and feature line segments by the method of FIG. 68 and processing that easily extracts straight lines, arcs, and the like as much as possible, such as binarization processing of images, are performed on the same image, About the feature points and feature lines extracted by the method of FIG. 68, which approximates the contours of figures extracted by binarization processing with straight lines, arcs, and smooth curves, and is close to those straight lines, arcs, and smooth curves 69, the feature points and feature line segments extracted by the method of FIG. 68 are fitted by the method of fitting with the straight line of (f) of FIG. 69, and more accurate straight lines, arcs, and smooth curves are fitted. May be newly defined, and those straight lines, arcs, and smooth curves may be used for 3D measurement.

  With reference to FIG. 70, the case where feature points and line segments are extracted by applying the method described with reference to FIG. 68 is compared with the case of another method.

  In FIG. 70A, the feature points extracted by the method described with reference to FIG. 68 and the color boundary line are superimposed on the image to be photographed. The image (A) is an enlarged image of the A portion. The object to be imaged is a bead of a weld line in water. In this case, since the inflection point of the brightness is extracted with high sensitivity, the uneven shadow of the bead of the welding line, the outline of the spot of the illumination, the dirt next to the bead, and the like are extracted. The feature of this method is that, as shown in (d), a uniform change in the tendency of the luminance distribution is extracted due to dust and slight irregularities in a uniform gradient in the luminance distribution when illuminated. Thus, the left and right color information can be easily provided as average color information of inclined luminance. In addition, if the intermediate position of the inflection point is extracted even when the change in luminance is gradual, as in the luminance part of (e), the position of the color boundary line can be detected closer to the actual and accurately. Can do. In some cases, the subject to be photographed actually has such a luminance portion, but the luminance distribution of (e) is present when the focus of the imaging device is out of focus even when the color boundary of the subject subject to photography is clear. Therefore, it is an effective technique even in such a case.

  FIG. 70 (b) shows a boundary line extracted by simply dividing the same photographed image by the level of the luminance value. With this method, the boundary line is always a closed loop, but the actual feature points are not extracted as feature points, but rather the illumination distribution is extracted. It is not possible to extract a locally small change portion such as the luminance distribution of (f). In addition, as for the portion where the luminance change is gentle as in the luminance distribution of (g), a location different from the inflection center is extracted.

  FIG. 70 (c) shows a case where the average color of similar colors within the predetermined range (ΔRGB) is obtained to be the same group, and the boundary line of the group is extracted as the color boundary line. This is advantageous in that the color boundary line can be detected with relatively high accuracy as shown in (h) if the boundary part where the color changes is within ΔRGB, and the boundary line becomes a closed loop. If the boundary becomes a closed loop, the surface can be defined, and the color information is accurately attached to the surface, so this method may be effective depending on the application. In this method, the boundary is not detected when the amount of change in luminance is smaller than ΔRGB. The deviation of the extraction position of the color boundary line affects the subsequent corresponding point search accuracy, and further the accuracy of the 3D measured point.

Among these methods, the method of obtaining the inflection point described with reference to FIG. 68 can detect the color boundary line with the highest accuracy and sensitivity. In particular, when applied to a VT (visual) inspection apparatus, the method of FIG. 68 that can generate CAD data accurately is preferable. If this method is used, CAD data can be generated with high accuracy and high sensitivity. Sensitivity to extract the inflection point is a threshold that distinguishes increase / decrease in luminance increase from noise, etc., but the sensitivity may be processed constant throughout the screen, or from the state of local luminance distribution The local threshold value may be changed for each local region so that the entire screen is processed with the best sensitivity. Further, from the viewpoint of processing speed, the sensitivity may be automatically adjusted so that the number of line segments in the processing area falls within a predetermined number so that the same level of line segments to be processed are always generated. . One method of automatic adjustment is to estimate the number of lines in the entire area by counting the number of line segments generated by processing in a minute area. It is only necessary to set the appropriate sensitivity by repeating this process several times.
FIG. 71 shows a basic configuration of the eighth means in the present invention. FIG. 71 shows a situation where feature points are being tracked.

  Initially, three line segments of line segment 14a, line segment 14b, and line segment 14c are detected, and the end points of these line segments are feature points. A description will be given of a processing method in a case where three line segments become one closed-loop line segment 14d when the line segments are well extracted at the stage of tracking these feature points.

  Usually, the feature point of a line segment is an end point, a branch point, a sharp corner point, or a point that changes from a straight line to a curved line. If there is no, it will be difficult to continue tracking. The eighth means enables tracking to be continued even in such a case, and the feature points are rearranged when a line segment having a feature point is changed to a closed loop line segment having no feature point. . In the example shown in FIG. 71, when the closed-loop line segment 14d is changed to the closed-loop line segment 14d, the closed-loop graphic center 14e and the closed-loop graphic directionality feature in this case are the direction 14f and the direction 14g. The direction information is, for example, a convex shape. Directions 14h and 14g are defined as vectors from the center 14e. The length may be the length from the center 14e to the boundary line. As processing, since the line segments 14a, 14b, and 14c should naturally not be detected when the line segment changes to such a closed loop, the subsequent processing is performed in place of the line segments 14a, 14b, and 14c. The closed-loop graphic is tracked by using information such as the shape of the closed-loop basic graphic, the center position, and the direction of the characteristic shape part. Then, later, when the closed loop is no longer the closed loop, the process of returning to the feature point tracking again is performed. In this example, the closed loop 14d shows a situation in which the line segments 14h and 14i are next obtained. At this time, the closed-loop figure no longer exists, so the process with the corresponding figure is stopped, and the process of tracking the feature points of the newly detected line segment will result in a closed-loop case. Will be able to continue tracking.

  FIG. 72 illustrates the basic configuration of the ninth means of the present invention. In FIG. 72, the subject 901 may be either monocular or stereo. However, when CAD data is generated while photographing, the line segments L1 and L2 are measured, and the world coordinate system O1 (the first time the image pickup apparatus is simply started to move). The camera coordinate system may be considered, or the relationship between the camera coordinate system and the reference world coordinate system is known and may be considered as a transformed world coordinate system). Here, since the measured CAD data is only the line segments L1 and L2, the surface data is not yet defined in order to avoid a closed loop. Since surface data is not defined, there is no color information as surface data. Even if the CAD data measured in such a state is displayed on the monitor and confirmed, the relationship between the actual imaging object 901 (such as a tank standing on the ground) and the measured line segments L1 and L2 is not known. . In the ninth means, when the line segments L1 and L2 are measured, the position and orientation of the imaged imaging device and the image data at that time are recorded in association with each other.

  72 (c) shows a concept in which the image data G1 and G2 are recorded in correspondence with the CAD data of the line segments L1 and L2, and the image file name “image 001.JPG” is stored in the image data G1 and G2. And the positions x, y, and z in the O1 coordinate system of the image pickup apparatus when the image is taken, and α, β, and γ of rotation angles about the x, y, and z axes are displayed as image no. The concept of data recorded in correspondence with is shown. If recorded in this way, when CAD data is displayed, it becomes possible to easily display a photographed image obtained by measuring the CAD data on the background of the CAD data L1 and L2 from this data.

  FIG. 72D shows an image of a monitor screen displaying such a corresponding image on the background. Image data G1 is displayed in the background of the line segments L1 and L2. In this case, since the line segments L1 and L2 completely correspond to the background image, the image data correspondence information includes the position and orientation information of the imaging device, so that the CAD data is automatically displayed on the monitor. If the position of the current viewpoint is displayed in accordance with the position of the viewpoint when the imaging apparatus takes a picture, the CAD data and the background coincide. In this state, it is possible to easily confirm what the measured line segments L1 and L2 correspond to by enlarging the screen or confirming in detail. In addition, in order to confirm the shape of the line segments L1 and L2 having the three-dimensional information, there are cases where it is desired to change the direction of the viewpoint or to enlarge and display it closer. Since it does not correspond to the background, a display device that can return to the corresponding state with a single switch may be used. Further, when the viewpoint changes slightly, the background image may be deformed and displayed with the portion of the background photograph with the line segment as the position of the CAD data. Also, if there are more line segments of CAD data, if the display viewpoint is changed, any CAD data will come to the center of the monitor. It is displayed so as to automatically switch to background image data corresponding to certain CAD data. When there are a plurality of candidates, the background image data closest to the viewpoint displaying the current CAD data may be automatically selected and switched to be displayed in an overlapping manner. In the case of the stereo camera system, depending on the object, especially when shooting an object that shines at a close distance, the appearance of the reflected situation may vary greatly depending on a slight difference in viewpoint. Therefore, in such a case, both the left and right images may be recorded in association with the shooting positions of the left and right cameras.

  Further, since there is color information on both sides of the line segments L1 and L2 of the CAD data, the color may be displayed as CAD data on both sides of the line segments L1 and L2 in an area that is easy to see on the monitor to some extent. In this case, even when there are few line segment data and background image data, the color information around the line segment can be confirmed as such. Of course, an image may be displayed on the background, and CAD color information may be displayed on both sides of the line segment. The display area of the color information displayed on both sides of the line segment should be small if the adjacent line segments are dense when displayed, and large if there are no other line segments in the vicinity. The size of the display area may be automatically adjusted so as to display.

Of course, if a closed loop surface is generated, an image may be pasted on the surface. In particular, when the extraction sensitivity of the color boundary line is lowered, it is effective when the portion not extracted as the color boundary line segment is pasted as the texture of the image and managed as the color information of the surface. In this case, the image may be updated to an image taken from the normal direction of the surface, and if the appearance differs depending on the viewing angle, a plurality of images corresponding to the viewing direction may be pasted on one surface. Anyway. For the portion where the closed loop is not generated, an image may be pasted on the virtual background generated first.
73 to 115 illustrate the basic configuration of the 11th to 15th means of the present invention.

  FIG. 73 shows a basic application configuration of the visual information processing apparatus, and basic commands and extended commands for the visual information processing apparatus. The visual information processing apparatus 100 receives a video signal from the imaging apparatus 10 such as a camera and cooperates with the control apparatus 300 on the application side. The basic concept is to issue a command command to 100 and return the result as a response from the visual information processing device 100 to the control device 300 on the application side. The application-side control device 300 is connected to a driver 400 that drives and controls various sensors 30 and actuators 401. Of course, various other devices may be connected. The visual information processing apparatus 100 is connected to a memory 200 that records CAD data and the like. These are only examples, and other configurations similar to this may be used.

  Here, basic commands from the control device 300 on the application side to the visual information processing device 100 include a function for clearing CAD data by resetting the origin, a camera position request for extracting the three-dimensional position information of the camera, and generated CAD data. A search command for performing a search and a file registration function for registering the generated CAD data in a file. Various extension commands corresponding to high-speed processing for visual information processing on the application side are prepared.

  Examples of these extended commands will be described below.

  The extended commands 1 to 4 are examples of commands for designating an area to be generated when CAD data is generated. Depending on the area designation method, four examples will be described here.

  The extended command 1 area specification method is a method of specifying the position coordinates of the X, Y, and Z in the world coordinate system and the size of the area centered there. Here, the radius is specified and the spherical area is specified. To do. The area may be a rectangular parallelepiped or a cylinder, but in this example it is a sphere. The image processing for this command requires processing time because the entire screen is always processed with only the basic command, so if you want to move the camera faster, for example, if you want to move the robot faster, In some cases, it is desirable to increase the speed of visual information processing. The specified area sent with the command is the point of X, Y, Z on the two-dimensional image data reflected in the camera from the three-dimensional coordinates of X, Y, Z and the current position information of the camera. I understand. In addition, since the size of the area can also be determined from the positional relationship for a sphere having an actual size of radius r on a two-dimensional image, the radius of the sphere can be obtained from the positional relationship. Is generated. Of course, since it is impossible to lose sight of the corresponding points for tracking, the tracking process is performed on the entire screen. If at least the processing area for generating CAD data becomes smaller, the processing speed increases accordingly, and this is an effective command for an application that wants to move quickly. Further, if the amount of CAD data to be generated is reduced, the search processing time can be shortened. Of course, the coordinates X, Y, and Z may be changed and designated frequently each time. In addition, the CAD data search may be performed after limiting the CAD data generation region. Similarly, the search range of CAD data may be limited to the designated area. For example, when the biped robot walks down the stairs, only the edge portion of the step stairs is 3D-measured to generate CAD data, and the stairs are generated using the reference CAD data of the stairs edge. This is an effective function for use in position control for detecting the position of the edge and then moving the foot next. Since the position where the robot will next step is predicted in the control plan, an area may be specified from the predicted position. Also, in the case of a robot having a swing mechanism equipped with a directional microphone together with a camera, if the target sound to be searched for can be identified by the sound processing of the directional microphone, There is also a usage method in which if a rough direction is first known using characteristics, CAD data is generated by designating the direction and a target to be searched is searched. In this case, a command for specifying the direction and the size of the area, not the X, Y and Z coordinates and the size of the area may be prepared.

  In addition, the area designation of the extended command 2 is designated within a range of speeds of a speed (such as slowly moving or fast moving) specified in the two-dimensional image, and the speed in the two-dimensional image. The moving part is extracted by two-dimensional difference binarization processing or the like, and the moving speed of the changing part in the two-dimensional image is obtained. It is an example of a command for generating three-dimensional CAD data around a moving object when it occurs in a two-dimensional image. In the two-dimensional target processing area, the area in which an object that has already moved in the two-dimensional difference binarization processing is detected becomes the 3D CAD data generation processing area as it is. This command also becomes an effective command by limiting the processing area when it is desired to search a moving object at high speed. Also, if this function is used to fix the camera and generate CAD data while rotating the object on the rotary table, there is no reference CAD data of the actual object. Even if it is not generated in the room, it is possible to generate CAD data only around the object moving on the rotary table even in a room with various objects in the background. Also good.

  The CAD data extension command 3 area designation is performed by detecting, for example, a shining portion in a two-dimensional image by two-dimensional image processing, and processing the periphery of the object if there is an object shining at a designated luminance level. It is a command to do so. This is also effective when the object to be searched is known to be a shining object. Alternatively, it is also effective when searching for an object that is not shining in a shining background.

  In the area designation of the extended command 4, an object having a specific shape, color, and size in a two-dimensional image is obtained by two-dimensional image processing. When detected on a three-dimensional image, this is a command for processing around the detected object. This is because the shape is searched by the shape, color, size, etc. when viewed from a specific direction in two dimensions before matching with the three-dimensional CAD data to be searched, so that the object searched by high-speed processing can be searched. It becomes like this. Since the size and the like affect the shooting distance, it is necessary to specify and specify to some extent from the positional relationship between the current camera position and the area to be searched. When searching in two dimensions, there are several different forms depending on the typical viewing direction, so a plurality of two-dimensional reference data with different directions may be indicated together. As processing, a plurality of types of matching may be performed on a two-dimensional image. In this case as well, a rough candidate is found by the two-dimensional search function, and then the detailed position and orientation of the object are searched by the three-dimensional search function, so that the total search time can be further increased. become. In general, a distant image when the shooting distance is far away does not make a big difference between the left and right stereo images. Therefore, when a distant object is processed at high speed with a two-dimensional search function, Because the difference in the images of the two becomes large and there is a risk of misrecognition when a two-dimensional search is performed, the measurement accuracy by the camera is improved. Is good. Whether the search is a two-dimensional search or a three-dimensional search may be automatically switched depending on whether the search object is near or far. In addition, instead of specifying the processing area around the predetermined three-dimensional place with the extended commands 1 to 4, only the direction information obtained from the image information from the two-dimensional camera is specified for the sake of simplification. In the designated direction, the direction may be designated basically by a method of roughly designating the range of the distance from the near side to the far side or the near side and the far side.

  The extended commands 5 and 6 have a more general two-dimensional search command and a three-dimensional search command as a search function, and when a search command is first issued by a two-dimensional search command, Next, when a three-dimensional search process is performed, the corresponding two-dimensional detailed search is performed by taking over the previous two-dimensional search data based on the previous two-dimensional search result. The data searched as candidates in the two-dimensional processing is taken over as a search candidate object when starting the three-dimensional search, and the three-dimensional search processing is performed. In addition, as a two-dimensional search function, when an object to be searched is known in the three-dimensional CAD data, the position where the camera wants to find the target by photographing it from which position If there are one or more candidates, generate a two-dimensional image consisting of two-dimensional feature points and feature lines on how to capture the object to be searched from those candidate positions. Then, the matching between the image image and the feature point and feature line of the actually photographed image is performed to perform a matching process for two-dimensional search processing and positioning control (detecting a deviation amount and responding as a response). May be performed.

  Further, when a two-dimensional search command is executed by a plurality of imaging devices such as stereo, the two-dimensional search processing may be performed by processing all image information of imaging devices with different viewpoints little by little. For example, in the case of a two-lens stereo imaging device, the image information of the left imaging device is usually processed, and the search target is not included in the image of the left imaging device, or the target is in front. When an object that is obstructed is captured and the object is not captured, the image processing processor is set in parallel by the number of image capturing devices by automatically switching multiple image capturing devices such as switching to the image information of the right camera. Thus, the image processing of the two-dimensional search command may be performed using images of a plurality of imaging devices with a simple configuration without being driven.

  When performing a two-dimensional or three-dimensional search process, it is natural that the object to be searched has a different shape depending on the direction in which the object is viewed, but the appearance changes if the object is deformed even if the object is in the same direction. . In such a case, since it is difficult to prepare 2D or 3D reference CAD data in consideration of the deformation in all cases from the beginning, it is assumed that the deforming coupling condition is provided as the reference CAD data. To.

  Examples thereof will be described with reference to FIGS. FIG. 112 shows an example of the case where the CAD data connection relationship is described as data, which describes the basic positional relationship between each of the movable parts Part1 and Part2, the operation axis (directly rotating), and the operation range. is there. FIG. 113 shows a mobile phone as an example of a movable photographing object. The component 18a is coupled to the component 18b of the main body by an antenna, and the coupling condition is set in the z direction from a predetermined basic positional relationship. The main body 18b and the main body 18c expand and contract in the range of Z1 to Z2, and rotate by θ1 to θ2 around the x axis. When searching for such reference data, first, a representative shape is selected. The same is true when selecting a viewpoint for two-dimensional search, but a viewpoint with a typical characteristic is selected. For those whose shape changes, some typical patterns are narrowed down from among the deformed patterns. In the case of this cellular phone, (a), (b), and (c) in FIG. 114 are representative shapes. In this case, reference CAD data is usually generated from knowledge data that the object often has such a shape. Knowledge data may be automatically input and recorded by providing a learning function for the robot, but first, there is a method in which a person inputs in advance. When the three patterns (a), (b), and (c) in FIG. 114 are generated, suddenly, instead of issuing a search command with three three-dimensional CAD data, first, a two-dimensional search process is performed. To do. It selects several directions that look like body shapes for the three patterns in FIG. 114, and generates two-dimensional reference CAD data when viewed from several viewpoints. The image data may be searched for matching at the image data level. Two-dimensional CAD data extracted in two dimensions (in this case, binarized shape data is also referred to as two-dimensional CAD data). May be matched with two-dimensional reference CAD data. Matching between CAD data is performed after matching the overall size. The two-dimensional matching in consideration of the movable state of the component is a two-dimensional figure matching because it is a basic contour matching and a typical contour shape matching. Further, when the CAD data has a mark on the surface or the CAD data has pattern color information, the color boundary line segment described above is extracted, and the boundary line segment color information comparison or line is extracted. The same processing method as the means for obtaining corresponding points, such as the connection of minutes, may be taken in and matching processing for patterns and the like may be performed. The matching here is a matching of a plurality of reference patterns viewed from a representative form and a representative viewpoint, and thus is a rough matching. Since it is approximate, the matching degree is not so high, or those having a high matching degree are narrowed down as search candidate objects. Then, with respect to the narrowed candidate object, the direction of the viewpoint and the condition for deformation with higher matching degree are searched while gradually changing the position of the viewpoint and the amount of deformation at the next stage. Finally, the position and orientation of the target object and the movable state of the object having the movable part can be obtained with high accuracy by a three-dimensional search matching command.

  By using rough matching at first and using a two-dimensional matching function, the overall search time can be further increased. Here, a function for generating a two-dimensional reference figure, a function for estimating the size, etc. may be provided entirely in the visual information processing apparatus, or a part of the function may be provided on the control apparatus side on the application side. You may consider it, and you may think about the function division which is easy to construct as an application. Also, there is a possibility that parts can move, for example, rotating tires etc., because the pattern of stationary tires and the pattern captured as image data by shooting with a normal camera are different, It is a tire that is rotating if it is registered in advance as CAD data combining conditions or other data to be such an object and is referenced even when the pattern does not match even when the pattern does not match. It may be possible to recognize this. Further, when a square object that is not round like a tire rotates, the shape of rotation when the object rotates may be generated from information on the axis of rotation and matched.

  The extended command 7 is an example in which a determination function is also provided in the visual information processing apparatus in order to further speed up the processing. In one example, whether the target object ID1 and the target object ID2 collide, It is an example of a command in the case of providing a function for predicting the time until a collision when a collision occurs.

  FIG. 115 shows the concept of movement of the object 20a1 and the object 20b1. In this example, the moving object 20a1 moves to the position 10a2 after a certain time, the object 20b1 moves to the position of the object 20b2, and further moves to the positions 20a3 and 20b3 after the same time. Here, since the moving direction and the magnitude of the object 20a1 in each three-dimensional coordinate are changed to the vectors δa1 and δa2, if the motion up to the next predetermined time is predicted, the object 20a1 further moves by δa3. Can be predicted. Since the moving direction and the size of the object 20b1 are also changed to the vectors δb1 and δb2, if the movement up to the next predetermined time is predicted, it can be predicted that the object 20b1 further moves by δb3. From this expectation, it is checked whether two objects collide. Further, if the further moving direction is estimated on the extension, the time until the collision can be estimated.

  FIG. 115G shows the movement of two objects on a two-dimensional image. In order to speed up the processing in this case, a command that can be determined in a two-dimensional process may be prepared and determined on a two-dimensional image. The vectors δa1, δa2, etc., which are three-dimensional movement trajectories, are amounts corresponding to the movement trajectory vectors δaXY1, δaXY2 on the two-dimensional image. Otherwise, it is possible to make a rough judgment even at the stage of two-dimensional processing if the gap does not collide while it is still open. In addition, when the predicted collision position is first obtained in three dimensions and the position is moved to the two-dimensional image, it can be determined more accurately depending on whether the object on the current image has reached the planned collision position. As a result, the time until the collision can be predicted with higher accuracy on a two-dimensional image. Also, confirm the relationship between the 3D and 2D in advance and set the 2D condition corresponding to the 3D (for example, if there is a collision between this part of the 2D image of this object and this part, do not collide Etc.) can be confirmed in advance with three-dimensional data, and the relationship is made to correspond on the two-dimensional image data, and when monitoring, the two-dimensional image is monitored at high speed, You may make it feed back to a control signal.

  In addition, as an extended command, after the command is given, instead of the absolute position and posture information of the imaging device (robot itself), the displacement and posture change amount, or the recognized object ID movement amount and posture change amount, etc. A response may be output, or a predetermined threshold value may be set for the amount of change and exceeded, and a response to that effect may be output. As described above, these extended commands may be prepared as easy-to-use commands for the control device 300 on the application side.

  As described above, if these extended commands are provided, CAD data can be generated and searched more easily at high speed. In addition, when the CAD data is created so as to correspond to the data of the electronic tag system when the reference CAD data is generated and the search is actually performed, the CAD data is managed using the information of the tag system. The target object may be searched. Push and deform, hit and listen to the sound to recognize the type of object, and prepare (or learn) data about the object in advance, for example, if you hit a metal, it will make a high sound, If it is rubber, it will be deformed when it is pressed, and if it is water, it will be deformed and the reaction force will not be detected. It would be nice to be able to use it to find out if the unknown object is a metal, rubber, or water. In addition, when an IC tag is attached to the searched object, the physical characteristics of the object are written in advance on the IC tag so that the read information can be incorporated into the generated CAD data. Also good. In addition, when a plurality of IC tags with different purposes are attached to one object, for example, when an IC tag is attached for the environment recognition system, it may be doubled with an IC tag previously attached for another purpose. In such a case, the identification number of the IC tag already attached to the same object is written in the tag information to be attached later, and it is an object with such another tag. Information that another tag has may be used as necessary.

  Further, as an extended command, a command for detecting the presence or absence of eye contact may be prepared as a command to be used next after a person, animal, or robot is recognized. As processing, the eye or camera position corresponding to recognition and the part CAD data are searched from the reference CAD data of the recognized person, animal or robot and the individual part combination data, and the direction of the line of sight is the current direction. Whether or not the recognition target is looking at this. In addition, if something with a brightness level higher than the set level shines, or if an object of a certain color enters the field of view, the position is measured three-dimensionally and the coordinates of the measurement result By preparing an extended command that outputs (X, Y, Z) as the response result of the command, if you search for a dangerous object and find it, you should protect the robot's own view from that direction anyway. It is also possible to take a simple action. If such processing can be left on the visual information processing device 100 side with a single command until the result is obtained, the processing load of the main control device 300 and the complexity of the exchange of results and commands are eliminated. Can do.

  Further, as an extended command, when there are no obstacles in the surroundings, CAD data is generated roughly, and when there are obstacles particularly near the surroundings (this may specify coordinates, For example, if a command for switching to generate CAD data in detail is prepared, for example, while the robot is moving, obstacles around it may be detected. When moving in a wide area, a wide range of CAD data is generated roughly, and when moving slowly near an obstacle, the position of the obstacle is accurately detected while always detecting the collision of the obstacle. You can easily use it to do it well.

  Prepare a command called interference check as an extended command, and if it is a target object to be checked for interference, if it is a robot, input the shape data of the robot itself and angle information of each joint at that time as command data, and enter the surrounding environment If there is a possibility of collision by performing interference chucking, as a response, the alarm signal and the collision site (where and where in the environment will collide), the current clearance amount, You may make it return time. At that time, the desired robot control data for better checking the interference to the robot to guide the position of the robot and the orientation and position of the imaging device in a direction that facilitates the interference check (the robot position, Specific control commands for actually controlling the robot based on rough data such as posture, position and orientation of the imaging device, etc. May be output as part of the response information.

  Also, for various commands related to the search function, if the desired robot control data is output as information related to the movement request of the imaging device in a direction that makes it easier to search, the location where you want to search with the shadow of an object cannot be seen It may be possible to efficiently search in cooperation with the robot control device, such as when light is reflected and the place to be searched cannot be seen, or when it is necessary to approach in order to search in more detail. Also, when the imaging device is mounted on the robot's hand, specify the robot's hand or foot reference shape data in advance, or if the robot has a tool, specify the tool's reference shape data And the purpose of the work (for example, specifying the target of the imaging device itself, the tip of the foot, or the hand, or making the hand face the coordinates X, Y, Z of the object ID of the environment, or the designated recognition target Of the robot for achieving the purpose by designating the object ID of the robot to be always included in the image pickup apparatus screen or by causing the designated object ID to follow in a predetermined positional relationship after the designated object ID). Control guidance information may be output.

  Since the visual information processing apparatus is configured to be able to access the robot shape data and the environment data, the response is in the X, Y, Z directions and around the X, Y, Z axes in the coordinate system of the imaging apparatus or object ID. It is possible to respond as guidance control command information with a speed command in the direction of rotation of the robot. If the motion joint structure on the robot side is known, the command information is decomposed into control signals for each motion axis and By providing a function capable of directly generating control data, it is possible to rationally configure a robot control system.

  In addition, if 2D and 3D reference databases are enhanced, for example, if a car running on the road can be recognized and specified, the location information of the driver's seat of the car is also referred to in the data. For example, if you prepare multiple reference data for typical men and women, you can drive by matching with them by generating CAD data around the driver's seat and performing two-dimensional or three-dimensional matching processing. It is possible to recognize whether it is a man or a woman, and by preparing personal reference data and performing more detailed matching processing, it makes it possible to recognize who is driving Control can be easily performed by the robot.

  In addition, when searching for the victim from the sky, if the reference data of the victim is not easy in advance, for example, ask the person who has the same appearance as the victim to cooperate. It is also possible to search for a victim by generating 2D and 3D reference data in various situations and postures. For this, an operation method may be used in which an image pickup device is mounted on a car, an airship, or a helicopter to search while moving, without using a robot. In combination with another sensor such as an infrared camera or X-ray CT at the time of exploration, it may be linked so that a place or direction corresponding to a part detected by these sensors can be searched with high accuracy.

  In addition, 3D measurement is used instead of stereo measurement for tracking and shape search in combination with 3D laser measurement for distance measurement or 2D distance sensor using CCD etc. that can obtain distance image and visible image. You may make it do.

  The CAD database may be associated with the shape information of various recognition IDs, and the data may be managed by associating the name, weight, and various information. For example, in the case of a bird, the type and size of the bird (this) If representative shape data is registered for each size), the sound data of bird calls (such as a sample recording of a time-series audio signal for a certain period of time) is registered for each data. A voice recognition function is also added to the information processing device. If it is difficult to identify the type of target object (bird) that is currently being searched for and recognized by the imaging device using only shape data, a microphone, etc. If you acquire voice data of bird calls acquired by another sensor from, and input the time-series data sampled for a predetermined time as reference data, the audio data will also be in the database Performs voice information matched in, in combination to a specific processing types of birds currently recognized, may be more can precisely recognize the object.

  As an extended command, once a specific target object is recognized, the object ID is designated, and even when the object ID moves, it is continuously recognized, and the position and orientation in the three-dimensional space are continued. A command for responding may be provided. Even when the moving object is out of the field of view of the imaging device or when the recognition process continued due to the movement behind the obstacle in front is interrupted, as shown in FIG. The visual information processing side may automatically estimate the place to be continuously searched for recognition processing. If it is still lost, respond with the result.

  Further, as an extended command, a search range and two-dimensional or three-dimensional reference CAD data may be designated, and the number of similar shaped objects, patterns, etc. may be counted and the counted number may be made to respond.

  Also, what has been identified once in the past and whose ID has been specified is a specific name (number) given to the object by designating the ID instead of CAD data or various data corresponding thereto. Data to be referred to by ID may be automatically referred to on the visual information processing side. Further, what is to be detected may be determined in advance, the detection range and the content to be detected may be designated by a command, and the detection result may be returned. For example, you can specify the location of the faucet of the water supply, specify whether or not water is coming out of the faucet, specify the location of the outlet, and detect whether or not the plug is inserted into the outlet. If so, the position may be outlined, and whether or not a lamp or signal is attached, what color, etc. may be responded.

  As described above, as an extended command, a valid command for the application side including AI-like processing may be determined in advance, and various commands may be prepared. In addition, by registering in a database of typical motion patterns (motion), and determining whether the motion of the recognition target is the same as the motion registered in the database, what action the recognition target performs It may be possible to make a response output of the type of motion registered in the database in advance (for example, the motion of a person walking, the motion of a long-lived person, the motion of bending and stretching). The reference pattern data of the operation is expressed as three-dimensional CAD data with time data, and the time axis scale is variably adjusted within a predetermined range registered for each operation pattern, so that time series Correlation of motion patterns of CAD data, specifically, matching is performed for each scene, and the current pattern is taken in the reference pattern data based on the matching degree of posture and the matching degree of the changing posture in time series order. If the movement of the recognition target in the image information matches a predetermined threshold value or more, it may be determined that the operation of the reference pattern is performed. Instead of describing it in dimensional CAD data, describe it in a time-series feature amount change pattern of movement features and trends (eg, hand movement magnitude, speed, position, etc.). It may be determined to be the operation of which reference pattern when the feature quantity of sequence to extract a comparison of a recognition object image. At this time, the database registers the feature quantity of the three-dimensional motion, and when performing the matching process, the reference data is obtained from the positional relationship between the recognition target and the imaging device, and the change pattern of the feature quantity on the two-dimensional image is obtained. Matching of the feature amount change pattern may be performed at high speed by two-dimensional processing. In addition, the command does not specify the search target, and the image information input from the imaging device is automatically matched with the reference CAD data registered in the database, and the matching degree is increased. A command may be prepared so as to respond to the object ID at that point in time for a reference image with a high result. There may be a plurality of ID outputs of recognized objects, and the position and orientation information may be output together with the ID. In addition, as a method of registering a lot of reference data in the database, by associating data that may exist together, and by associating with location information such as where it is, At present, it is possible to narrow down the number of reference data to be subjected to matching processing from information on the location where the image capturing apparatus is photographing, and to perform processing more efficiently.

  74 to 91 exemplify basic block diagrams in the case where the first to fifteenth means of the present invention are implemented by an LSI circuit or a circuit of a substrate combined with an LSI. 92 to 99 show examples of timing charts at that time, and FIGS. 100 to 111 exemplify basic processing flowcharts for explaining the operation of each circuit. These examples are just examples, and there are various ways to incorporate functions, circuit combinations, timing, etc., so consider the constraints such as circuit scale and development costs according to the purpose. Then it is good to deform.

  In FIG. 74, image information (video signals) from the imaging devices 10 and 20 is taken into the respective image memories by the A / D conversion circuits 15 (a) 0 and 15 (a) 33, and the horizontal feature point extraction circuit 16 ( a) Line segment generation by extracting feature points of color boundary line segments by 1, 2 and 16 (a) 21 and 22 and vertical feature point extraction circuits 16 (a) 3, 4 and 16 (a) 23 and 24 (1) to (4) 16 (a) 5 to 8 and line segment generation (1) to (4) 16 (a) 25 to 28 generate line segments of two-dimensional CAD data, respectively, and integrate line segments (1) to (4) 16 (a) 9 to 12 and line segment integration (1) to (4) 16 (a) Line segments of image information captured by the left and right imaging devices 10 and 20 Are integrated and stored in each memory. In order to increase the operation speed, each processing circuit is configured so that two horizontal and vertical feature point extraction circuits can be processed in parallel, and line segment generation and line segment integration can be processed in four each. The number of circuits to be processed in accordance with the processing speed may be planned. Line segment generation is divided into areas on the screen and performed at high speed by para processing, so there are line segments that connect across each area, so they are connected across those areas. The process of connecting the line segments as the same line segment is executed in the circuit of line segment integration (1) to (4).

  In addition, as preprocessing for stereo measurement, point extraction (1) to (n) 16 (a) 12 to 14 extract a symmetric point from the line segment data memory of the right image. For example, the point data of the start point and end point of the line segment are extracted and the point data is written into the 2D point memory.

  Next, the S candidates (1) to (n) circuits 16 (a) 15 to 17 extract stereo corresponding point candidates from the line segment data memory of the left image. In stereo measurement, since a corresponding point exists on one line, candidate points in the left image on the line are extracted.

  Corresponding searches (1) to (n) circuits 16 (a) 18 to 20 narrow down corresponding point candidates using a vector matching process or the like. As a result of narrowing down, when the corresponding point candidates are narrowed down to one, the left and right points are output to the S corresponding point memory. Here, since the processes of the correspondence searches (1) to (n) take longer than other processes, n circuits that can be processed in parallel are prepared.

  Memory transfer (1) 16 (a) 10 confirms the contents of the tracking flag memory, and when the transfer flag is ON, the contents of the line segment integrated memory (left image) are T-measured (left image). ()) Transfer to line integrated memory. This is a process for temporarily storing the processing result of the left image in order to perform 3D measurement by tracking. The tracking flag memory becomes a transfer ON flag at the beginning, when losing tracking and restarting, or after measuring by tracking. Memory transfer (1) 16 (a) 11 is a process for temporarily storing the left line segment integrated data in the previous (left image) line segment integrated memory for tracking.

  The left and right corresponding point data written to the S corresponding point memories (1) to (4) are the circuits 16 (b) 1 to 4 of the following 3D measurement (1) to (4) from FIG. Are read and stereo-measured, and the result is written to the 3D point memories (1) to (4). Next, 3D measurement is also performed on the line segment connected to the point measured in 3D line generation (1) to (4) by the circuits 16 (b) 5 to 8 to generate CAD data of the three-dimensional line segment. Write to the line memories (1) to (4). Also, here, in order to speed up the processing, the same four circuits are operated in parallel, so the line segments generated in each may be connected. The process of integrating the two line segments into one is performed by the circuit 16 (b) 9 of 3D line integration (1), and the integrated 3D line segments are output to the 3D line segment integration memory. The 3D line segment integrated memory output here is 3D line segment data in the coordinate system of the left camera.

  FIG. 76 shows a circuit portion for tracking processing. Extraction of points that are characteristic of the color boundary line segment of the start point, end point, branch point and acute point from the previous (left image) line segment integration memory, as in the case of stereo measurement, point extraction (1) to Performed by the circuits 16 (c) 1 to 3 of (n) and output to the 2D point memories (1) to (4). Next, tracking corresponding point candidates are extracted from the line segment integrated memory (left image) by the circuits 16 (c) 4 to 6 of the T candidates (1) to (n). Next, the candidate points are narrowed down by using the vector matching process or the like in the circuits 16 (c) 7 to 9 of the correspondence searches (1) to (n), and the corresponding point candidates are stored in the T corresponding point memories (1) to ( Write to 4) and output. Here, the processing of correspondence search (1) to (n) can use the same processing circuit as in the case of stereo measurement. However, since it takes time similarly, the number n of circuits to be processed in parallel should be sufficiently increased. .

  Next, the corresponding point candidate of tracking is read from the input (B) of FIG. 77, and the movement amount is calculated by the circuits 16 (d) 1 to 4 of the movement amount calculation (1) to (4). ) To (4), and finally, three corresponding points of tracking considered to be most correct in the circuit 6 (d) 5 for stability evaluation are selected, and the movement amount by the three points is set as the determined movement amount. Determine and write to the determined movement amount memory. If it cannot be determined, it means that the tracking has been lost, and NG is written to the T continuation flag memory. When tracking is continued, the ongoing flag is written in the T continuation flag memory. In the stability evaluation here, when a 3D point measured in 3D with a candidate movement amount is converted into the previous coordinate system, a movement amount that matches many 3D points is selected as a more correct movement amount.

  Next, in FIG. 78, the 3D line segment data obtained in the 3D line segment integrated memory (left camera coordinate system) is converted into a 3D line segment integrated memory (world at the start of tracking) by the coordinate conversion circuit 16 (e) 1 by tracking. Coordinate system). Next, in consideration of the magnitude of the error of the measured point and the shape of the error, the newly measured point and line segment data are added, and the process of updating the one with a smaller error is updated processing 16 ( e) In step 2, the entire 3D line segment in continuous tracking expressed in the world coordinate system at the start of tracking is written into the integrated memory. Next, when the first tracking is interrupted and resumed, the circuit 16 (e) 3 performs coordinate conversion for converting to the first world coordinate system when the tracking is interrupted. The converted point and line segment data is written to the integrated memory of the entire 3D line segment expressed in the first world coordinate system. If the same point exists, processing such as updating to a point having a high overall accuracy and a small error shape or adding an error caused by this conversion is also performed. Since the world coordinate conversion 16 (e) 3 cannot be converted in a state where the current position is lost due to tracking, processing is performed when the coordinate conversion matrix is not determined in the 3D determination movement amount memory with reference to the 3D determination flag memory. Do not do.

  79 and 80 show circuit blocks that perform processing for determining a corresponding transformation matrix when tracking is lost. The circuits 16 (f) 1 to (3) of the 3D points (1) to (n) extract 3D points from the line segment data in the first world coordinate system of the entire 3D line segments. As a reference for extraction, a range that is likely to move in the vicinity of the lost place is considered in consideration of the amount of movement due to subsequent tracking, and 3D points in the range are extracted. Next, in the circuits 16 (f) 4 to 6 of the W candidates (1) to (n), 3D points are extracted from the 3D line segment data generated after resuming tracking corresponding to the extracted 3D points. As a criterion for extraction, 3D points existing in a range corresponding to the amount of movement after losing sight are extracted as candidates. Next, in the circuits 16 (f) 7 to 9 in the 3D searches (1) to (n), candidates are narrowed down for the corresponding points using a technique such as three-dimensional vector matching.

  Next, the circuits 16 (g) 1 to 4 of the 3D movement amount calculation (1) to (4) calculate the movement amount when the combinations of the narrowed-down corresponding point candidates are shared and their correspondence is correct. To the 3D movement amount memory. Finally, the movement amount (coordinate conversion coefficient) determined by determining the combination in which many points coincide with each other more accurately by the circuit of correspondence evaluation 16 (g) 5 is written in the 3D determined movement amount memory. Also, the determined flag is written in the 3D determined movement flag memory. In the process of finding candidate points from among the 3D points, the longer the time from losing sight, the greater the amount of movement that follows, so the number of candidate points increases. Since the number of points may exceed the processing capability, in such a case, the processing may be divided into several area blocks and processed in several times. In addition, when CAD data is first generated and lost, the amount of CAD data generated after that increases, and the amount of CAD data generated before the first loss is small. However, in such a case, it is possible to give up the connection to the CAD data generated before losing sight first, and to make the world coordinate system whose generation is resumed later as the first coordinate system. . In that case, the previously generated CAD data may be discarded, or may be left in the hope that it may be connected sometime.

  FIG. 81 shows a circuit block of an input / output I / F with the outside of the entire circuit. The command input circuit 16 (h) 1 inputs a command command from the processing device 300 such as an external microcomputer from the input / output I / F unit and stores the contents of the command in the command storage memory. Similarly, the attached data input circuit 16 (h) 2 reads information necessary for executing a command attached to the command and writes it in the command attached data storage memory. The camera position output circuit 16 (h) 3 obtains information on the current position and posture of the camera obtained by the visual information processing apparatus 100 by performing 3D measurement and tracking from a memory in which the current position and posture of the camera are stored. The data is read and output to the input / output I / F unit so that it can be read by the apparatus 300 such as a microcomputer. Similarly, the search result output circuit 16 (h) 4 reads out a result when a command such as a search is executed from the stored memory and outputs it to the input / output I / F unit. The 2D-CAD data input / output circuit 1 (h) 5 reads the search 2D reference CAD data from the external memory 200 via the external memory 200 that can handle a large amount of data and the input / output I / F unit, and conversely, 2D CAD data that can be used later as search 2D reference CAD data and 2D-CAD data as a processing result such as search are output to the memory 200. Similarly, the 3D-CAD data input / output circuit 1 (h) 6 reads the search 3D reference CAD data from the external memory 200 via the external memory 200 that can handle a large amount of data and the input / output I / F unit. On the contrary, 3D CAD data that can be used later as 3D reference CAD data for search and 3D-CAD data as a processing result of search or the like can be output to the external memory 200.

  FIG. 82 shows a peripheral circuit of the tracking measurement determination circuit 16 (i) 1. The tracking measurement determination circuit 16 (i) 1 reads the state of whether tracking is continued from the T continuation flag memory, and also reads the movement amount of the camera by tracking from the determined movement amount memory. When the movement amount suitable for 3D measurement is reached, it is determined that tracking measurement is possible, and the timing for executing tracking measurement is determined. 3D measurement by tracking uses the current (left image) line integration data and the T measurement (left image) line integration data stored for tracking measurement in the same way as stereo measurement. In addition, the processing circuit includes circuits 16 (j) 1 to 3 of point extraction (1) to (n), circuits 16 (j) 4 to 6 of S candidates (1) to (n), correspondence search (1 ) To (n) circuit 16 (j) 7 to 9, 3D measurement (1) to (4) circuit 16 (k) 1 to 4 in FIG. 84, 3D line segment generation (1) to (4) circuit 16 (k) 5-8, 3D line segment integrated circuit 16 (k) 9 generates 3D line segment integrated data, and movement from the start of tracking in the coordinate conversion circuit 16 (l) 1 by tracking shown in FIG. The coordinate conversion of the quantity is performed, and the update process 16 (l) 2 and the world coordinate conversion 16 (l) 3 are applied to the first world coordinate system. That is to update the 3D line segment integrated data. When this tracking processing is performed, the measurement accuracy is improved with the same effect as stereo measurement with a wide camera interval, but because it is a series in terms of timing, the cycle time for capturing image information accordingly. Therefore, in the case of providing, it is also possible to devise such that the cycle time is not increased by performing the measurement by tracking at the same timing instead of the execution timing of the stereo measurement. Also, even if this function is not necessarily included, even if only stereo measurement is performed, the error shape is updated when the position of the viewpoint changes, and the accuracy of the inaccurate depth direction is compensated by stereo measurement from another angle. Since it can be used depending on the application system, it may be omitted.

  86 and 87 show block diagrams of a processing circuit for 3D search. The circuits 16 (m) 1 to 3 of the 3D points (1) to (n) extract 3D feature points such as a start point, an end point, an acute vertex, and a branch point in the search 3D reference CAD data. The circuits of W candidates (1) to (n) extract correspondence candidates from the integrated data of the entire 3D line segments of the first world coordinate system storing the generated 3D-CAD data. As a reference for extraction, if there is a command for specifying an area in the command and the command attached data, the area is extracted from the area. In particular, when an area specification to be specified is not given by a command, a 3D point is extracted from an area in a direction that enters the field of view from the current camera position and camera orientation. The circuit 16 (m) 7 of the 3D search (1) to (n) obtains corresponding point candidates between the 3D point of the extracted generated CAD data and the 3D point of the reference CAD data, and the W corresponding point memories (1) to (4). )

  The 3D similarity calculation circuits 16 (n) 1 to 4 share all the combinations of 3D point corresponding point candidates, calculate the 3D similarity to the para, and store the similarity calculation result in the 3D similarity memory (1 ) To (4). Then, the similarity evaluation circuit 16 (n) 5 finally determines the combination having the highest similarity among the corresponding point candidates. If a candidate having a sufficiently similar degree is found, the CAD data transfer circuit 16 (n) 6 is instructed in the order of the high degree of coincidence and stored in the 3D-CAD data storage memory as a result of processing such as search. As a result, how many candidates are found and the degree of matching of each candidate are output to a processing result memory such as a search.

  88 and 89 show block diagrams of circuits for executing a two-dimensional search command. The circuits 16 (o) 1 to 3 of point extraction (1) to (n) extract 2D feature points such as a start point, an end point, an acute vertex, and a branch point from the search 2D reference CAD data. The search candidates (1) to (n) circuits 10 (o) 4 to 6 extract correspondence candidates from the current line segment integrated data of the left image. As a reference for extraction, if there is a command for specifying a direction or an area in the command and command-attached data, the direction or area is extracted. In particular, when the specified direction or area designation is not given by the command, it is extracted from all the line segment integrated data of the current left image. Corresponding searches (1) to (n) circuits 16 (o) 7 to 9 obtain corresponding point candidates between the extracted 2D point of the left image and 2D point of the reference CAD data, and search corresponding point memories (1) to ( Write to 4).

  The 2D similarity calculation circuits 16 (p) 1 to 4 share all the combinations of 2D point corresponding point candidates, calculate the 2D similarity to the para, and store the similarity calculation result in the 2D similarity memory (1 ) To (4). Then, the similarity evaluation circuit 16 (p) 5 finally determines the combination having the highest similarity among the corresponding point candidates. When a candidate having a sufficiently similar degree is found, the CAD data transfer circuit 16 (p) 6 is instructed in the order of the high degree of coincidence and stored in the 2D-CAD data storage memory as a result of processing such as search. As a result, how many candidates are found and the degree of matching of each candidate are output to a processing result memory such as a search.

  90 and 91 show circuit blocks for generating search 2D reference CAD data. The 2D projection circuit 16 (q) 1 for the 3D line segment generates 2D reference CAD data when viewed from the direction based on the command command from the 3D-CAD data generated up to now, and searches for 2D reference CAD. Write to data storage memory. The reference CAD data to be created may be 2D reference CAD data viewed from a plurality of viewpoints. This command is used when it is desired to search, position, etc. again for already generated CAD data.

  The 2D projection circuit 16 (r) 1 for the 3D line segment generates 2D reference CAD data when viewed from the direction based on the command command from the given 3D reference CAD data for search, and 2D reference CAD for search. Write to data storage memory. The reference CAD data to be created may be 2D reference CAD data viewed from a plurality of viewpoints. This command is used when searching for an object of designated reference CAD data by 2D matching faster than 3D matching.

  92 to 99 show the operation timing of each block circuit described above.

Starting from the A / D circuit 16 (a) 0, the A / D circuit 16 (a) 33 in FIG. 92, and the command inputs 16 (h) 1 and 2 in FIG. 96, each block circuit is in turn at a predetermined timing. Finally, the camera position output 16 (h) 3 and search result output 16 (h) 4 in FIG. 96 are obtained, and the operations in the same order are repeated thereafter. FIG. 98 shows the state of use of the integrated memory of the entire 3D line segment of the first world coordinate system, and FIG. 99 shows the state of use of the line segment integrated memory of the left image. Since the access to the memory is not overlapped, the timing is taken and the entire cycle, that is, the A / D circuit 16 (a) 0 and the A / D circuit 16 (a) 33 are imaged. The cycle from the acquisition of information to the acquisition of the next image information is devised so as to be as short as possible. The order of the overall timing is determined so that the data in the memory to be used is updated before the circuit of the block is operated. Although the time length of one operation timing shown here is illustrated at the same interval, the length of the actual time varies from the actual processing time. In this example, the number of parallel processing of the circuit is described as four or n so that the processing can be efficiently performed at high speed as a whole. Try to determine the number.
100 to 111 show basic processing flowcharts of main processing circuit blocks. FIG. 100 shows a basic flowchart of 16 (a) 1 horizontal feature point extraction (1). 92. At the start timing of the timing chart of horizontal feature point extraction (1) 16 (a) 1 in FIG. If the direction is specified in the generation area, the processing areas (IX1 to IX2, IY1 to IY2) on the image are specified from the processing direction.

  Next, in processing 16 (t1) b, one horizontal luminance (RBG luminance in the case of color) data in the processing area is read from the image data, and then in one processing 16 (t1) c, one horizontal luminance information Are examined in order from the edge, and the inflection point (intermediate position from the point at which the change in inclination suddenly changes upward to the point at which the inclination changes suddenly downward, or from the point at which the change in inclination changes downward suddenly to the next The position of the intermediate position up to the point where the point suddenly changes is obtained, and stored in the point memories (1) to (4) in order using the address coordinates on the image memory.

  Next, it is determined whether or not the process has been completed up to the opposite end of one horizontal line in the determination process 16 (t1) d. In this case, the process is repeated from the process 16 (t1) c.

  Further, in the process 16 (t1) e, it is determined whether or not the horizontal luminance data has been completed for all of the vertical processing areas IY1 to IY2. Then, a function that ends when all the ranges are completed is incorporated in the circuit.

  FIG. 101 shows a basic flowchart of the correspondence search (1) of the processing block 16 (a) 18. In process 16 (t2) a, the points P (i) extracted from the first right image from the candidate point memory (1) and the corresponding point candidate points KP1 to KPn previously extracted from the corresponding left image are read. Next, in step 16 (t2) b, a vector around the right image at the point P (i) is obtained for each region delimited by the color boundary lines. Here, it is obtained including the color information of the vector tip. Further, the vector may be obtained at the time of point extraction, stored in the memory, and simply read out from the candidate point memory (1).

  Next, in processing 16 (t2) c, a vector around the left image of the corresponding point candidate point KPj (including the color information at the tip) is obtained for each region delimited by the color boundary line. Then, in process 16 (t2) d, the vector of the point P (i) and the vector of the corresponding point candidate point KPj are compared for each region delimited by the color boundary line to obtain the matching degree. Then, in the determination process 16 (t2) e, the corresponding point candidate KPj determines whether all the combinations of 1 to n have been completed. If No, the process is repeated from the process 16 (t2) c. In the case of Yes, in the process 16 (t2) f, the vector matching degree in one region is high from the corresponding point candidates KP1 to K for the point P (i), and the matching degree of the second candidate point is higher than a predetermined degree. If there is a point with a high degree of matching, it is written in the S corresponding point memory (1) as a combination of corresponding points PofL (i) and PofR (i) of stereo measurement.

  Then, in the discrimination process 16 (t2) g, it is discriminated whether or not all the extraction points P (i) have been completed. Incorporate into the circuit a function that terminates after processing.

  FIG. 102 shows a basic flowchart of 16 (b) 5 3D line generation (1). First, in process 16 (t3) a, the command storage memory and the attached data storage memory are referred to, and if the direction depth is specified in the CAD data generation area, the processing area (distance range) for generating the 3D line segment from the depth is specified. To identify. Next, in step 16 (t3) b, the point 3DP (i) measured three-dimensionally from the 3D point memory (1) and the corresponding point information PofL (i) and PofR (i) of the left and right images are read. Then, in the next determination process 16 (t3) c, it is determined whether the depth of the three-dimensional point 3DP (i) is a 3D point in the designated generation area. ) D and process 16 (t3) e are passed. In the case of Yes, on the line segment connected to the corresponding point information PofL (i) of the left image corresponding to the point on the line segment connected to the corresponding point information PofR (i) of the right image in the process 16 (t3) d. A point is searched while checking the degree of correspondence, and if there is a corresponding point, a 3D point is also obtained by stereo measurement, and a 3D line memory (1) is obtained as line segment data connected to the first 3D point 3DP (i). ) Then, a point on the line segment connected to the corresponding point information PofL (i) of the left image corresponding to the point on the line segment connected to the corresponding point information PofR (i) of the processing right image is searched while checking the degree of correspondence. If there is a corresponding point, the 3D point obtained by stereo measurement is also obtained, and is written in the 3D line memory (1) as line segment data connected to the first 3D point 3DP (i). The 3D line includes the coordinates in the world coordinates of the viewpoint O1 of the left image obtained by 3D measurement of the points constituting the 3D line and the left and right color information extracted as feature points of the color boundary line when viewed from the viewpoint. Write as information attached to the point information of the memory (1). Here, when the CAD data generation time is referred to in the subsequent processing, the time data taken by the imaging apparatus is written as attached information as information attached to the point information.

  Then, it is determined whether or not all 3D-measured points 3DP (i) in the 3D memory (1) have been processed in the determination process 16 (t3) f. Then, a function that ends when all 3D measured points 3DP (i) in the 3D memory (1) have been processed is incorporated in the circuit.

  FIG. 103 shows a basic flowchart of the movement amount calculation (1) of 16 (d) 1. First, in Step 16 (t4) a, points PofBL (i) i = 1 to n extracted from the previous (left image) line segment from the T corresponding point memory (1) and corresponding (left image). The tracking corresponding point candidates KPofL (i, j) j = 1 to m narrowed down from the line segment and information related to the matching degree when the tracking corresponding point candidates are narrowed down by the correspondence search are read. Next, the points PofBL (i) i = 1 to n extracted from the previous (left image) line segment in the process 16 (t4) b and the tracking corresponding point candidates narrowed down from the corresponding (left image) line segment. KPofL (i, j) j = 1 to m of combinations n × m of 3 is selected from the first to the total of the total number of combinations in a predetermined order (4 is the number of circuits to be processed in parallel) ) One combination S1 (one of PofBL (i) and KPofL (i, j)) in the second combination, S2 (one of PofBL (i) and KPofL (i, j)), S3 (one of PofBL (i) and KPofL (i, j)) is set to the next processing data.

  Then, in step 16 (t4) c, 3D coordinate data in the world coordinate system at the time of tracking start at the point PofBL (i) i = 1 to n extracted from the previous (left image) line segment is read. . Then, the three points extracted from the previous (left image) line segment in processing 16 (t4) d, the three tracking corresponding point candidates narrowed down from the corresponding (left image) line segment, and the point PofBL (i) i The movement amount (coordinate conversion coefficient) of the imaging device 10a is calculated using the three-dimensional coordinate data in the world coordinate system at the time of the tracking start when 3D measurement of = 1 to n is performed, and the combination information S1, S2 is stored in the movement amount memory. , S3 and the movement amount are written.

  Next, it is determined in the determination process 16 (t4) e whether or not calculation of all 1/4 of the total number of combinations (4 is the number of circuits to be processed in parallel) has been completed. Repeatedly, a function that ends when all calculations are completed is incorporated in the circuit.

  FIG. 104 shows a basic flowchart of 16 (d) 5 stability evaluation. First, in process 16 (t5) a, points PofBL (i) i = 1 to n extracted from the previous (left image) line segment from the movement amount memories (1) to (4) and the corresponding (left image ) Tracking correspondence point candidates KPofL (i, j) j = 1 to m narrowed down from the line segment A combination of selecting 3 out of n × m combinations, and a movement amount when selecting 3 points of the combination Read the calculation result of.

  Next, in process 16 (t5) b, 3D point coordinate data is read from the 3D line segment integrated memory (left camera coordinate system), coordinate conversion is performed with all movement amounts (coordinate conversion coefficients), and the 3D line segment integrated memory is read. The number of 3D points that coincide with a point in the (world coordinate system at the start of tracking) with a predetermined accuracy or higher is obtained, and movement amount candidates with a large number of matching 3D points are narrowed down. Next, among the movement amount candidates narrowed down in the process 16 (t5) c, the movement amount candidate having the best degree of coincidence of the 3D points that are matched in the preprocessing is further determined as the determined movement amount to the determined movement amount memory. Write.

  When the determined movement amount is determined in process 16 (t5) d, the “continuing flag” is written in the T continuation flag memory, and there are few coincidence points or when the most cannot be narrowed down A function to finish writing the “continuation interruption flag” in the T continuation flag memory is incorporated in the circuit.

  FIG. 105 shows a 3D search (1) basic flowchart of block 16 (m) 7. First, in the process 16 (t6) a, the point PofRfCAD (i) extracted from the 3D reference CAD data storage memory for search from the 3D complementary point memory (1) and the corresponding integrated memory (first world) 3D corresponding point candidates KPoCAD (i, j) j = 1 to m extracted from the coordinate system) are read. When there are a plurality of search 3D reference CAD data, the subsequent calculations are performed separately for each type.

  Next, in a process 16 (t6) b, a vector to a point around the point PofRfCAD (i) is obtained including the color information at the tip of the vector. You may obtain | require for every 3D area | region. Alternatively, it may be obtained at the time of 3D point extraction and only read from the 3D candidate point memory (1).

  Next, in step 16 (t6) c, vectors (including the color information of the tip) to the points around 3D corresponding point candidate KPoCAD (i, j) j = 1 to m are obtained for each of j = 1 to m. . You may obtain | require for every 3D area | region. Next, in the process 16 (t6) d, the vector subtracted from the point PofRfCAD (i) and the 3D corresponding point candidate KPoCAD (i, j) j = 1 to m including the color information of the tip of the vector are compared. Find the matching degree. You may compare for every 3D area | region. When comparing the color information, the matching is performed in consideration of the left and right directions of the color boundary line from the position information of the viewpoint when the color information is registered.

  Next, in step 16 (t6) e, when the vector matching degree is high and there is a point having a matching degree higher than a predetermined value than the second candidate point, points PofRfCAD (i) and 3D having a high matching degree as corresponding points. Corresponding point candidates KPoCAD (i, j) j = 1 to m are written into W corresponding point memories (1) to (4). Then, it is determined whether or not all i = 1 to n of the points PofRfCAD (i) extracted in the determination process 16 (t6) f have been completed. If No, the process is repeated from the process 16 (t6) a. When the extracted points PofRfCAD (i) for all i = 1 to n are completed, a function for ending is incorporated in the circuit.

  FIG. 106 shows a basic flowchart of the 3D similarity calculation (1) of the block 16 (n) 1. First, in step 16 (t7), the points PofRfCAD (i) i = 1 to N extracted from the 3D reference CAD data storage memory for search from the aW corresponding point memory (1), and the entire 3D line segment corresponding to each point The 3D corresponding points (plurality) in the integrated memory (first world coordinate system) are read. The number of the plurality is ¼ of the total number (4 is the number of circuits to be processed in parallel). In addition, when there are a plurality of search 3D reference CAD data, the subsequent calculations are performed separately for each type.

  Next, in process 16 (t7) b, a vector from the point PofRfCAD (k) to another point PofRfCAD (i) i = 1 to N (excluding k) is obtained including the color information of the tip point. Here, k is selected from k = 1 to N where the corresponding point exists in the integrated memory (first world coordinate system) of the entire 3D line segment corresponding to the point PofRfCAD (k). .

  Next, in process 16 (t7) c, a vector from a point corresponding to the point PofRfCAD (k) in the 3D corresponding point to a point corresponding to another point PofRfCAD (i) i = 1 to N (excluding k) Including the color information of the tip point. Ask for multiple sets. Next, in process 16 (t7) d, a plurality of matching degrees between the vector group from the point PofRfCAD (k) and the vector group from the point corresponding to the point PofRfCAD (k) are obtained, and the calculation result such as the matching degree is 3D-like. Write to memory once. Further, when the scale factor is taken into consideration when obtaining the 3D corresponding point candidate KPoCAD (i, j), the matching degree is similarly obtained in consideration of the scale factor at that time.

  Next, it is determined whether or not all of k = 1 to N are completed in the determination process 16 (t7) e, or whether the search is sufficient. If No, the process is repeated from 16 (t7) b. Then, if everything is finished, a function to be finished is included in the circuit.

  FIG. 107 shows a basic flowchart of similarity evaluation of block 16 (n) 5. First, in process 16 (t8) a, the command is read from the command storage memory and the command attached data is read from the command attached data storage memory. Next, points PofRfCAD (i) i = 1 to N extracted from the 3D similarity CAD data storage memory for searching from the 3D similarity memories (1) to (4) in process 16 (t8) b, and the corresponding 3D A set of 3D corresponding points in the integrated memory (first world coordinate system) of the entire line segment and the 3D similarity calculation result at that time are read. In addition, when there are a plurality of search 3D reference CAD data, the subsequent calculations are performed separately for each type.

  Next, in the process 16 (t8) c, when the number of commands is specified in the command attached data in descending order of the similarity from the 3D similarity calculation result, the integrated memory of the entire 3D line segment (first Command for transferring CAD data related to the corresponding point in the world coordinate system) is sent to the CAD data transfer circuit and transferred to the 3D-CAD data storage memory as a result of processing such as search. Then, based on the coordinate data of the set of 3D corresponding points in the integrated memory (first world coordinate system) of the entire 3D line segment in the descending order of the similarity in process 16 (t8) d, the reference of the 3D reference CAD data for search The position and orientation in the first world coordinate system of the coordinate system are calculated and written into the processing result storage memory such as search.

  Then, in the determination process 16 (t8) e, it is determined whether or not the command includes other instructions such as a determination process. If No, the next process 16 (t8) f is not performed, and Yes is determined. In the case of (1), processing of the command content included in the other command is executed in processing 16 (t8) f, and the result is written in a processing result storage memory such as search. As an example, in the case of a command for determining whether or not the searched CAD data ID1 and the searched CAD data ID2 collide, the distance between the searched CAD data ID1 and the searched CAD data ID2 is obtained and the distance is obtained. If is less than the predetermined distance (use the value if it is given in the command attached data), search for “has no collision risk” if it is less, and “no risk of collision” in the other cases The process is such as writing to the result storage memory. Then, the function to be terminated is included in the circuit.

  FIG. 108 shows a correspondence search (1) basic flowchart of block 16 (0) 7. First, in process 16 (t9) a, the point PofRfCAD (i) extracted from the candidate 2D reference CAD data storage memory from the candidate point memory (1) and the corresponding line segment integrated memory (left image) are extracted. The corresponding point candidate KPoL (i, j) j = 1 to m is read. When there are a plurality of search 2D reference CAD data, the subsequent calculation is performed separately for each type.

  Next, in process 16 (t9) b, a vector to a point around the point PofRfCAD (i) is obtained including the color information at the tip of the vector (may be obtained for each color boundary region). It may be obtained at the time of 2D point extraction and only read from the candidate point memory (1).

  Next, in process 16 (t9) c, vectors (including the color information at the tip) to the points around the corresponding point candidate KPoCAD (i, j) j = 1 to m are obtained for each of j = 1 to m. You may obtain | require for every area | region divided by a color boundary line segment. Next, in step 16 (t9) d, the vector subtracted from the point PofRfCAD (i) and the corresponding point candidate KPoCAD (i, j) j = 1 to m including the color information of the tip of the vector are compared and matched. Find the degree. You may compare for every area | region.

  Next, in process 16 (t9) e, when the vector matching degree is high and there is a point having a matching degree higher than a predetermined value than the second candidate point, it corresponds to a point PofRfCAD (i) having a high matching degree as a corresponding point. M points that have been narrowed down from point candidate KPoCAD (i, j) j = 1 to m are written into corresponding point memories (1) to (4). Then, it is determined whether or not all i = 1 to n of the points PofRfCAD (i) extracted in the determination process 16 (t9) f have been completed. If No, the process is repeated from the process 16 (t9) a. When the extracted points PofRfCAD (i) are all completed for i = 1 to n, a function to be terminated is incorporated in the circuit.

  FIG. 109 shows a basic flowchart of the 2D similarity calculation (1) of the block 16 (p) 1. First, points PofRfCAD (i) i = 1 to N extracted from the search corresponding point memory (1) from the search corresponding point memory (1) in process 16 (t10) a and corresponding to each point (left image) ) Read the corresponding points (plural) in the line segment integrated memory. The number of plural pieces is 1/4 of the total number (4 is the number of circuits to be processed in parallel). Further, when there are a plurality of search 2D reference CAD data, the subsequent calculation is performed separately for each type.

  Next, in processing 16 (t10) b, a vector from the point PofRfCAD (k) to another point PofRfCAD (i) i = 1 to N (excluding k) is obtained including the color information of the tip point. k is selected from k = 1 to N where a corresponding point in the line segment integrated memory (in the left image) corresponding to the point PofRfCAD (k) exists. Next, in process 16 (t10) c, from the corresponding point in the line segment integrated memory (in the left image) corresponding to the point PofRfCAD (k) to the point corresponding to another point PofRfCAD (i) i = 2 to N. Including the color information of the tip point. Ask for multiple sets.

  Next, in process 16 (t10) d, a plurality of matching degrees between the vector group from the point PofRfCAD (k) and the vector group from the point corresponding to the point PofRfCAD (k) are obtained, and the calculation result such as the matching degree is 2D-like. Write to memory once. When the scale factor is taken into account when obtaining the 2D corresponding point candidate KPoCAD (i, j), similarly, the matching factor is obtained in consideration of the scale factor at that time.

  Then, it is determined whether or not all of k = 1 to N have been completed in the determination process 16 (t10) e, or whether the search is sufficient. If No, the process is repeated from the process 16 (t10) b. Include a function in the circuit that ends when everything is finished.

  FIG. 110 shows a basic flowchart of similarity evaluation of block 16 (p) 5. First, in process 16 (t11) a, a command is read from the command storage memory, and command attached data is read from the command attached data storage memory.

  Next, points PofRfCAD (i) i = 1 to N extracted from the 2D reference CAD data storage memory for search from the 2D similarity memories (1) to (4) in the process 16 (t11) b, and corresponding to ( A set of corresponding points in the line segment integrated memory (left image) and the similarity calculation result at that time are read. When there are a plurality of search 2D reference CAD data, the subsequent calculation is performed separately for each type.

  Next, in process 16 (t11) c, if the number is specified in the command attached data in descending order of the similarity from the 2D similarity calculation result, the integrated memory of the entire 3D line segment (the first A command to transfer CAD data related to a point corresponding to a corresponding point in the line segment integrated memory (of the left coordinate system) in the world coordinate system is sent to the CAD data transfer circuit, and 3D- Transfer to CAD data storage memory.

  Then, in the next process 16 (t11) d, the integrated memory (first world coordinate system) of the entire corresponding 3D line segment of the corresponding point set in the line segment integrated memory (in the left image) in descending order of similarity. The position and orientation in the first world coordinate system of the reference coordinate system of the reference 2D reference CAD data is calculated from the coordinate data of the set of points in and is written in the processing result storage memory such as search.

  Then, in the determination process 16 (t11) e, it is determined whether or not the command includes other instructions such as a determination process. In the case of Yes, the content of the command included in the other command is executed in processing 16 (t11) f, and the result is written in the processing result storage memory such as search. For example, in the case of a command for determining whether or not the searched CAD data ID1 and the searched CAD data ID2 collide, the distance between the searched CAD data ID1 and the searched CAD data ID2 is obtained and the distance is a predetermined distance ( (If the value is given in the command attached data, the value is used.) In the following cases, “with collision risk” is written into the processing result storage memory such as search. It becomes. Then, the function to be terminated is incorporated in the circuit.

  FIG. 111 shows a basic flowchart of 2D projection of the 3D line segment of the block 16 (r) 1. First, in process 16 (t12) a, the command is read from the command storage memory and the command attached data is read from the command attached data storage memory.

  Next, in process 16 (t12) b, 3D-CAD data is read from the search 3D reference CAD data storage memory. When 3D-CAD data is composed of a plurality of parts (ID1 to IDn), the data is managed for each part. Further, if there are coupling conditions for each component (ID1 to IDn), these conditions are also read.

  Next, at 16 (t12) c, if there is a range condition limited to the generation condition in the command and command-attached data, if there is no limit condition within the range, the reference CAD data is read from all directions. 2D projection CAD data of all kinds of various combination conditions of a plurality of parts by combining the movement amounts of all the joint conditions of the above, and various combination conditions of the plurality of parts by combining the observation direction and each movement amount of the joining conditions Is written into the search 2D reference CAD data storage memory for each type. For the number of divisions in the observation direction and the movement amount of the coupling condition, the values are used if specified in the command attached data, and if not, the predetermined values set as default are used. In addition, the amount of data to be written into the search 2D reference CAD data storage memory at one time is written in several cycles for every amount equal to or less than a predetermined data amount determined in advance from the processing capability. Then, the function to be terminated is incorporated in the circuit.

  The basic flow of the main processing functions has been described above. However, this is only an example, and there are various combinations and sharing methods of functions. Try to design the circuit.

  116 to 128 illustrate the basic configuration of the sixteenth to seventeenth means of the present invention.

  FIG. 116 shows several examples when applied to various inspection apparatuses when a nuclear power plant is taken as an example. Of course, it may be applied to all inspection robots or monitoring / inspection robots and other work robots other than the examples given here, and is not limited to nuclear power plants, but also other nuclear facilities, thermal power plants, hydropower plants, chemical plants, marine It can be applied to inspection, work robots in all fields in plants, space bases, public facilities, hospitals and homes.

  FIG. 116 (a) shows a concept of applying an inspection apparatus to inspection of various devices in the nuclear reactor 21r in the nuclear power plant. In addition to nuclear fuel and control rods, various devices such as shrouds are installed in the reactor 21r. The fuselage of the reactor 21r itself is made by processing a plurality of plates. It is a structure joined by welding, and various nozzles connected to the piping members are attached to the body plate and the bottom by welding. There are also welded structures in various furnace structures. It is important to check the soundness of these weld lines, and in nuclear power plants, these weld lines are regularly inspected. The inspection is also performed by visual inspection (VT inspection), ultrasonic UT inspection, or the like. When inspecting the inside of the nuclear reactor 21r, after stopping the nuclear power plant, the lid on the upper part of the nuclear reactor is opened, the internal structure that can be taken out is taken out, and fuel etc. are taken out if necessary. The inspection devices 21A and 21C are put into the nuclear reactor from the state where the lid is opened and inspected. The inspection device 21A and the inspection device 21C are connected to a processing device 100 that includes an imaging device such as a TV camera and an ultrasonic sensor and processes those signals with the cable 21K. Since the inside of the nuclear reactor is an environment with high radiation, the reactor 21r is filled with water, and all of these inspection operations are performed remotely in water.

  116 (b) shows the configuration of the inspection apparatus 21A shown in FIG. 116 (a), and shows an example of an underwater ROV type inspection apparatus that moves while floating in water. The underwater ROV includes a propulsion method by rotating a screw and a jet method that injects water. The control can be performed remotely by controlling the rotation of a plurality of screws and controlling a plurality of jets. In the case of this example, the right camera 20 and the left camera 10 are incorporated in the underwater ROV main body 21A so that stereo image information can be obtained. In addition, if necessary, illumination is also mounted on the underwater ROV. As for the illumination, the preparation of the external illumination from the image recognition is also the object to be imaged even when the underwater ROV 21A moves, here the welded part 909 of the bottom of the reactor is the object to be imaged Since the stable image can be obtained when the illumination condition for the lens is not changed, the processing performance is improved. Still, in places where lighting cannot be placed outside due to space, image processing is performed with the mounted lighting. However, the feature point extraction method as shown in FIG. By extracting only corresponding points and converting them into 3D CAD data, even when lighting is installed, the image data is processed to generate 3D CAD data, while the position of the underwater ROV 21A itself It is possible to detect. Further, in order to reduce the influence of the moving illumination, an illumination method may be used in which both the moving illumination is locally brightened and the whole is brightened with sufficiently strong fixed illumination.

  The environment in which the operator remotely controls the underwater ROV main body 21A is an environment in which a plurality of similar nozzles stand in a remote place where the operator cannot directly view the underwater ROV main body 21A. It is very difficult for the operator to recognize the current position of the. Also, even when an inspection is being performed, it is difficult to take correspondence as to which inspection is being performed, but if the visual information processing apparatus of the present invention is applied, the position of the imaging apparatus is detected while being managed by tracking. Therefore, if the position and posture information of the underwater ROV 21A is obtained from the position information of the imaging device and the result is displayed on the monitor of the processing device 100 of the operation unit, the operator can easily confirm the current position and posture. Will be able to.

  Further, the position and orientation information of the underwater ROV 21A can tell where the inspection is performed by the sensor, so that the inspection result data can be automatically recorded in association with the inspection location. Become. In addition, the underwater ROV main body 21A may be equipped with an acceleration sensor or a gyro sensor in order to interpolate the position information of the underwater ROV 21A when tracking by image information is interrupted, and the sensor signal may be processed. In addition to normal illumination, a marker 21e such as a laser for applying one or a plurality of light lines to a subject to be imaged may be mounted. For example, if the line of light by the marker 21e is red laser light, the color of the welded portion to be imaged and the normal illumination will be different, so the shape of the underwater ROV 21A body is cylindrical and the corners are as rounded as possible. In addition, an outlet for the cable 21K is provided at the top.

  A plurality of propulsion devices are provided so that the surface unevenness formed by the screw and jet injection ports can be reduced as much as possible. This is because in the case where the welded part of the control rod drive mechanism or the like is inspected at the lower part of the reactor, the underwater ROV 21A is introduced from the upper part of the reactor 21r to the bottom of the various in-reactor structures. It is necessary to move by remote control through, and it is necessary to dive under the fuel support part at the lower part from the narrow space where the fuel is drained on the way, so if there is a projection it will be caught and movement control will be difficult In addition, even if the propulsion device breaks down, the surface is made as smooth as possible so that the underwater ROV body can be recovered by pulling the cable, taking into account the emergency recoverability. Is.

  By controlling a plurality of jets, the underwater ROV main body 21A can be moved forward, backward, on the spot, moved up and down, and controlled in posture. Since the shape is a cylinder, it is better to cause the cameras 10 and 20 to be tilted inside the underwater ROV main body 21A rather than performing tilt control of the posture.

  In the case of jet propulsion, if a large pump is prepared on the floor to be operated and a fluid whose pressure has been increased is incorporated in the cable 21K so that high propulsion energy can be obtained from an external pump, an underwater ROV body No pump is required for 21A, and a small solenoid valve can be mounted only for the number of injection nozzles, or the injection can be controlled on the pump side with a plurality of hoses. Can do. The underwater ROV main body 21A is connected to a processing device 100 corresponding to an operation unit at a remote place by a cable K.

  The processing device 100 is configured to be capable of remote manual operation or automatic control of the underwater ROV main body 21A in addition to the function of processing the signal of a sensor such as a camera and recording the inspection result. The manual operation may be performed with a mouse switch on the screen of the processing apparatus 100, or an operation switch such as a joystick may be provided separately.

  FIG. 116 (c) shows that the operator or machine simply hangs the camera 10 with the cable 21K that is inserted from the top and the camera 21 is out of the part except the bottom in the reactor of FIG. 116 (a). An example in the case of applying to an inspection camera handled in the air on the upper floor is shown. The operator can move the camera 10 in the vertical direction by loosening or pulling the cable 21K, and if the operator moves the cable 21K, the camera 10 also moves in the same direction. An illumination 21s is attached to the camera 10 together, and the whole is referred to as a suspended camera body 21C. Of course, a simple propulsion mechanism, a guide mechanism, a mechanism for moving the wall surface by suction, and the like may be provided in the suspended camera body 21C. Two cameras 10 may be used as a stereo camera system. Further, a laser spot line may be irradiated to the imaging target part. In addition, an ultrasonic sensor (UT sensor) other than a camera is mounted so that the wall of the furnace and the structure can be accessed in a predetermined posture by the guide mechanism and the suction mechanism, and the wall thickness of the furnace wall and the structure is increased. You may make it measure or inspect an internal defect with a UT sensor.

  In the case where the camera 10 of the suspended camera body 21 </ b> C and other sensors such as a gyro are mounted, the signal information thereof is input to the processing device 100, and the image information processing function in the processing device 100 is used. While processing the image information of the camera signal and displaying the current position of the camera on the monitor, the inspection result (image data, UT inspection data, etc.) is converted into the part information of the three-dimensional CAD data generated by processing the image information. Correspondingly, inspection data may be automatically recorded. If three-dimensional CAD data is generated by processing image data, even if there is a defective part, the defect is generated as CAD data. Therefore, the CAD data has three-dimensional position information. Information on the shape and size of the defect and where the defect is located in the reactor facility can be easily confirmed by displaying on the monitor of the apparatus. A marker that can irradiate a subject with a plurality of laser lines 21e may be mounted on the suspended camera body 21C.

  116 (d) is applied to a crawler type inspection device for inspecting a welding line of a lining such as a pool filled with water in order to temporarily store fuel separately from a nuclear power plant nuclear reactor. An example of the case is shown. Steel linings are applied to the wall and bottom of a pool or the like, and the linings are joined by welding plate materials so that water in the pool does not leak. As an inspection device for inspecting the weld line of the lining, the wall surface can be inspected using the underwater ROV 21A and the suspended camera body 21C described in (b) and (c). For the lining welding line 909, the crawler type VT device 21B is used, which can move stably and at high speed, so that the inspection efficiency is good. The crawler-type VT device 21B has two left and right cameras 10 and 20 mounted on a traveling mechanism that can move forward, backward, and turn on the spot by controlling left and right crawlers. If necessary, lighting is also installed to enter underneath the fuel storage rack and inspect the weld line. Also, an acceleration sensor or gyro sensor that interpolates image tracking may be installed, or information such as an encoder that detects the number of rotations of the left and right crawlers is installed to correct the image tracking by integrating the rotation angle. It may be to. A wheel traveling system may be used instead of the crawler system, or an encoder signal that is in contact with the floor surface and reduces slippage may be taken in independently.

  The welded part 909 is VT-inspected with image information from the mounted cameras 10 and 20, but CAD data is also generated, and if there is a weld bead shape or scratch, its location and size are automatically converted as CAD data. It is good to record it. Further, a system in which image data captured by a camera is left as an inspection record may be used.

  The crawler type VT apparatus main body 21B is connected to the processing apparatus 100 via a cable K, and communicates crawler operation control signals and camera and sensor signals via the cables. In this example, the cable 21K is used, but the mobility of the crawler may be improved by using a wireless system. In addition, a line marker is mounted on the crawler 21B so that the laser light line 21e can be applied to the welding line 909 and its periphery. In addition to the VT inspection by the cameras 10 and 20, the UT sensor 21b may be UT-inspected by tracing the periphery of the weld line with the arm mechanism 21a. Since the part of the weld line 909 and its periphery can be generated as three-dimensional CAD data, the location where the UT inspection is performed can be easily recorded in association with the CAD data.

  The inspection probe is not limited to the UT sensor, and may be controlled so that the radiation source (RT) is aligned with a predetermined position, or may be an inspection probe such as ECT. Since the position and posture of the inspection probe 21b such as the UT sensor are known by image information processing, the position of the probe 21b at the tip of the arm mechanism 21a relative to the crawler is obtained from information such as the joint angle of the arm mechanism 21a. Thus, it can be associated with the position of the coordinate system of CAD data. Here, the arm mechanism 21a may be mounted with a simpler one-dimensional or two-dimensional scanning mechanism.

  As a variation of the embodiment, (b) underwater ROV, (c) hanging type camera, (d) crawler type VT device was described as an example. It may be a humanoid robot that moves, or a balloon-type mobile robot in which a sensor and a propulsion device are mounted on a balloon. Since the processing apparatus 100 has a common part for capturing image information and performing image processing, the part for controlling the operation of each robot needs to be adjusted for each robot. Since the parts for generating the three-dimensional CAD data and displaying the current position information are common, it is possible to connect the cables 21K of a plurality of types of inspection devices to the same processing device so that they can be operated. .

  FIG. 117 shows the concept of automatically generated as-built three-dimensional CAD data around the pool bottom weld line 909, using the crawler type VT inspection device 21B as an example. The position and orientation of the camera, the angle of view of the camera, etc. so that the same part of the weld line 909 shown in FIGS. (A) and (B) of the images of the cameras 10 and 20 mounted on the crawler type VT inspection apparatus 21B can be seen. Adjust it. Since both the image data (A) and (B) are captured at the same time in the processing apparatus 100, in the stereo measurement process before the tracking process, the difference in how they are seen even if the apparatus is equipped with illumination. Is only a difference in perspective. In addition, since there may be a case where a sufficient number of feature points cannot be obtained in the weld bead 909 when corresponding points are taken with the same metal color and bead bulge or pattern, the laser mark line 22e can be seen in the image. For example, if the line is red, the contrast is very good and the laser mark line can be extracted as a feature point or feature line segment, so that more shape data of the weld line bead can be obtained by stereo measurement. Also good. 116 (d) shows a state in which one laser marker line segment 21e is irradiated. However, a plurality of marker lines can be simultaneously formed in parallel, in the vertical direction or diagonally in addition to the horizontal number. May be used for stereo measurement of cross-sectional shapes at a plurality of locations at the same time. However, in this case, since it is different from the scratch and dirt data originally in the weld line bead, the data should be managed separately. In addition, the red line makes it possible to easily distinguish data. Further, the marker may irradiate one point or a plurality of points at the spot point instead of the line. The laser mark line 22e can record the shape of the weld bead in detail. The convex shape of the weld bead is also important inspection data. When tracking processing is performed, if there is a bead bulge outline 22b, a bead surface pattern 22c, dirt 22a, or a flaw, tracking is performed by detecting them as feature points. If the cross-sectional shape is acquired as densely as possible by simultaneously irradiating a plurality of lines 22e, the detailed uneven state of the weld line bead becomes a characteristic part as a three-dimensional shape. Therefore, three-dimensional CAD data is obtained by stereo measurement. When generated, a three-dimensional shape in a predetermined range of the bead can be obtained. Therefore, if the crawler moves and the same part overlaps from another viewpoint, the three-dimensional CAD data by the line of the laser mark is generated. There are convex and concave positions with the same characteristics in the data measured overlaid, so tracking can be made easier by matching the shape. It is possible to perform in.

  These are not limited to crawlers, but are the same for underwater cameras and underwater ROVs. In the case of a crawler, since image tracking can be interpolated also from crawler rotation angle information, even if CAD data cannot be continuously tracked, it is possible to estimate the amount of movement in a range that cannot be estimated by crawler rotation. Therefore, the CAD data to be generated can be generated as data continuously converted into one coordinate system (world coordinate system Ow). Since the crawler type VT inspection device 21B moves along the weld line, three-dimensional CAD data around the weld line is generated, and the inspection data is recorded in association with the position information of the CAD data.

  The CAD data can be displayed on the monitor while changing the viewpoint in various ways, and the shape of the weld line, the position of the flaw, and the position of dirt can be confirmed. When the positions of the virtual viewpoints for displaying CAD data are separated from each other on the monitor display, the situation as seen from many is displayed, and the weld line 22B of the entire pool is displayed. If a part of the image is designated and enlarged, that is, the CAD data is displayed with the virtual viewpoint close to the position, it is possible to enlarge and confirm the local weld bead bulge 22d, pattern 22c, dirt 22a, and the like. become able to. In addition, if stereo measurement is performed in slightly dirty water, the floating dust is measured in 3D. In this case, since the point is not measured when the viewpoint changes, the first flag does not disappear. Therefore, such CAD data may be erased later, or floating dust Is floating from the bottom of the pool or reactor vessel, check the coordinate position of the floating dust, and if the position is floating from the bottom surface, it is determined that it is floating dust The CAD data can be erased. This is an effective method because floating dust can be surely erased.

  FIG. 118 exemplifies a system configuration block diagram of the crawler type VT inspection device 21B. In the large block of the crawler type VT inspection device 21B, left and right motors 23a and 23b are connected to a driver 23d as main components, and are driven by a command signal from the main control routine 23e in the processing device 100. It can be controlled. The output of the command signal from the main control routine 23e may be output by automatic control from the program of the main control routine 23e. Here, although not shown in the figure, the joystick operated by the operator It is also possible to output an operation command signal such as for example via the main control routine 23e. Each of the motors 23a and 23b is provided with an encoder, and the signal can be taken in and referred to the main control routine 23e through an interface such as a counter board. In addition to the left and right cameras 10 and 20, the crawler traveling vehicle 21B is also equipped with an acceleration sensor 30 and a gyro sensor 40. Image information of the cameras 10 and 20 is processed via an interface such as a capture board. The image processing routine 23g in the apparatus 100 is loaded, and the information of the sensors 30 and 40 is loaded into the main control routine 23e via an interface such as an A / D board. In this example, the sensor processing is put into the main control routine 23e and the image processing routine 23g is divided, but the signals of the sensors 30 and 40 may be directly taken into the image processing routine.

  The image processing routine 23g performs processing such as stereo measurement and tracking processing, and the generated CAD data is accumulated in a predetermined storage device 200. The storage device may be generated on a real-time RAM memory at the time of generation, and recorded and stored in a hard disk or the like when the generation operation is completed. Then, when operating again at the same location, the data previously recorded on the RAM memory from the hard disk may be read out in the first stage and regenerated and recorded again. In the case of comparative analysis, new data may be generated, and the result may be automatically compared with previously recorded data so that different portions can be output and displayed. Further, the main control routine 23e can input the plan contents of how to move the crawler VT device 21B from where to where as a STAR / END point and an intermediate movement plan route. Based on the processing result of the sensor information and the image processing result, an appropriate control command can be automatically output to the driver 23d so that the crawler traveling vehicle 21B can move and control in accordance with the contents of the plan.

  From the main control routine 23e, the current position and orientation information estimated from the sensor information is output as an auxiliary position signal. Instead, the main control routine 23e performs current processing obtained by image processing from the image processing routine 23g. Receive true position and posture information. Here, the main control routine 23e and the image processing routine 23g are separated from the viewpoints of versatility and processing speed, but they may be combined.

  The processing apparatus 100 is assumed to be realized by a single computer. However, in order to increase the processing speed, the main control routine 23e and the image processing routine 23g are executed by different CPUs so that they can be used in service memory or communication. Data may be exchanged at high speed using a line. An example of a basic flowchart of the main control routine 23e and the image processing routine 23g for realizing this configuration will be described later.

  Also, this system generates as-built CAD data, but the environment data such as the pool, the size of the pool and room, the obstacles in the pool and the room, and the weld line to be inspected are three-dimensional CAD. In order to make it easy to see where in the entire environment when displaying the current position of the ROV or crawler by registering it as data, it is possible to display CG by overlaying previously registered pools, obstacles and welding lines It is good to leave.

  Also, by displaying the CG scale alongside the CG display or raw image display of the target object you want to check, such as a weld line, you can immediately see the approximate size of the target object at a glance. It is good to make it. If the object is a scratch on a plane, a scale, a measure, etc. are displayed in CG along the scratched plane. Even if the plane is displayed at an angle depending on the point of view, if some of the plane's feature points have already been generated as 3D CAD data, the scale or measure is actually set along the plane. It is also possible to display the CG scale and measure along the screen in the same situation as when shooting. Further, even if the object is a curved surface, it is possible to display the image as if it was actually shot with a scale or measure along the curved surface based on the three-dimensional CAD data of the curved surface.

  By doing so, three-dimensional VT inspection data is automatically generated without a sense of incongruity with a VT inspection method in which a scale or a measure is actually photographed near the object and recorded as normal image data. Can also be done easily. Of course, there is no need to insert a scale or measure when shooting in the field, but the generated VT recording will display the scale and measure displayed as if they were actually inserted into the CG display. Can be confirmed.

  FIG. 119 shows an image when the CAD data 22B (C) in which the pool and the weld line are registered in advance as CAD data and the CAD data 22B (R) of the as-built weld line generated by image processing are displayed in an overlapping manner. Show. The floor surface of the pool may be a crawler traveling inspection device 21B, and the wall surface of the pool may be a magnetic or adsorption type crawler inspection device that can move the wall. This is an example of generating data. Since these systems generate as-built CAD data, it is sufficient to create only parts that serve as landmarks that can be used to identify the approximate position of CAD data registered in advance. The CAD data registered in advance displays the entire pool and the entire weld line, but the generated CAD data displays only the generated part. By overlapping and displaying together in this way, even in a situation where the generated CAD data is still only a part, it is possible to know at a glance which hit the generated CAD data is.

  In addition, since the error caused by tracking is accumulated in the generated CAD data, the location in the whole environment is registered in advance by the accumulated tracking error even if the location or the location is generated with high accuracy. It is different from CAD data. The deviation tends to increase as the distance from the start point where the generation is started. However, during operation, periodically, or during offline work after generation, for example, an operator creates a feature part of CAD data generated in advance. If a feature part of the CAD data being specified is designated, it is preferable to provide a function of smoothly correcting the entire generated CAD data by matching the points. This correction is a condition that matches because the first start point is started after matching with the actual location. In the middle, if there are a plurality of points taught that this point is the same as this point, processing may be performed so that other points are smoothly connected so that these points coincide with each other. . In addition, the data of other taught corresponding points is registered in the CAD data, and when the processing for newly generating or updating the CAD data after the teaching is performed, the correction processing is automatically performed. It should be done. In addition, it is easy for the operator to designate several points that match, but a correction program can be easily created. By automatically obtaining the position from the generated CAD data by matching, the error due to tracking may be automatically corrected each time the operator does not specify and operate the corresponding point.

  The CAD data registered in advance may be data based on a drawing, but if the actual measurement dimensions are known, it is preferable to create CAD data based on the actual measurement dimensions. The data registered in advance can be easily dealt with because the size of the room is only the position of the main obstacle, and the number of dimensions to be measured is small.

  On the other hand, the generated CAD data is detailed and locally accurate CAD data. Therefore, if the generated CAD data is corrected so that the generated CAD data is matched with the input CAD data by actually measuring only the dimension of the entire distant portion, the generated CAD data is converted into more accurate as-built detailed CAD data. Will be able to.

  In addition, when CAD data of the inspection object is prepared in advance, the position of the surface of the inspection plane can be roughly specified, so that a portion with poor accuracy in the depth direction during stereo measurement or tracking measurement is prepared in advance. If the detection of the distance in the depth direction is corrected based on the assumption that the surface of the inspection surface of the CAD data is detected, the 3D measurement is easily performed while improving the accuracy in the depth direction. can do. This is the same for curved surfaces and flat surfaces, but it is different when inspecting the depth of shallow scratches, but generally, the vertical and horizontal positions of scratches and dirt are often measured and recorded. If 3D CAD is generated using depth information of a known surface, it is possible to easily generate accurate CAD data in the depth direction. This error in the depth direction is greatly affected by the matching error of corresponding points, and is therefore an effective processing method in the case of an inspection apparatus that performs inspection on a known surface.

  With reference to FIGS. 120 to 124, an example of a dedicated file that is convenient for these systems will be described. This example will be described with reference to an example in which an inspection is performed by floating an underwater ROV in a pool.

  FIG. 120 shows an example of a movement locus data file for inputting an instruction to the main control routine 23e. The file is described in a file that can be edited by the character editor, and the main control routine first reads the contents of the file. This is an example of description information in the file. For the START point, the coordinate data and the attitude data are described in the world coordinate system Ow.

  FIG. 121 shows the relationship between the world coordinate system Ow and the robot coordinate system Or of the ROV main body 21A. The START point is an example of a case where the start posture (orientation) when the ROV main body 21A is located at the START point is expressed by the posture of the ROV robot coordinate system Or placed at the START point. Similarly, in the file of FIG. 120, the position is described in the world coordinate system as the END point and the intermediate point is the Nth point, and the orientation of the robot coordinate system Or of the ROV 21A at all the points is also specified. Like that. The movement trajectory is a straight line, circle, arc, or the like, which is also described as a line segment in the world coordinate system Ow. It is also possible to specify the color, line type, etc. when displaying points in CG. The flag indicating whether or not the posture is displayed is a flag indicating whether or not to display the ROV target posture at each point when performing CG display. Also, the stop different time at each point means the stop time at that place from when the ROV arrives at that point until it starts moving toward the next point. In this case, the main control routine 23e The control is such that hovering control is performed.

  The trajectory data for straight lines and arcs should be able to describe the target speed of how much to move the section near the start point and end point, and the details of the deceleration control and acceleration control plans, In this case, the main control routine 23e appropriately moves the ROV according to the planned value, and performs control so as to decelerate or perform acceleration control near the point. The END point is provided with a flag indicating whether or not to return, and when the return flag is ON, control is performed so as to return the same locus route as when it came.

  Also, when displaying CG so that the contents of point and trajectory data can be understood by the operator, there is a case where it is desired to annotate what kind of point, what kind of trajectory, and the point where the character string and the world coordinate system are to be displayed. Is described, an annotation is automatically displayed at the position when CG is displayed. In addition to the START point and the END point, the first inspection point, the second inspection point, and the like are displayed. This annotation always causes the character to be normally displayed on the monitor even when the virtual viewpoint of the CG display is changed. Also, the size of the characters may be fixed to a size that is easy to see, or may be automatically set according to the situation. In addition, when displaying CG, CAD data registered in advance can be displayed and deleted as necessary. Displaying where the world coordinate system Ow is located is convenient when the coordinate data can be confirmed on a monitor later.

  FIG. 122 shows an example of a file in which the control logic of the robot such as ROV is registered. Similarly, this file is also a character file that can be easily corrected by an editor. At the time of execution, the main control routine 23e reads first, and when read, the contents of the control program are changed to the contents corresponding to the contents described in the file. Keep the part replaced. That is, a program for converting this file into an actual control program is executed when the first file is read, and a part of the control program is created and incorporated at that time. As a result, if you want to try changing the robot control logic by trial and error, or if you need to create another ROV and create a control program suitable for it, use the C language. Thus, any program expert can easily change the control logic when necessary without rewriting the control program. In this example, by simply determining the rules to be written, the conversion program to be executed first can be simplified. Here, for each promise, the input data is described as δX for the X-axis deviation, δY for the Y-axis deviation, δZ for the Z-axis deviation, and δα, δβ, and δγ for the rotational deviation of each axis. To do. The deviation amount here is the deviation amount between the current position Or of the ROV 21A and the current target position Ori on the planned route. The current target position Ori is obtained on the basis of the movement trajectory data read from the file of FIG. 25. However, the time axis needs attention because the actual control time may be delayed or earlier than planned.

  As an example of appropriate processing, first, the perpendicular line from the current position of the ROV to the movement locus route is lowered to obtain the shortest point (P) to the movement plan locus route, and the original plan at that place is obtained. The moving speed is obtained, the amount of advance from the shortest point (P) after a predetermined time is obtained, and the point to be advanced is considered as the target position point Ori to go next. In this way, since it is adapted to the target locus at the closest location based on the position where the ROV is present, smooth guidance control is possible. Since the target position Ori corresponds to the target robot coordinate system at that location, in addition to the position information, the ROV target posture information, and further, at which speed and at what speed must be moved at that position. The dynamic of whether or not it may be possible may define a target value of speed and control the amount of deviation from it.

  In addition, the output data finally requires the amount of control of the ROV screw and jet injection, or the amount of rotation control of the left and right motors in the case of a crawler, but here the processing device 100 is assumed to be general purpose. Therefore, the output data from here is, as control information for the ROV coordinate system Or, the X-axis control amount Vx (target speed command value), the Y-axis direction control amount Vy (target speed command value), and the Z-axis direction control data. Control amount Vz (target speed command value), rotation control amount Vα (target rotation angular speed command), Vβ (target rotational angular speed command), and Vγ (target rotational angular speed command) around each axis are not speed command values. Alternatively, a predetermined time may be determined in advance, and the target movement amount or target rotation angle amount within the predetermined time may be output.

  In addition, as symbols that can be described, numbers 0 to 9 and (), four arithmetic operators, sin, cos, tan, square root SQR, and the like are provided. Further, 20 variables D1 to D20 may be defined as variables, and intermediate calculation results may be temporarily stored with the = operator. Here, as an example, if the speed control command Vx in the X-axis direction is described to be an amount obtained by multiplying the deviation amount in the X-axis direction by a value of 2.5 as a coefficient, Vx = −δX * ( 2.5), a control amount −Vx that is proportional to the deviation amount δX by a coefficient 2.5 is actually output as the control amount. In general, the ROV moving in space is controlled by controlling the position in the X, Y, and Z directions and the angles around the X, Y, and Z axes in the ROV coordinate system. Therefore, the processing apparatus 100 is manufactured so as to output these six control commands. The amount of control output as a general ROV attitude control command needs to be converted into actual screw or motor rotation control, but that part moves in the number of drive shafts and the three-dimensional space of the ROV. Since there are cases where the number of drive axes is less than 6 on the condition that there is freedom to move or the floor surface is always moved, conversion of control commands peculiar to ROV is required. Since this is a conversion process unique to ROV, a control command conversion program is provided in the driver 23d mounted on the ROV. Alternatively, this conversion program may be provided with a control command conversion unit newly as a dedicated unit between the processing apparatus 100 and the driver 23d, or in the form of another program in the main control routine in the processing apparatus 100. You may make it crowded. In the case of incorporating in the form of a program, if another type of ROV is connected and used, a conversion program for another control amount may be prepared in advance with various programs specific to the ROV. To use the program. It is preferable that an appropriate program is automatically selected when the type of ROV is selected.

  FIG. 123 shows the concept of the world coordinate system Ow and the ROV coordinate system Or and the concept of CAD data registered in advance in the basic environment such as the pool 26a. As environmental data, it is sufficient to register only the main parts of the main obstacles in the pool.

  FIG. 124 shows an example of a definition file of basic environment data and shape data of the ROV itself. The ROV shape data is first read in accordance with the ROV actually used. Also, environmental CAD data is first read from the environmental data of the inspection place of the plant that will actually operate. The environment CAD data can be defined using coordinates on the world coordinate system Ow using basic figures such as a straight line, a sphere, a rectangular parallelepiped, a circle, and an arc. The ROV shape data is defined so that the basic figure can be similarly defined by the coordinates on the robot coordinate system Or. The character string is an example in which the character string can be described together with a place to be displayed as an annotation of environmental data.

  Next, an example of a basic flowchart of the main control routine 23e and the image processing routine 23g will be described with reference to FIGS. Here, an example in the case of a crawler movement type VT apparatus will be described.

  125 and 126 are basic flowcharts of the main control routine 23e. If it is started, the ROV START point, END point, and intermediate locus data are read from the SYSTEM \ movement locus data \ (task) \ (name) file in the process 30a. Next, in process 30b, CAD data of the basic environment data from the SYSTEM \ basic environment data \ (task) \ (name) file, and ROV shape data from the SYSTEM \ ROV data \ (task) \ (name) file. Read. In step 30c, the ROV control program is read from the SYSTEM \ control program \ (task) \ (name) file. This is the first process. At first, the actual position and posture of the initial ROV are set to match the position and posture of the START point. In order not to do this, the generated CAD data and environment CAD data are matched with the characteristic part, and until it is accurately matched, it is not known where it is, but until then the surrounding environment data is generated However, only control that does not cause a collision may be performed, and first, a process of searching for the current position may be performed.

  In the next process 30d, the CAD data of the basic environment data and the current position and orientation of the ROV are displayed on the screen monitor of the processing apparatus 100 from an arbitrary viewpoint. Next, in process 30e, a process of reading the signals of the left and right encoders of the crawler traveling vehicle from the counter board is performed. Next, in processing 30f, processing for reading the sensor signals of the acceleration sensor and the gyro sensor from the A / D board is performed. It should be noted here that when a plurality of types and a plurality of types of sensors are mounted in the respective axial directions, in order to properly recognize the state of a certain time of the ROV 21B, there is a slight amount that can be simultaneously or simultaneously. It is better to read within a time lag. Next, in the process 30f, the estimated current position and orientation of the ROV 21B are updated from the signals of the left and right encoders, the signals of the acceleration sensor 30 and the gyro sensor 40, and transferred to the image processing routine 23g.

  Next, in process 31a, the true current position and orientation of the ROV are received from the image processing routine 23g, and the current position and orientation data of the ROV are updated. Next, in a process 31b, a deviation between the planned position and orientation based on the current position and orientation data of the ROV and the movement trajectory data is obtained. Next, in process 31c, ROV control data is obtained based on the deviation between the current position and orientation of the ROV and the planned position and orientation and the control program, and control command signals are output from the D / A board to the left and right motor drivers of the crawler. . Thereafter, the program returns to the process 30d and processes so as to repeat the above-described process.

  127 and 128 show a basic flowchart of the image processing routine 23g. When the process starts, the process 30a reads the image information of the left and right cameras 10 and 20 from the capture board. Here, it is necessary to read the information of the left and right cameras at the same time. If the time is shifted, the ROV is moving, so that it affects the error of stereo measurement.

  Next, in process 32b, the image information of the left and right cameras 10 and 20 is corrected for distortion with the parameters obtained in advance during calibration. Next, in process 32c, feature points (line segments) in the image information of the left and right cameras 10 and 20 are extracted and converted into three-dimensional point coordinates (line segments) by stereo measurement. At this time, the relationship between the positions and postures of the left and right cameras 10 and 20 is preferably obtained in advance with accuracy and reflected in the calculation formula for stereo measurement. Next, in process 32d, the left and right color information of the feature point (line segment) is recorded as CAD data from the image information of the left and right cameras 10 and 20. Further, image data is recorded in association with three-dimensional point coordinate (line segment) data.

  Next, in the process 32e, when there is 3D CAD data previously measured by stereo, at least three corresponding points to be tracked are determined, and the process of converting the 3D CAD data measured this time into the previously measured coordinate system is performed. carry out.

  Next, in process 32f, it is confirmed whether there is the same point (line segment) data in the already generated CAD data. If there is, it is determined whether the accuracy of the CAD data is improved. The accuracy includes error due to image distortion, error due to the accuracy of the coordinate conversion calculation formula of the position and orientation relationship between the two cameras 10 and 20 for stereo measurement, and error due to the distance between the measurement point and the camera (the closer the stereo measurement accuracy is, And the farther away the accuracy becomes worse.) Then, an error caused by coordinate conversion by tracking ((the matching point of the corresponding point) is the best among the measurement points as shown in (e) of FIG. 67, and Find coordinate conversion formulas so that the three points with good measurement accuracy match (even if the three points measured at the previous viewpoint and the three points measured at the subsequent viewpoint are matched, the three points should be matched) If one point is combined, the other two points will not match easily because there is an error, so the positional relationship of the coordinate systems of the two viewpoints that match each of the three points as they are is obtained.) If there is a shooting condition that causes the camera system to be out of focus, etc., an error that occurs when a new point cloud measured at a later viewpoint is converted into the previous coordinate system) , Taking into account overall measurement error due to out-of-focus (this is because the measurement accuracy of the point measured under the in-focus condition is high, and the accuracy is deteriorated when shooting under the out-of-focus condition) Judge whether to update or not. Further, since there is an error in the degree of coincidence of corresponding points during stereo measurement (error radius r of point P4 in FIG. 44), this is also taken into consideration. This matching point error is large for contour feature points, small for sharp corner contours, and most accurate for feature points that make up actual lines such as patterns. Although it becomes better, most of the environment generated when it is simply excluded from the target points to be CAD data with a certain threshold value is a contour line, and the error of the contour line such as a cylinder or a curved surface becomes large. Lines are also important environmental information. Therefore, this error may be distinguished from other errors, and the magnitude of this error may be managed as an index representing the attribute of the contour line or the actual line.

  Next, in the process 32g, for the point data whose accuracy has been improved, the generated CAD data is updated. In addition, line segment CAD data is updated. Next, in the process 32h, when there is surface data related to the point data whose accuracy has been improved, a process of updating the surface data is performed. Next, in the processing 33a, when the image data corresponding to the point (line segment) data becomes better (an index such as a direction facing the surface to which the user belongs and closer, closer to the center of the image, etc.) The image data update process is executed.

  Next, in the processing 33b to 33f, when the movement amount of the ROV has reached a predetermined movement amount from the position measured in the past, the processing is executed when there are at least three corresponding points, and more accurate than the stereo measurement. It is intended to perform 3D measurement well. In this case, past two-dimensional feature point data measured in a certain range area suitable for 3D measurement from the current viewpoint position may be recorded without being erased when this processing is performed. is necessary. As a means for selecting recorded information, it is assumed that even if the user moves far away and returns to the same place again, the direction of the representative viewpoint for photographing each block divided into predetermined blocks in the space ( For example, if the vertical and horizontal directions are defined, the direction may be defined in 6 directions and more finely in 45 ° increments.) By recording only the representative scene closest to the above, the increase in the data amount can be suppressed. Of course, scene data captured in the past may be automatically deleted from the data recording capacity.

  As shown in FIG. 128, the specific processing flow is when the ROV movement amount reaches a predetermined movement amount from the position measured in the past in step 33b, and there are at least three corresponding points. Then, a coordinate transformation calculation formula is obtained based on the corresponding points, and stereo measurement is again performed using two pieces of image information in the coordinate relationship (corresponding to tracking processing of a large motion that has passed minute movement tracking on the way). In the process 33c, the obtained three-dimensional CAD data is converted into the initially measured coordinate system and the currently measured three-dimensional CAD data is converted. In the next process 33d, the same point (line segment) data is included in the already generated CAD data. It is checked whether or not there is, and if so, it is determined whether or not the accuracy of the CAD data is improved. When determining here, as described above, it is determined whether or not to update by comprehensively determining possible errors. The error due to the distance between the measurement point and the camera here is that in stereo measurement, the closer the stereo measurement accuracy is, the farther the accuracy is worse. When measuring, if the distance between the viewpoints becomes wide even at the same distance, the accuracy is improved accordingly. Next, in the process 33e, the generated CAD data is updated for the point data whose accuracy is improved. At the same time, line segment CAD data is updated. Next, in the process 33f, when there is surface data related to the point data with improved accuracy, the surface data is updated. Then, in process 33g, the process of handing over the current position and orientation information of the left and right cameras obtained by tracking to the main control routine as the true current position and orientation information of the ROV is performed, and the process is repeated from the process 32a of FIG. Execute. This process 33g is executed even when the process block 33A (process 33b to process 33f) is not executed.

  As described above, while moving, it recognizes its own position, automatically generates the 3D CAD data of the environment, and automatically records the test result data corresponding to the part of the 3D CAD data. (In the case of VT inspection, the generated 3D CAD data itself becomes inspection record data), and it can also be automatically moved and controlled along the planned route. Since the comparison is also made by associating the inspection result data with the same part, it is possible to obtain a mobile inspection apparatus that can be easily compared and can easily check the trend of the inspection result at the location to be confirmed. If various plant facilities and various facilities, equipment, and assets are recorded and stored in 3D CAD data at a certain point, 3D CAD data of the same place is generated and compared again after time passes. It is also possible to easily find scratches and dirt that did not exist in the past. Of course, different parts may be automatically extracted by matching processing.

  In addition, if CAD data can be generated by instantaneously measuring moving objects by increasing the processing speed, by adding time data to the CAD data, objects that move in the environment can also be moved to the 3D CAD data. It is also possible to record and store world phenomena, craftsmanship, and precious arts as 3D CAD data that moves continuously and create a library for later generation.

  129 to 136 illustrate the basic configuration of the 21st to 24th means of the present invention.

  FIG. 129 shows a basic system configuration of the monitoring service providing system. It may be in public facilities such as towns, streets, parks, private land, indoors, etc., but it is installed in close proximity to terminals 34b (1), 43b (2) etc. that have sensors and communication functions such as high-resolution TV cameras. If an application is made using a mobile phone 34c or the like, information from each terminal 34b (1), 43b (2) etc. is sent to the general monitoring center 34g via the network line 34f of the telephone company 34e or the like. This system monitors applicants based on sensor information. By applying the above-mentioned visual information processing system to this monitoring service system, the user can check the user at the general monitoring center on the monitor and follow the user with the monitor at any time. It is also easy to identify CAD data and automatically track the user's position later by tracking processing of image processing. If you install adjacent terminals so that the shooting area of the terminal wraps, the user will be placed in the field of view of the next terminal when the user deviates from the field of view of one terminal. Processing can be easily performed continuously along a plurality of terminals. Of course, if the user's position is known by the GPS function of the mobile phone and can be identified by image recognition, the first user may be identified automatically. Even if it is not fully automatic, the approximate position obtained by GPS may be displayed so that the operator can characterize the user. The tracking process in this case is the two-dimensional image matching process described with reference to FIG. 53 by processing the image information captured by the terminal and extracting the feature points and line segments by the method shown in FIG. The user's feature points and line segments may be tracked by matching processing, or a plurality of terminals are always placed in a positional relationship where the user can be photographed, and the user's characteristics are used using images of at least two terminals. Points and line segments may be converted into three-dimensional CAD data and tracked by matching processing of the three-dimensional CAD data, or on the two-dimensional image using the search function described in the visual information processing system. Tracking may be performed by the search process in FIG. 5, or may be tracked while always searching for the user's three-dimensional CAD data by the three-dimensional search process.

  In this case, since the user is a human being, the neck, both arms, and both feet move within a predetermined range. Therefore, it is preferable to search using the CAD coupling conditions described in FIG. By registering in the reference file with the 3D CAD data (shape and pattern color information) that first measured the user's clothes and appearance when the user was first identified, Even when it is lost in tracking, it is possible to automatically perform tracking by automatically searching with a three-dimensional CAD data search. Further, a new CAD data combining condition may be easily defined for an article that the user is looking at. In such a system, even if there is no application from the user, a recording device is provided in each terminal, and if there is an application from the user at a later date, it is recorded in association with time data in the past. This is an extended system for monitoring services that allows the necessary locations of monitoring data to be retrieved arbitrarily from each terminal. An example of the terminal in this case will be described separately. As for the time, a radio clock may be incorporated in the terminal so that an accurate time can be referred to so that a monitoring image can be recorded at a predetermined time interval. In this way, if recording is performed for each terminal, it is possible to easily accumulate past recorded data in all locations without installing a device that records a huge amount of data in the general monitoring center. . The time that can be recorded depends on the performance of the terminal, but is assumed to be 24 hours, 1 week, 1 month or more. In the unlikely event that an incident occurs due to a suspicious person 34d, even if no user has applied for the monitoring service, the incident will occur even if the incident is discovered at a later date and the police investigates. If the monitoring record data before and after the time can be arbitrarily extracted from the general monitoring center 34g, the information necessary for the investigation can be obtained easily and reliably.

  When the applicant applies for the monitoring service and confirms the applicant at the center 34g and starts the monitoring service, a signal is sent from the monitoring center 34g to the mobile phone 34c of the applicant and the monitoring service starts. In such a case, a mark that indicates that the monitoring service is in progress, for example, the lamp 34i is blinked. By doing so, the suspicious person 34d can be informed that the monitoring service is being performed, and the incident can be prevented in advance. The lamp 34i may be attached to a brooch, earring, bag, etc. so that it can be casually attached to a place where the applicant is conspicuous. In addition, in order for the suspicious person 34d to recognize that he / she really receives the monitoring service, the light emission timing, light emission intensity, light emission color, etc. of the lamp are the same as the lamp 34z provided on the terminal 34b (1), 34b (2) side. You may make it shine synchronously. Since the light emission control is controlled by the center 34g or the monitoring system, a person who does not receive the monitoring service cannot easily imitate the light emission state of the monitoring lamp. If it is confirmed that light is emitted in the same manner as the terminal 34z, it is possible to recognize that the monitoring service is actually received and to further suppress the occurrence of the incident.

  As another means of the notification method, a laser irradiation device may be mounted on the terminal so as to irradiate in the direction of the applicant. The current location of the applicant is recognized by tracking in 3D coordinates, so it is easier to understand by writing a circle or pattern with a laser in that direction, or displaying a prominent mark on the applicant's head. It becomes a method that cannot be easily imitated. If it is difficult to imitate a simulated monitoring lamp, the suspicious person will not be doubted that he / she is receiving the monitoring service by making the fee relationship that it is cheaper to apply for this monitoring service.

  FIG. 129 (b) shows a data transfer method between the general monitoring center 34g and each terminal 34b. Basically, there are few radio frequency allocation problems when the number of terminals is increased by applying a method such as the remote monitoring system disclosed in JP-A-10-227400 by providing a relay function between nearby terminals. A wide-area sensor network is formed at the assigned frequency. In (b), the applicant's phone 34c directly uses a telephone line, but the terminal data communication is provided with a dedicated line. If all terminals up to the remote telephone company 34e are relayed by a terminal, the number of times data is relayed increases and communication time is required. Therefore, the relay transmission group is limited to a certain area and the long-range wireless device 34k is located in some places. It is possible to construct a wide-area sensor terminal network without the time loss due to the relay. You may make it utilize a mobile telephone for the long-distance radio | wireless apparatus 34k. The transmission route between the sensor terminals may not be a series, and may be a mesh wireless network. By using a mesh, even if one terminal fails, if the terminal itself has a diagnostic function, other terminals can recognize the failed terminal and transmit data via another route. Become. In the case of (b), the monitoring start signal is sent from the center 34g directly to the applicant's form phone 34c via the telephone company 34e and controlled to emit light from the mobile phone 34c to the lamp 34i. is there.

  FIG. 129 (c) shows that the terminal 34b (1) of the mobile phone 34c communicates with the data of the terminals 34b (0) to (3) that relay each other. The sensor data may be transmitted or a command signal from the center to the terminal may be transmitted via the form phone 34c. Further, the facility plan may be made so that both the method (b) and the method (c) can be used.

  FIG. 130 shows a basic example of a specific hardware configuration of the general monitoring center shown in FIG. The components of the telephone company 34e and the net 34f are considered so that normal telephone lines and Internet lines can be used. Of course, a dedicated line may be prepared, but the constituent devices can be realized with the same configuration as a normal telephone line, Internet line, and LAN line. FIG. 130 shows a basic hardware configuration of the general monitoring center in which a PC terminal is connected to a telephone line terminal.

  Prepare n units as many as the maximum number of lines that can simultaneously perform monitoring services. Each PC is rational because it is sufficient to operate the same program. When statistical data or the like is automatically aggregated, each PC terminal may be connected via a LAN so that accumulated data and monitoring service history can be aggregated, referenced, and used.

  FIG. 131 exemplifies a basic program of the monitoring service operated by the PC terminal. First, if there is a call to the telephone line in the process 34 (e1) a, the account information of the user who made the call in the next process 34 (e1) b is recorded. Next, in the process 34 (e1) c, the voice line of the user and the center member is turned on, and the user and the center member talk directly to specify the service request and the user. Specifically, the user is specified by the following process. The center staff asks the user the current location, selects the most appropriate terminal from the map information, etc., and designates it. In step 34 (e1) d, the program calls the terminal specified by the center member via another telephone line or the telephone line that the user is calling if the mobile phone that the user is calling has a communication function with the terminal. Then, a command for transmitting the video (image information) in the direction designated by the center staff is transmitted to the terminal, and the video data returned from the terminal is displayed on a predetermined window of the PC monitor in process 34 (e1) f. The center member searches for a user by repeatedly requesting an image in an arbitrary direction of an arbitrary terminal in order to confirm the user with an image sent from the terminal. At first, a wide-angle image is searched for in a wide range, and if a person who seems to be a user is specified, the user is zoomed and enlarged to confirm the user. Since the confirmation work is performed while talking to the user over the telephone line, the user confirms the contents while lifting the hand or looking at the terminal while confirming the contents while talking, so the user is definitely identified correctly become able to. If the center member confirms the user and designates the vicinity of the user on the screen, the program side determines whether or not there is a coordinate designation input for identifying the user in the display window in process 34 (e1) g. If there is an input, a plurality of representative two-dimensional reference CAD data obtained by photographing a person in advance in a plurality of directions (possible directions when viewed from the installation position of the terminal) in process 34 (e1) h are read. Then, after converting the image data into a line drawing, a person in the image is detected by a two-dimensional matching process, the person closest to the specified coordinates is identified, and a recognition mark (for example, a person is included) (Rectangular etc.) is displayed, and generation of CAD data of the person is started. The generation and search of CAD data may be started while performing 3D measurement using video from another terminal whose position enters the field of view from the stage where the coordinates are designated. When the center member confirms the state and inputs OK if OK, the program side determines whether or not the center member has input OK with respect to the recognition result in process 34 (e1) i. Branch processing.

  If the input is OK, a command to start the monitoring service is transmitted via the telephone line to the mobile phone terminal that the user is calling in step 34 (e2) a. This command turns on the service ON state display function connected to the user's mobile phone. In addition, even if you hang up the phone in the user application, if you continue the monitoring service until you call again to end the service, the service ON status display function will continue to turn on even if you hang up the phone. You can do it.

  Next, in process 34 (e2) b, a person is continuously detected from the screen, and the specified person (user) is identified again as a person close to the previous specific position, or previously identified from the generated three-dimensional CAD data. It is determined and re-identified by the matching process of the three-dimensional CAD to display the video around the person and the video of the person around the person and record them. If the specified person (user) cannot be re-specified in process 34 (e2) c, an alarm is output to notify the center member and a monitoring service interruption is transmitted to the user. If the specified person (user) cannot be re-specified in the determination process 34 (e2) d, the process returns to the process 34 (e1) d to re-specify the user.

  If the specified person (user) can be re-specified, the next process 34 (e2) e is set so that the position becomes the center of the image information from the current position information of the specified person (user). , Select the required terminal, calculate the shooting direction of each terminal, and send the image information to each terminal, a communication function with the terminal on another phone line or the mobile phone the user is calling If there is an input from the center member to the user (or the user and the people in the vicinity) by processing 34 (e2) f by sending the command via the telephone line that the user is calling The terminal transmits a command for notifying the user's mobile phone or a terminal around the user and data (such as voice data).

  In addition, when a suspicious person is recognized from a suspicious person's database from surrounding people, the information is automatically transmitted. Then, the monitoring service is continued by repeating the processing from the processing 34 (e2) b until the end of the service. When the service ends, the program records the account information of the user who made the call in process 34 (e2) h and ends. This is an example in which the user tracking process is automatically performed. Of course, if one center member is assigned to each terminal, all people may follow the user manually, or the part set manually in this example may be processed automatically. good. If automated, the burden on the center staff is reduced and the risk of losing sight is reduced. Further, if the automation efficiency is improved, it becomes possible to efficiently operate a plurality of terminals with a plurality of center members. For example, 10000 PC terminals can be operated by 10 center staff. In addition, the general monitoring center may be configured to be distributed and managed in each area instead of being provided at one place throughout the country. Moreover, the function sharing between the PC terminal of the monitoring center and the terminal installed at each site described here is only one example.

  In this example, a system configuration has been described in which a person or a specific person is identified by generating CAD data or searching after transmitting image information to the center PC. However, each terminal is equipped with a visual information processing device. In each terminal, 2D CAD data is transmitted to and received from other peripheral terminals from the generation of 2D CAD data to each other, and a plurality of terminals cooperate to perform 3D measurement and 3D CAD data. The processing function to generate the data is also provided on the terminal side installed in the field, and the data transmitted / received to / from the center is limited to two-dimensional or three-dimensional CAD data and representative image scenes, so that the data transmission amount is reduced. A simple system configuration may be used. In addition, since processing can be performed immediately at the terminal, CAD data can be generated and searched quickly.

  In addition, since there is a risk that crimes will increase in places where there is no monitoring terminal if a monitoring terminal is installed, the manufacturing cost of the monitoring terminal will be lowered by mass production, and many public and private places will be included. It is good to be able to install in place. Also, since it is expected that there will be more crime cases because people can not be identified later due to masking etc., it is good to install many terminals so that even if you escape you can not escape from the terminal network . If a suspicious person is specified, the suspicious person may be automatically tracked. In that case, in the unlikely event that the suspicious person is lost, the search command for the suspicious person is automatically distributed to the entire terminal around the place where the suspicious person is lost. It may be possible to search. You may make it expand the search area of the terminal to be used automatically with a predetermined ratio with the elapsed time from the time of losing sight. In this case, as a special measure, all terminals installed in public places and private places may be used for that purpose.

  Since such a method of use is also assumed, it is a matter of course that each terminal is equipped with a CAD data generation and search function. It is preferable that time shearing processing can be performed for a different command with respect to a plurality of different directions by capturing information.

  In addition, when the number of violent crimes increases, not only the user is informed of the danger by voice, but it may be operated in combination with equipment capable of firing paint bullets at each terminal toward the suspicious person. Suspicious person by squeezing the laser beam when using a facility that installs a laser on the terminal and irradiates a weak laser spot to inform the surroundings of the monitoring service status when the danger of a violent crime is imminent It may be possible to irradiate a threat. At this time, the system configuration is such that the PC terminal can be notified urgently to another police line using a separate telephone line. In an emergency, the system can be operated from the PC terminal of the police institution by switching to the operation of the police institution. It may be possible to quickly switch between stationary and intimidating operations for suspicious persons. When shooting a laser or intimidation bullet, a device such as a pan / tilt mechanism of a camera that sets the direction of irradiation of the laser or the like is provided at each terminal so that it can be remotely operated or automatically controlled. In order to scan image information of a normal terminal over a wide range, when a terminal having a configuration obtained by mirror scan is used, the mirror scan may be used also as a mechanism for setting the laser irradiation direction. Of course, apart from the terminal, a dedicated laser or a device for shooting intimidation may be installed at a ratio of about one to several terminals. When shooting a laser or intimidation, the direction of the laser or intimidation is detected by recognizing the hand or foot of the suspicious person from the 3D-CAD data and detecting the position of the suspicious person's hand or foot. May be set automatically. Even if the operator performs manual threatening shooting, if the aim is accidentally set to a location that would cause a fatal injury to a suspicious person, the safety is automatically corrected to a location that avoids the location that would cause a fatal injury. The upper interlock may be provided. By applying the visual information processing apparatus of the present invention, the head, body, and limbs of a suspicious person can be detected including the position and posture of each component and the combined state (the posture state of the whole body) of these components. Therefore, it is also possible to easily aim at the sight of the suspicious person by avoiding the deadly part.

  FIG. 133 illustrates a basic configuration of a program of a PC terminal used when extracting information from a monitoring center when the terminal has an image data storage function.

  In process 34 (f) a, the terminal designated by the center member is called on the telephone line, and the video (image information) in the designated direction at the time designated by the center member (from what time to what time of the month, what time) is displayed on the terminal. Send the command to be sent to the terminal. Next, in step 34 (f) b, the transmitted video is displayed on the PC terminal. Here, if specified, a person or a specific person is detected and only that person and its surroundings are displayed, or a recognition mark (for example, a rectangle containing a person) is displayed at a position corresponding to the person's position. Or using the image data of another terminal that captures the direction of the designated terminal to generate and display human 3D CAD data. Then, if there is an input of 3D CAD data generated from image information obtained by photographing a person who wants to look for or a person who imitates the searched person in the process 34 (f) c, a matching process is performed with the input. Only the video portion of a person with a high degree is displayed together with the matching degree.

  FIG. 135 shows an example of a terminal suitable for such a monitoring service. A configuration in which a small camera 35A in which a CCD camera 35b and a lens 35a are combined can be installed in a socket 35d provided on the terminal body 35e side by a connector 35C. I have to. A directional microphone 35m may be attached to the small camera 35A so that sound can be monitored and recorded. Of course, although not shown here, the terminal may be provided with a speaker so that a suspicious person or a surrounding person can be alerted as necessary from the center. The speaker may wear a dedicated loudspeaker from the cellular phone of the applicant so that the voice signal from the cellular phone can be heard from the loudspeaker. In this way, even if the applicant is targeted by a suspicious person, it can be stopped from the center even if it approaches a dangerous place.

  In this example, the socket 35d of the terminal 35e is provided in all directions uniformly so that the small camera 35A can be installed in a spherical shape in all directions. In this way, when the terminal is installed, it is possible to easily install a plurality of small cameras 35A only in a necessary direction. In this example, the camera 35A can be installed on a spherical surface. However, the camera 35A may be simply attached to the front and back of the open plate, or a triangular prism, triangular pyramid, four You may make it attach to surfaces, such as a prism, a quadrangular pyramid, a polygonal column, a polygonal pyramid. If so, for example, when installing a terminal on the wall of the building, there is no need to monitor the wall side of the building, so a lot of small cameras 35A should be installed on one side and not on the other side. Therefore, it is possible to assemble a terminal on which only reasonably necessary sensors are mounted according to the installation location. When a pole is built in the middle of the park and a terminal is installed, the small camera 35A is mounted in the 360 ° all-around direction. With such a configuration, it is possible to obtain a terminal with an imaging sensor capable of photographing a wide range with high resolution by combining a plurality of inexpensive small CCDs at a low cost. If the dome-shaped cover 35f is fitted in the groove 35g after the small camera 35A is installed, the assembly of the terminal is easily completed. Also, in the event of a failure, the cover 35f can be removed and the small camera 35 can be replaced only at the failed location, and the case where the installation location of the terminal is changed or the arrangement of the small camera 35A can be easily changed or increased or decreased. Can be supported.

  FIG. 136 shows a block diagram of the terminal of FIG. Each small camera 35A (1), 35A (2)... Can be connected to any position of each socket 35d (1), 35d (2), 35d (3), 35d (4), 35d (5). To. Further, the wiring from each socket recognizes the small camera 35A actually connected by the sensor data control circuit 36a, and performs data transmission control between the small camera 35A and the sensor signal processing LSI 36b. In the data transmission control, the sensor data control circuit 36a is instructed of a switching command signal such as which small camera 35A to take in based on the content of signal processing to be performed next by the sensor signal processing LSI 36b. The sensor signal processing LSI 36b recognizes the applicant, performs image tracking, and compresses the image information around the changed portion only when a difference occurs in the image information and changes the time to the image data recording device 36d. Operate to record with data.

  The microcomputer 36c manages time and outputs operation conditions such as an applicant's tracking command and a condition for periodically recording an image to the sensor signal processing LSI 36b based on a command from the center 34g via the wireless communication device 36e. At the same time, a response to a command from the center 34g or another terminal 34b is returned, and information from a nearby terminal entering via the wireless communication device 36e is recognized, and a necessary relay function is not seen, or This is performed using the sensor signal processing LSI 36b. When relay processing is performed, image information to be relayed to the image data recording device 36d may be temporarily recorded. When the sensor signal processing LSI 36b performs all tracking control, image recording, and relay functions, if the processing speed is severe, the sensor signal processing LSI dedicated to tracking processing, the sensor signal processing LSI dedicated to image recording, and the relay processing dedicated These sensor signal processing LSIs may be connected to the microcomputer 36c to perform parallel processing. Further, the signal processing may be provided with a function of analyzing the sound of the microphone and obtaining the direction thereof, and when recording the image data, the sound data of the microphone in the direction of the image may be recorded. If there is a microphone, it is possible to track the direction with a directional microphone, to generate 3D CAD data by processing an image with sound, or to record an image with sound. . In addition, an infrared sensor other than the camera and microphone, an electromagnetic wave sensor, an ultrasonic sensor, and the like may be mounted to monitor and record the signals.

  As described above, according to the monitoring service of the present invention, the suspicious person is not aimed at, the incident can be prevented in advance, and even if an incident occurs, the police investigation time can be shortened. Therefore, it becomes possible to build a safe society with few police officers.

  In addition, when an autonomous mobile robot becomes a society that moves in various places, the robot user and the manufacturer of the robot are responsible for safety, so such a monitoring system monitors the autonomous mobile robot. In this way, in the unlikely event that a dangerous situation occurs, it is possible to contact the robot user and promptly stop the robot on the dedicated line. In addition, if the robot manufacturer and the monitoring service company decide on commands for the dedicated emergency stop line in advance, specify the robot ID and stop it directly from the terminal, independent of the normal control of the robot. A command can be transmitted to the control circuit. If the target robot is identified by the ID, only the dangerous target robot can be stopped even when a plurality of robots are moving nearby. In addition, when providing this service to a user who drives a car, if the car is tracked and a monitoring service is performed and it is likely to collide with another car, or if the car is about to be driven in a dangerous state May be able to notify the driver of the car in advance, or in a situation that requires urgency, a brake signal can be sent directly from the terminal to the car control device. good. In the case of an automobile, usually the situation is recognized and only guidance is given to the driver. However, safety side control such as braking may be directly executed when the driver is delayed.

  In addition, when installing terminals for this monitoring service, install at least one terminal so that each other's terminals are within the field of view of the monitoring target, and monitor and suppress the situation even if the terminal itself is in danger. It becomes possible to do. In addition, image data taken from various viewpoints of the object to be searched, two-dimensional CAD data, or three-dimensional CAD data is transferred to the command from the center to each terminal, and the terminal side views from various viewpoints. If two-dimensional CAD data is generated and a target object having a high matching degree is searched from the image information to be photographed, the information is returned to the center. May be used for monitoring and searching the entire monitoring service area. Of course, these processes may be performed in parallel with response processes for other commands by time shearing.

  FIG. 134 shows a flowchart of basic processing of the microcomputer 36c of FIG. 136 (a). This is also an example. In the operation of the microcomputer 36, first, in a process 34 (g) a, a command to record the current video to the image data recording apparatus together with the current time is output to the sensor signal processing LSI. Next, in process 34 (g) b, a command sent from the wireless communication apparatus is read, and a process corresponding to the command is performed. As a specific example, (1) in the case of a data transfer command, data sent from a nearby terminal is fetched and transmitted and output as it is as data for the next predetermined terminal. (2) In the case of an image output command, a command to output image information in a specified direction at a specified time to a wireless communication device is output to the sensor signal processing LSI. (3) If there is no normal reception response from the transmission partner terminal within the predetermined time, the same data is retransmitted. If it continues multiple times, the next route is selected and transmitted. (4) If abnormality of downstream terminal is found, send to center. In the transmission data, in addition to the final transmission destination (center or specific terminal), when there are a plurality of route routes, a terminal that is routed is designated according to a predetermined rule.

  Next, in the discrimination process 34 (g) c, when a normal reception response cannot be obtained from the transmission partner, or when an abnormality is detected when there is a self-diagnostic sensor, a lamp or the like is displayed on the microcomputer in the process 34 (g) d. If a connection is established, an alarm is displayed to light it up and the process ends. If there is no abnormality, the process 34 (g) a may be operated repeatedly.

  FIG. 136B illustrates a basic flowchart of the sensor signal processing LSI. The sensor signal processing LSI 36b performs processing according to the content of the command according to the command from the microcomputer. Specifically, (1) in the case of an image recording command, a camera terminal switching command is given to the sensor data control circuit to capture the image data of the designated camera, and the image data recording device records it with the time data. When there are a plurality of camera terminals, image data from all the camera terminals is recorded while switching. If the capacity of the image data recording device is exceeded, the old data is overwritten. Also, an index that is easy to search should be recorded together. (2) In the case of an image output command, image data in a specified direction at a specified time is retrieved from the recording device, read, and output to the wireless communication device as transmission data together with transmission destination information. (3) If there is a CAD data generation function, two-dimensional line segment CAD data is generated and recorded in the image data recording apparatus. When two-dimensional line segment CAD data is received from another terminal, 3D measurement is performed from the positional relationship of the terminal, and three-dimensional CAD data is generated and recorded. A mechanism for repeatedly performing the operation process 36 (b) a is incorporated.

  FIG. 137 is a block diagram showing an example of the basic configuration of the eighteenth means of the present invention. FIG. 137 (a) shows a humanoid robot, but the present invention is not limited to a humanoid robot, and may be applied to any type of robot.

  In this example, the robot (a) side and the fixed control units (b) and (c) side are separated via the communication system 37k. In particular, the part of the control system 37w that performs real-time control by the virtual dynamic model is shown as (c), but (b) and (c) are fixed control units, and may be a single computer system. The blocks such as the voice analysis system 37n, the image analysis system 37o, and the action command analysis system 37u indicate software program blocks. However, the functional blocks, that is, the programs can be configured in various ways, and are not necessarily as in this example. Need not be.

  The image analysis system 37o among them corresponds to a system in which a function corresponding to the visual information control apparatus 100 described above is incorporated in this system. Of course, when the whole of the control units (b) and (c) is mounted on the robot (a), these programs are also made into LSIs to increase the speed and reduce the size and weight.

  The communication system 37k is a large-capacity high-speed wireless communication system with a large number of sensor signals and control command signals for each actuator. The communication system 37k is equipped with a mobile station that transmits and receives data on the robot side. The computer system is provided with a fixed station that inputs and outputs data. The robot head has a small and light stereo sensor head 37c. Moreover, the power source 37b of the robot has a charging system from the outside with a battery. The robot hand 37j has the same five fingers as a human being, but in most normal operations, it is not necessary to move all five fingers delicately, so three of them (thumb, forefinger, middle finger) Alternatively, the mechanism and the control system may be simplified by using a finger that can be delicately controlled. Each joint of the robot's neck, torso, arms, and legs has a servo motor. When a joint angle command signal is input, servo control can be performed to the joint angle. The update period of the target angle is set to about 10 msec as a guide, and servo control is stabilized by performing control while interpolating to a smooth target value of 2 msec or less in the servo control portion. In addition to obtaining each joint angle from the encoder of the servomotor, the robot includes a force sensor 37g for reaction force applied to each part, a force sensor 37g for each part, and each acceleration sensor 37i for detecting acceleration of the body or each part. Further, a sensor 37h for detecting the posture (tilt) and a gyro sensor may be mounted as necessary. Since the signal of the force sensor 37g is also captured, when a force as a disturbance acts on the robot body, the state can also be captured. In addition, a diagnostic sensor 37e for each part is provided so that it can always be checked that actuators such as each sensor and each motor are functioning normally.

  The self-diagnosis system 37r comprehensively judges the signals of these diagnostic sensors to judge the abnormal part of the robot and its degree, and outputs it to the voice output system 37q if necessary to indicate that it is abnormal. To let you know.

  In addition, the behavior planning system 37v plans an optimal behavior step for the purpose execution in consideration of the abnormal part of the current robot. For example, when the right arm is out of order, a work plan for working with only the left arm is made.

  The eyeball viewpoint control system 37p controls the driving of the eyeball of the stereo sensor head by the eyeball lens system driver 37d. The two cameras of the stereo sensor head 37c move together by moving the robot's neck up / down / left / right, but each camera can be moved up / down / left / right independently like a human eyeball. The camera can be turned by moving the eyeball without moving the robot's neck. At this time, the up / down / left / right rotation angles that determine the orientation of each camera are detected by a signal from an encoder of an eyeball drive motor or the like, and are input together with image information to the image analysis system 37o. As eyeball control, in addition to the up / down / left / right movement of the eyeball, the lens system may be driven to control zoom, focus, and iris. In this case, at least information on the zoom operation status is preferably input to the image analysis system 37o from the zoom drive system 37d. In the case of stereo measurement, the relationship between the position and orientation of the two cameras needs to be accurate, so the approximate current position of the eyeball is input to the image analysis system 37o by a sensor signal driven by an eyeball lens system driver. Although it can be used as known information, in order to accurately obtain the correspondence between the two cameras, that is, the calculation formula for coordinate transformation, the image analysis system 37o has obtained stable feature points of the actual line up to that point. The position relationship between the two cameras for stereo measurement is obtained more accurately each time the eyeball is operated using the corresponding 3 or 8 points selected from the CAD data generated in the above. When the zoom is driven and controlled, the zoom ratio is also calibrated with high accuracy.

  Moreover, although this example is via the radio | wireless communications system 37k, as long as it becomes mountable, you may mount in a robot.

  The acoustic analysis system 37n recognizes a command by listening to a command in words, and the behavior command analysis system 37u and the behavior planning system 37v generate a robot behavior plan corresponding to the command, but the behavior plan cannot be made In response, a command execution enable / disable response is output from the voice output system 37q through a speaker provided at the position of the mouth of the robot. In addition to the language extraction function, the acoustic analysis system 37n has a function of listening to sounds and identifying water flowing sounds, metal sounds, and what sounds, and environment recognition in cooperation with the image analysis system 37o. Self-directive microphones are attached to the left and right ears of the robot and the position of the sound source is estimated by searching the direction of the sound source based on the movement of the neck of the robot. An auditory guidance function for obtaining information for moving in the direction of the position identification function or the sound source may be provided. The image data from the stereo sensor head 37c is subjected to environment recognition and recognition of its own position by the image analysis system 37o, and those CAD data are recorded in the database 37t and updated as needed. The image processing may be operated in cooperation with the acoustic analysis processing.

  Also, the sensor signals from the robot, the information detected by the voice analysis system 37n, and the information analyzed by the image analysis system 37o are all input to the database 37t and updated in real time, so that other systems are also updated in real time. It is good to be able to refer to the information.

  The behavior command analysis system 37u executes a conversion process by associating a language command, which is a human language, with an operation command of the robot. If the verbal command is to “take this box to the next room next to the desk”, first, “this box” recognizes the master (human) who issues the command first. Although the acoustic analysis system 37n and the image analysis system 37o cooperate with each other, it recognizes the object (box) that is pointed to by the master's hand when the master says “this”. This box "is analyzed as object ID = C in the three-dimensional CAD data. Before this, it is necessary to recognize the box as one object from the environment in advance, but this was generated and registered in advance for the 3D CAD data of the box and was first generated by a three-dimensional search command. Search from the environment as a candidate with a high degree of matching and isolate it from the environment and set the object ID = C, or if this is specified by the master, it can be recognized in advance centered on the position ahead It is also possible to perform reference matching with an object file and replace it with object ID = C when instructed.

  Next, “neighboring room” needs to recognize the concept of the room on the environmental data of CAD data, and the word “neighboring” It must be recognized that it is an adjective meaning "the next in the". Then, if you hear that word, since the room you are currently in is ROOM1, if there is only ROOM2 in the environmental data, the room next to it will be recognized as the “next room”. When there are a plurality of candidate rooms, a voice output system may be used to confirm with the master until the command can be identified. Similarly, “next to the desk” is recognized to be the position of the coordinate G on the CAD data. If the desk is previously separated from the environmental CAD data as the desk ID, the coordinates can be specified by searching in the same manner.

  First, after moving to the next room ROOM2, the desk may be searched, and when it is found, the desk may be separated from the environment and registered after desk ID = DESK. Next to the desk, the floor surface with a space for a box is considered as the first candidate location, and its position coordinates are set. In this way, it can be replaced with a command controllable by the robot that takes the object ID = C from the current position P to the coordinate G.

  In addition, the word “hold it” is equivalent to a command to perform a series of actions, such as moving the robot's arm to grab the target object, moving to the destination, and placing the target object at the target. To do this, it is necessary to register in advance so that it can be converted into a command directly linked to the robot's operation from the human language.

  Here, you can register various correspondences between words and actions, but you may answer that you cannot execute because you cannot understand unregistered words, and you can ask the meaning of the words at that time. Then, the robot may be able to learn. Learning means associating a new human language with a robot motion command at this time. In addition to the learning operation command, an object name or the like may be registered. Adjectives should be defined at the beginning as much as possible because learning with words requires complex processing. Then, to the action planning system 37v, an instruction that can be executed by the robot (an instruction that is not a human language), for example, an arm is moved to grab an object at the coordinate B (ID = C), and a robot body (Body) is grasped. Move to the coordinate G, and output a command to place the object ID = C on the floor surface next to the desk ID = DESK at the coordinate G.

  The action planning system 37v receives a rough command input from the action command analysis system 37u, the configuration, specifications and performance of the robot motion drive system, the current robot position and posture information, and the latest environment of the current robot location. Under the conditions, the action plan is divided into further broken action steps. Specifically, create a movement plan route to move to the next room ROOM, plan a step to open / close the door if it is in the middle, and jump over it if there is a weir in the middle Also plan. These are planned with reference to three-dimensional CAD data as map data.

  The posture and force control planning system 37x creates a posture and force control plan for each step input from the behavior planning system 37v, and breaks down to specific command data for each joint angle. In that case, prepare multiple types of basic motion patterns that are often used (data that describes the specified simple basic robot movements in time-series joint angle command values). A first plan of a series of operation command signals may be created by selecting from the basic operation patterns and combining them. If the broken command data is not executable, the result is fed back to the action plan system 37v and the action plan is started again.

  If it is an operation command that can be executed as a robot (determined that the joint is operable), the result is input to the control prediction simulation 37y, and the current robot environment data, robot posture data, robot position Data, external force acting as a disturbance to the robot, force sensor data for each part corresponding to its own weight and reaction force due to robot motion, robot motion data (data for each part in terms of speed and acceleration) When the simulation is performed and the result is sufficiently according to the control plan, each joint angle is determined by the determination unit 37z and output to the servo control unit 37f of each joint. If the simulation results are not sufficient, restart the posture and force control plan, and in some cases, start again from the action planning system and confirm that the simulation can operate as planned, and then output a control command signal to the robot. .

  First, from the basic pattern that defines the movement of the robot, while referring to the simulation results, it is possible to rearrange the patterns from among the candidate patterns that have been prepared multiple times, or to modify the basic pattern data little by little. Since it takes time to find an optimal solution even if the data is changed unnecessarily, it is preferable to prepare and incorporate a correction rule (rule) that converges the simulation result for each operation pattern in advance. The simulation is a dynamic simulation. For example, the dynamic stability and controllability when jumping over a weir with a load like 37y (1), 37y (2), and 37y (3) are evaluated. . Of course, the robot's neck and eyeballs may be moved while flying to perform 3D measurement around the landing point and always measure the current position of the latest robot.

  In addition, when the robot performs a work such as pushing a wall, the posture and force control planning system 37x takes a plan of what force is applied to the wall, for example, taking time on the horizontal axis. The pressing force to the wall is taken on the vertical axis, and how the force is applied to the wall with time is specifically planned according to the purpose. Then, the control prediction simulation system 37y uses the three-dimensional CAD data to determine the positional relationship between the robot and the wall and the physical characteristic data of the wall (the spring constant corresponding to the hardness or the rigidity of the structure on the back side by suppressing the wall. If data such as whether the wall is moving or not, and what kind of characteristics it moves, if it is recorded as environmental data in advance, refer to that information and control the posture and force Based on the joint angle command of the robot generated by the planning system 37x, the robot is simulated, and it is confirmed by simulation whether the pressing force against the wall against the time can be given according to the planned target. Based on the above, if the joint angle command cannot be applied to the wall sufficiently as planned, change and simulate again Generates a joint angle command signal that can apply more force to the wall as planned, and if any external force occurs in the robot, the external force acting on the robot is also detected by the robot's force sensor. Therefore, by simulating under the condition that the external force is also acting, it is possible to generate an operation command signal that can withstand the disturbance and operate as planned. It is a series of motion trajectory command signals that execute the actions of each step broken by the action planning system 37v.The unit to be simulated is divided into parts that are particularly difficult to control and parts that correspond to motion commands just before execution. A joint motion angle command trajectory from a predetermined time to a predetermined time may be used, or a simulation of the motion command immediately before execution may be performed. While performing the down quickly, it may be able to perform a bit operation simulation of the previous several simultaneously.

  The simulation processing and the like can be executed at high speed by a supercomputer or the like if the robot control device is not mounted on the robot via wireless communication. If the simulation is executed using a small, lightweight and high-speed computer, the controllers (b) and (c) may be mounted on the robot. In any configuration, in this way, three-dimensional CAD data of the environment is generated in real time, and the position, posture and movement state of the robot in the real time are detected in real time, and then dynamically stable. Appropriate control commands for control can be obtained by simulation under the same conditions as the actual environment. Even at the moment when the robot jumps and flies in the air, the sensor detects what kind of posture, what kind of disturbance, what kind of speed and acceleration is flying, so how to shake the arm, If you twist, you can check in advance how the robot's posture changes in the air by simulation, so you can give the robot's motion command in the air that is most suitable for the situation at that time to land safely in the planned posture It is also possible to output, and it is possible to stably control a dynamic control operation that causes the robot to fly or bounce. In addition, basic motion patterns are prepared in advance, such as how to move the robot's body, arms and legs in the air and how the posture of the robot can be corrected in the air. good. Also, when the robot is moving over the weir and straddling it, an external force acts on the foot from the weir, so that force is applied to the foot. By making it possible to detect the magnitude and direction of each actuator's torque (servo motor current) or the force sensor attached to the foot structural member, the external force is also reflected in the simulation results. Since it is known in advance from the simulation result that the control data is overturned, it is also possible to correct the motion pattern data to avoid it before the actual robot falls over. For disturbance factors that often occur such as catching a foot or hitting the shoulder somewhere, if such a situation occurs in advance, it is possible to recognize the disturbance from the tendency of external force at an early stage If the disturbance can be recognized, the basic control pattern to recover from it is registered in the database in advance, so that the correction process of the operation data to recover at an early time can be converged. Anyway. When the disturbance is large, if a satisfactory operation command signal cannot be obtained within a predetermined time, for example, within 10 msec by performing simulation, a command signal that can be determined to be the best at that time is output. To.

  The simulation is performed again by the time output after the next 10 msec. The cycle time of 10 msec will not exceed that if the speed can be increased to a 2 msec cycle where the servo system can be stabilized, but the target command value is output at 10 msec, and the sensor input can be captured at a high speed cycle of 2 msec. Thus, a more appropriate motion command signal may be generated by capturing the state of the robot more precisely and in detail. In addition to controlling the posture of the entire robot, when assembling or disassembling parts on the desk, the 3D CAD data of the parts and fingers is generated by visual information processing, While detecting the reaction force acting on the object, handle the object to a predetermined posture, or perform a posture control or force control simulation so that a predetermined load is applied to the object. A control signal is output.

  In addition, when the robot runs or flies, the movement of the finger of the robot's hand has a small effect on the robot's posture control. When assembling or disassembling by holding an object on the desk, the simulation part of the arm and finger is simulated at high speed and accurately, and the appropriate simulation model is switched according to the situation. It is good to do.

  Also, if you have an object recognized by the visual information processing device or if you hold it with your finger, prepare the conditions for the simulation model when incorporating it into the simulation for each object ID in advance. You may make it incorporate in a simulation. The object mass, degree of freedom configuration, etc. are prepared in advance as object ID data, and the detailed position of the coupling location with the robot model and the coupling conditions (conditions such as firmly grasping or lightly grasping) The grasping condition when grasping is set from the positional relationship between the object recognized by the visual information processing and the hand of the robot, and the combination condition of how to grasp is set from the contents of the action plan. In the control of the force acting on the robot, when the robot is operated with sufficient stability, the reaction force acting on the robot's foot is detected while controlling the basic movement of the robot. Then, the optimal posture control of the robot may be determined while simulating so that the reaction force changes as a predetermined plan.

  This means may be applied not only to the humanoid robot but also to the control of a bird or fish robot, an automobile, a two-wheeled vehicle, an airplane, or the like. In the case of a car, in addition to the speed and acceleration of the car body, the wheel shaft and the rotation speed and reaction force of the wheel are detected, and the environment where the car is currently moving and the position and orientation of the car in the visual information processing device To detect and control the target vehicle's movement trajectory and the collision with obstacles while simulating the target reaction force of the wheel corresponding to the stability of the vehicle as planned. The present invention may be applied as a control device for a simple automobile. Even in a flying or floating robot, the visual information processing device detects the environment and the position and posture conditions of the robot in the environment in real time, and the reaction force includes the resistance of the fluid acting on the body and the wing If it is controlled by simulating the force that acts on the rotor, propeller, screw, etc. in real time as per the predetermined plan, it is stable and optimal considering the strength and power limitations of the robot itself. A control device for outputting a control command can be obtained.

  Also, the database in the visual information processing device or another database may be used, but the information obtained as a result of recognizing the situation is registered as the current environment information, and the database is used when referring to the current environment information. You may be able to do it. For example, a situation where another robot A or human B has the object C is recognized and specified for the robot A or human B, and the object C is recognized and specified, and the object C is held by the robot A or the human B. Recognize that the situation is related to the position of the robot and the parts of the arm of robot A or human B and that it has the position of object C (of course, “having” means that the hand and the object are 2 It is defined that the contact position is more than one point and the contact position is a position that is stably supported when the object is supported by those points. In addition, (1) what time (recognized time data), the database of the definition should be prepared for all recognizable operations. , (2) where (the location of the recognized robot A or human B), (3) what the situation is (the robot A or human B had the object C) in the database as the situation recognition result If registered, for example, if the command information says “Go to the robot A with a red box”, search the database and hold the red box (object C). If you have data on the robot A, you can see where and when the robot A with the red box was recognized in the past. First, make an action plan to search for the robot A around that location. It is also possible. If it is not in the database, an action plan is made to search for robot A with a red box after receiving the command, but if it is searched, the situation will be registered next time by registering the situation in the database. From now on, it will be possible to make an action plan while referring to the situation data at this time.

  FIG. 138 to FIG. 148 exemplify basic process follow charts of main circuit portions (a), (b) and (c) in FIG.

  FIG. 138 is a basic process flowchart of the acoustic analysis system. In the process 37 (f1) a, the audio data from the left and right microphones of the robot body (a) is read. When continuous processing is performed without interruption, processing is performed on previously stored audio data while storing the audio data in the built-in memory for a predetermined time.

  Next, in process 37 (f1) b, if the speech data contains human language, the human language is extracted by speech signal pattern matching or the like and decomposed into words to be divided into the subject and predicate. Then, the robot commands are assembled from pre-registered words (nouns, verbs, adjectives, adverbs, etc.) and output to the action command analysis system.

  In the process 37 (f1) c, if the sound data includes a sound signal of a bird, a sound of water, a sound of hitting a metal, or a characteristic sound that has been registered in advance, the sound signal The pattern is extracted by pattern matching, etc., and the direction in which the sound is heard is obtained from the direction of the left and right microphones and the directivity characteristics of the microphone, and the recognized sound type and sound data and the direction in which the sound is heard are determined from the current position and time data. At the same time, it records and updates it as part of the environmental data in the database and outputs the direction and type of sound to the image analysis system. Record human voice. As for the current position, information on the position of the robot body, the position of the robot head, and the orientation is acquired by referring to the self-position data in the database. Also, when recognizing the direction and type of sound, the accuracy of recognition should be improved by using the type of object and its position information that can be recognized by the object being photographed or sent from the image analysis system. To do.

  Next, in the process 37 (f1) d, when there is a command from the action planning system, the current direction of the designated sound source and the volume of the sound, and the direction of the same sound source detected from the past microphone position and orientation, The position of the robot main body with respect to the sound source is specified from the sound volume, and is written and updated as information detected by the acoustic analysis system of the self-position parameter in the database.

  If there is a command from the action planning system in process 37 (f1) e, the direction to be taken when the robot body is guided in the direction of the designated sound source is written in the detection parameter in the database (update) To do). These processes are repeatedly executed.

  FIG. 139 and FIG. 140 show a basic processing flowchart of the image analysis system. In process 37 (f2) a, first, image data from the left and right cameras (eyeballs) of the robot body (a) is read. When continuous processing is performed without interruption, processing is performed on the previously stored image data while storing the image data in the built-in memory for a predetermined time.

  Next, in process 37 (f2) b, two two-dimensional line drawing CAD data is generated from the left and right image data, and the tracking process is performed while performing 3D measurement by stereo measurement to generate three-dimensional CAD data. While performing 3D measurement by tracking and updating 3D CAD data, the part matched by CAD data matching processing of various objects registered in advance is converted to the CAD data of that part as one object type ( 3D CAD data is generated by recognizing it as an object ID), and the generated 3D CAD data, 2D CAD data, image data, etc. are recorded and updated as part of the environmental data in the database together with the generation time data. . Register the CAD data for each part of the robot arm, hands and feet, and the object the robot is holding in advance, recognize it as a single object ID, and include its position and orientation information in the environment data. To update the record. Further, the current position is obtained by referring to the self-position data in the database and information on the position of the robot body, the head position of the robot, and the orientation. Further, information on the direction of the camera (eyeball) in the head and the zoom state, the focus state, and the iris state in the head that operates at higher speed is received directly from the eyeball lens system driver for reference. Also, when searching for the current position when image tracking is interrupted, information such as the direction of the sound source sent from the acoustic analysis system, as well as information such as environmental data recorded in the database, past self-position data, etc. The current position is specified (identified) so that tracking can be resumed by using.

  Next, in process 37 (f2) c, the position and orientation of the head are obtained from the current position and orientation information of the camera obtained when performing 3D measurement and the orientation information of the camera (eyeball), and the database is detected. Referring to each joint angle information of the robot in the parameter, the current position, direction and orientation of the body of the robot body are obtained from the position and orientation of the head, and a part of the information of the self-position data in the database As described above, the current position, direction, and posture information of the body of the robot body are recorded and updated. If the robot body's arm, hand, foot, or what it has is already recognized as 3D CAD data as an object ID, refer to the current position and orientation information of the robot body's current body. The position, the direction, and the posture information are corrected so as to be more accurate.

  Next, in process 37 (f3) a, if the image data contains characters, they are extracted by character pattern matching etc. and broken down into words, which are then divided into subject and predicate and registered in advance. Certain words (nouns, verbs, adjectives, adverbs, etc.) are output to the acoustic analysis system together with the type of object (object ID) for which characters are displayed.

  Next, if there is a command (visual guidance command) from the action planning system in process 37 (f3) b, the object of the specified robot body part (including the direction of the head camera (eyeball), etc.) When moving the ID or the object ID held by the robot body with the arm, leg or body along the specified direction or the specified target movement trajectory, the direction, distance, etc. to which the object should be guided Write (update) information into the detection parameters of the database. The information to be guided may be written as direction and size information for which the operation of each part of the robot body should be controlled.

  Next, in process 37 (f3) c, it is necessary to execute a visual guidance command from the action planning system, or to specify the generation direction of environmental CAD data from the action planning system. The position and orientation of the correct head is obtained from the current position and orientation of the head camera and the position and orientation in which CAD data is to be generated, and written (updated) in the detection parameters of the database. Necessary head position and orientation information may be written as direction and size information that should be used to control the operation of each part of the robot body.

  Next, in process 37 (f3) d, it is necessary to execute a visual guidance command from the action planning system, or to specify the generation direction of environmental CAD data from the action planning system and execute the command. The direction of the correct camera (eyeball) is obtained from the current position and orientation of the camera and the position and direction in which CAD data is to be generated, and an operation command signal is output to the eyeball viewpoint control system. Also, the camera zoom state, focus state, and iris state for making it easier to extract the color boundary line segment of the target object are obtained from the color boundary line segment extraction state, and output as an operation command signal to the eyeball viewpoint control system. . Processing is performed so that such processing is repeatedly executed.

  FIG. 141 shows a basic processing flowchart of the eyeball viewpoint control system. First, if there is a command from the image analysis system in the process 37 (f4) a, based on the command, operation command signals for the camera (eyeball) direction, zoom state, focus state, and iris state are output to the eyeball lens system driver. To do. The eye lens system driver controls the direction of the left and right cameras and the lens system of the robot based on the command signal, takes the camera direction and the current position of the lens system as camera sensor information, performs feedback control, and performs image control. Outputs the current state directly to the analysis system.

  Next, in the discrimination process 37 (f4) b, it is discriminated whether or not there is a camera (eyeball) direction or a camera zoom control command from the image analysis system. Return and try again. In No, it progresses to process 37 (f4) c.

  If there is a command from the action planning system in process 37 (f4) c, the direction of the camera (eyeball) and the zoom control command of the camera are output to the eyeball system driver based on the command.

  In process 37 (f4) d, the information output from the action planning system to the eyeball system driver based on the command is output to the image analysis system. The image analysis system records and updates this information in the detection parameters of the database. In this way, repeated processing is performed.

  FIG. 142 shows a basic processing flowchart of the audio output system. In the process 37 (f5) a, when there is a command from the action planning system, a sentence composed of pre-registered words (nouns, verbs, adjectives, adverbs, etc.) attached to the command is received, Output to the speaker of the robot as voice data corresponding to the word. The words from the action plan system output that the commanded command cannot be executed, the reason, and a call when it is necessary to obtain cooperation from the surroundings in order to execute the action plan. The type of speech data corresponding to the words is basically the same as the type of words possessed by the acoustic analysis system.

  If there is a command from the self-diagnosis system in process 37 (f5) b, the voice data corresponding to the content of the abnormality output from the self-diagnosis system is output to the robot speaker. The processing function is to repeatedly execute these processes.

  FIG. 143 shows a basic process flowchart of the self-diagnosis system. In the process 37 (f6) a, information of the self-diagnosis sensors (temperature sensors, current sensors, etc. of the actuators and their drivers) of each part of the robot is captured.

  In process 37 (f6) b, the normality / abnormality of each part is determined from the threshold value or below the threshold value based on the detection signal from the self-diagnosis sensor (temperature sensor, current sensor, etc.) of each part of the robot. A threshold value is set in advance as a condition for determining an abnormality.

  Next, in process 37 (f6) c, if an important abnormality to be notified to the action planning system has occurred as a result of the determination, the result is output to the action planning system. The types and levels of important abnormalities to be notified to the action planning system should be set in advance.

  Next, in process 37 (f6) d, if an important abnormality that should be notified to the surroundings by voice occurs as a result of the determination, the result is output to the voice output system. The types and levels of important abnormalities to be notified to the surroundings should be set in advance.

  Next, when there is a command from the action planning system in process 37 (f6) e, the specified location is diagnosed in more detail and accurately based on the command, and the result is output to the behavior planning system. These processes are repeatedly executed.

  FIG. 144 shows a basic process flowchart of the action command analysis system. In the process, command information in human language input from the speech analysis system in process 37 (f7) a is read as a subject or predicate composed of words (nouns, verbs, adjectives, adverbs, etc.).

  Next, in process 37 (f7) b, the noun is replaced with the pre-registered object ID, the verb, the adjective, the adverb are also replaced with the pre-registered word ID, and the robot is pre-registered from human words. Convert to a recognizable corresponding ID.

  Next, in process 37 (f7) c, from the subject and the predicate, a pre-registered command pattern (for example, (1) have an object ID, or (2) change the object ID to an adverb ID (slowly and quickly) Equivalent ID), Adverb ID (above or below the object ID representing the location), Verb ID (move), Example (3) Object ID (right hand) to Adverb ID (up or down) In response to the verb ID (raise), it is replaced with a command executable by the robot and output to the action planning system. In the case of command information that does not apply to the registration pattern, output to the voice output system via the action planning system, “You cannot understand the command. Please command in another language.” These processes are repeatedly executed.

  FIG. 145 shows a basic process flowchart of the action planning system. In the process 37 (f8) a, the command pattern registered in advance from the action command analysis system (for example, (1) have an object ID, for example, (2) the object ID is an adverb ID ( Slowly or early ID), adverb ID (above or below the object ID representing the location), verb ID (move), example (3) object ID (right hand) adverb ID (up, Or, the command information based on the verb ID (raise, etc.) is read in below).

  Next, in process 37 (f8) b, the noun is replaced with the pre-registered object ID, the verb, the adjective, the adverb are also replaced with the pre-registered word ID, and the robot is pre-registered from human words. Convert to a recognizable corresponding ID.

  Next, in process 37 (f8) c, based on the rules prepared in advance for executing the command information, the moving speed (fast, slowly to the actual moving speed), moving amount, moving coordinates, moving route, etc. Make it concrete. In actualization, the environment data and self-position data in the database are used to refer to the environment where the robot is currently located, the posture and motion state of the robot, and the like, and to realize it with appropriate numerical values according to the rules.

  Next, in process 37 (f8) d, the specified command information is divided into the minimum robot operation units (action steps) prepared in advance, and the operation conditions for each action step are specifically set. The instructions are output step by step to the posture & force control plan. If there is an NG signal from the posture & force control plan, review the plan and re-output. Examples of action steps are grabbing the object ID with strength A with the right hand, grabbing the object ID with strength A with the left hand, moving the arm from A to B at a speed C while grabbing, and moving the robot from A to B. Change at C, move from coordinate A to B at speed C, move from coordinate A to B while holding the object, move right foot on object A with speed B, staircase ID = A Move from position B to position C at speed D, change robot posture from A to B at speed C, move object A across speed B to position C, push wall A with force B, etc. It is. Further, in concretely, the environment data in the database and the self-location data are used to refer to the environment where the robot is currently located, the posture and motion state of the robot, etc., and materialize according to the rules.

  Next, in the process 37 (f8) e, when the command information cannot be executed or when it is necessary to obtain cooperation in order to execute the action plan, the voice output system generates a sentence by the generated words. Output to. When there is a request necessary for executing the plan to the acoustic analysis system, the image analysis system, and the eyeball viewpoint control system, the command signal is output. These processes are repeatedly executed.

  FIG. 146 shows a basic processing flowchart of the posture & force control plan. In the process 37 (f9) a, the action step, which is the minimum operation unit of the robot, is read from the action planning system together with specific operation conditions such as speed and movement amount. Examples of action steps are grabbing the object ID with strength A with the right hand, grabbing the object ID with strength A with the left hand, moving the arm from A to B at a speed C while grabbing, and moving the robot from A to B. Change at C, move from coordinate A to B at speed C, move from coordinate A to B while holding the object, move right foot on object A with speed B, staircase ID = A Move from position B to position C at speed D, change robot posture from A to B at speed C, move object A across speed B to position C, push wall A with force B, etc. It is.

  Next, in step 37 (f9) b, the action step is determined from the current robot posture by referring to the environment data in the database, the environment of the location where the current robot is located, the posture of the robot, the operation state, etc. To generate an operation command pattern of each joint of the robot (time-series angle command values (θri (t) t = 0 to n) of each joint. When there is an NG signal from the control prediction simulation, Review the motion command pattern and re-output.Even if the command content from the action planning system does not change, the environment data in the database, self-position data, the environment where the robot is currently located, the posture and motion state of the robot, etc. Is updated sequentially so that the latest motion command pattern of each joint from the current position is always generated and output. A rule is registered in advance, and is generated according to the rule (including a rule that is improved by referring to the update result of the control result database from the initial generation rule). However, if data or a generation program is prepared with the step specifications of the stride or staircase as parameters, the operation command pattern is corrected by using it to correct the initial position / posture and the end position / posture. To generate an action pattern by specifying a waypoint from the current position to the target position on the hand or toe, prepare a program that generates an action command signal from the teaching process of a general industrial robot. The motion command pattern is generated by using the motion command pattern for controlling the force (reaction force) to a predetermined value. In order to achieve this, an action to operate the amount of movement of the contact point when there is no pressed object that causes reaction force of the contact point of the robot while referring to the hardness and elastic characteristics of the object pressed by the robot in the database A command pattern is generated, and when NG is reviewed, it is reviewed according to a predetermined rule prepared in advance with reference to reference data to be fed back.

  Next, if the action step cannot be executed in process 37 (f9) c, the action step is output together with the reason information to the action planning system together with the NG signal. These processes are repeatedly executed.

  FIG. 147 shows a basic process flowchart of the control prediction simulation. In the process 37 (f10) a, the current position, posture, and other conditions of the robot are read from the self-position data in the database and initial conditions of the simulation model are set.

  Next, in process 37 (f10) b, three-dimensional CAD data is read from the environmental data in the database, and environmental conditions are set in the simulation model.

  Next, in process 37 (f10) c, the motion command pattern of each joint of the robot (time series angle command value (θri (t) t = 0 to n) of each joint) is read from the posture & force control plan. Since there may be changes depending on the situation of the robot and the environment, the latest operation command pattern is read every time.

  Next, in process 37 (f10) d, it is confirmed that the time series angle command value θri (t) t = 0 of each joint in the motion command pattern of each joint of the robot matches the initial posture of the simulation model. If it is within the allowable range, the initial conditions of the simulation model are matched. If it is not acceptable, it returns to the posture & force control plan together with an NG signal and an error code indicating an initial posture mismatch.

  Next, in process 37 (f10) e, the time series angle command values (θri (t) t = 0 to n) of the respective joints are input to the robot simulation model as the motion command patterns of the respective joints of the robot. Run a dynamic simulation. The simulation result is recorded in the motion trajectory of each part of the robot on the way so that it can be referred to during simulation evaluation. Since the action step is long, if an operation simulation that can sufficiently evaluate the simulation (dynamic stability and safety) is performed on the way, it may be interrupted without performing t = 0 to the end of n. .

  Next, in process 37 (f10) f, the simulation result is evaluated (dynamic stability, safety, and the degree of achievement of the objective are evaluated), and the action step objective cannot be achieved at a predetermined level or higher. & Returns NG signal and supplementary information to force control plan. If the evaluation result is OK, OK is output to the determination circuit for each joint angle θri. These processes are repeatedly executed.

  FIG. 148 shows a basic processing flowchart for determining each indirect angle. In the process 37 (f11) a, when there is a command of the evaluation result OK from the control prediction simulation, the motion command pattern of each joint of the robot (the time command angle command value (θri (t) t of each joint) = 0 to n) The data of t = 1 of the next angle command value θri (t) of each joint is read and output to the servo control circuit for each joint axis. The motor is driven and controlled based on the angle command value, and the current position information of the motor, such as an encoder, is input from the motor and used for feedback control. Update this process so that it is repeated.

  With the basic processing flow described above, it is possible to realize a robot that can actually perform work autonomously according to a human command. This is just an example, so there are various variations in the way of sharing the blocks and the detailed contents of the functions that realize each functional block, so the most reasonable depending on the function and performance of the target robot A simple configuration may be used. In addition, this example uses a humanoid robot as an example, but it enables horses, cows, cars, airplanes, guard dogs, etc., to understand instructions in human language so that work can be performed autonomously. May be. The difference between robots and humans and animals realized in this way is that humans and animals will act to ensure their own safety if an abnormality occurs during their work. In the case of, the robot can be repaired even if it breaks down, so by setting the priority order of self-protection low, the program will execute the action plan with the highest priority on executing instructions or ensuring safety for the surroundings. Will be able to.

  As described above, according to the present invention, as-built 3D-CAD data can be automatically generated in real time, and in combination with a CAD data search function, as a visual guidance control device for various automatic machines such as robots, A visual information processing device applicable to a navigation system, a mobile inspection device, a monitoring service system, and the like, and an application system thereof can be obtained.

It is a block diagram which shows the basic composition in the visual information processing apparatus of this invention. It is a block diagram which shows another basic composition in the visual information processing apparatus of this invention. It is a flowchart which shows the basic flow of a process of the image information processing apparatus which implement | achieves this invention. It is drawing which illustrates the content of the process (1) shown in FIG. It is drawing which illustrates the content of the process (2) shown in FIG. It is drawing which illustrates the content of the process (3) shown in FIG. It is drawing which illustrates the content of the process (4) shown in FIG. It is drawing which illustrates the content of the process (5) shown in FIG. It is a figure which shows the concept of utilization of the floating line segment produced | generated as CAD data. It is a figure which shows the concept of utilization of the floating line segment produced | generated as CAD data of a cylinder or a sphere. It is a figure which shows an example of the definition as CAD data of a special curved surface. It is a conceptual diagram which shows embodiment in the case of updating CAD data of this invention. It is a conceptual diagram which shows embodiment of the 2nd means of this invention. It is a conceptual diagram which extracts an outline using the color information of this invention. It is a block diagram which shows embodiment of the 10th means of this invention. It is a block diagram which shows other embodiment of the 10th means of this invention. It is a block diagram which shows other embodiment of the 10th means of this invention. It is a block diagram which shows other embodiment of the 10th means of this invention. It is a flowchart which shows embodiment of a process of the visual information processing apparatus in the case of implementing the 10th means-the 11th means of this invention as a visual information processing apparatus only for robot control. It is a conceptual diagram explaining embodiment of the 11th means of this invention. It is a conceptual diagram explaining embodiment of the 11th means of this invention. It is a conceptual diagram explaining embodiment of the 11th means of this invention. It is a conceptual diagram explaining embodiment of the 11th means of this invention. It is a conceptual diagram explaining embodiment of the 11th means of this invention. It is a conceptual diagram explaining embodiment of the 11th means of this invention. It is a conceptual diagram explaining embodiment of the 12th means of this invention. It is a conceptual diagram explaining embodiment of the 11th means of this invention. It is a block diagram of the visual information processing apparatus which is embodiment of the 11th means of this invention. It is a conceptual diagram explaining embodiment of the 11th means of this invention. It is a conceptual diagram explaining other embodiment of the 11th means of this invention. It is a block diagram of the visual information processing apparatus which is embodiment of the 1st means-3rd means of this invention, 10th means, and 11th means. It is a flowchart of the main process which the visual guidance chip | tip in embodiment shown in FIG. 31 performs. It is a flowchart of the main process which the visual guidance chip | tip in embodiment shown in FIG. 31 performs. It is a detailed flowchart of 3D coordinate calculation and CAD data generation processing in the process (2) shown in FIG. FIG. 35 is a detailed flowchart of additional generation processing in 3D-CAD generation processing of processing (4) and processing (9) shown in FIG. 34. FIG. It is a detailed flowchart of the point data update process in the 3D-CAD generation process of the process (4) and the process (9) shown in FIG. FIG. 35 is a detailed flowchart of line segment data update and addition processing in the 3D-CAD generation processing of processing (4) and processing (9) shown in FIG. 34. FIG. FIG. 35 is a detailed flowchart of surface data update and addition processing in 3D-CAD generation processing of processing (4) and processing (9) shown in FIG. 34. FIG. It is a conceptual diagram of the data update in the 3D-CAD data generation process shown in FIGS. FIG. 39 is a diagram depicting the concept of generating and updating new data in the 3D-CAD data generating process shown in FIGS. 35 to 38 by applying it to the concept of generating and updating general graphics. It is drawing which shows the concept of the basic Example by the stereo measurement by two cameras. It is drawing explaining the method of calculating | requiring the line segment in which the corresponding point of a feature point exists. It is drawing which shows an example of the processing method of a stereo measurement. It is drawing which expanded P4 periphery by an example of the processing method of a stereo measurement. It is a drawing showing the concept of error P4 (r1) related to the shooting distance and resolution and error radius P4 (r2) due to the contour line. It is drawing which shows the concept of the error sphere P4 (r2) related to the imaging distance, the resolution of the imaging device, and the imaging relationship (interval) of the imaging device. An error shape P4 (δ) of a point P4 that is first generated by the stereo camera from the two imaging devices of the viewpoints O1 and O2 is shown. An error shape P4 (δ) of a point P4 that is first generated by the stereo camera from the two imaging devices of the viewpoints O1 and O2 that have moved is shown. It is a drawing showing a concept of expressing an error by approximating a sphere having an error radius δR in a direction connecting P4 from a PC point so as to be between δL lengths around P4. It is drawing explaining the tracking method which determines the position and attitude | position of O2 coordinate system in O1 coordinate system. It is a figure explaining an example of the method in the case of a monocular camera system when three known points are needed initially. It is drawing explaining the method of narrowing down the corresponding feature point candidate using color information and other rules on a two-dimensional image in the case of stereo measurement. It is a figure explaining an example of the narrowing-down method of a corresponding point candidate in the case of tracking using the color information of a color boundary line part, etc. on a two-dimensional image. It is a figure explaining an example of the narrowing-down method of a corresponding point candidate in the case of tracking using the color information of a color boundary line part, etc. on a two-dimensional image. A part of a line segment obtained as a result of extracting a color boundary line segment of an image is shown. It is drawing explaining some examples of the color surface extraction method. It is a drawing for explaining an example of a method of narrowing down corresponding surfaces, corresponding line segments, and corresponding points when a plurality of the same patterns exist in one image. It is drawing which shows the image which converted the image information input from the imaging device into two-dimensional CAD data. It is drawing explaining the concept of a matching method. It is drawing explaining the concept of a matching method. It is drawing explaining the concept of a matching method. It is drawing explaining the concept of a matching method. It is drawing explaining the example of the discrimination logic in a matching method. It is drawing which shows the example at the time of extending a matching process to three dimensions. It is a figure explaining the example of narrowing down when the data of the corresponding feature point actually does not exist. It is drawing which shows the example in the case of using a vector matching method for a search function. It is drawing explaining the concept of the error at the time of coordinate conversion by tracking. It is drawing which shows a fundamental example of the 7th means of this invention. It is a figure which shows a basic example of the 7th means of this invention, and shows an example of the preparation method of a color boundary line segment from an inflection point. FIG. 69 is a diagram for explaining a case where a feature point and a line segment are extracted by applying the method of FIG. 68 and a case of another method. It is drawing which shows a fundamental example of the 8th means of this invention. It is drawing which shows a fundamental example of the 9th means of this invention. It is drawing which shows the structure of the basic application of the visual information processing apparatus of this invention, the basic command with respect to a visual information processing apparatus, and an extended command. It is a functional block diagram of the visual guidance chip | tip in FIG. It is a functional block diagram of the visual guidance chip | tip in FIG. It is a functional block diagram of the visual guidance chip | tip in FIG. It is a functional block diagram of the visual guidance chip | tip in FIG. It is a functional block diagram of the visual guidance chip | tip in FIG. It is a functional block diagram of the visual guidance chip | tip in FIG. It is a functional block diagram of the visual guidance chip | tip in FIG. It is a functional block diagram of the visual guidance chip | tip in FIG. It is a functional block diagram of the visual guidance chip | tip in FIG. It is a functional block diagram of the visual guidance chip | tip in FIG. It is a functional block diagram of the visual guidance chip | tip in FIG. It is a functional block diagram of the visual guidance chip | tip in FIG. It is a functional block diagram of the visual guidance chip | tip in FIG. It is a functional block diagram of the visual guidance chip | tip in FIG. It is a functional block diagram of the visual guidance chip | tip in FIG. It is a functional block diagram of the visual guidance chip | tip in FIG. It is a functional block diagram of the visual guidance chip | tip in FIG. It is a functional block diagram of the visual guidance chip | tip in FIG. It is an operation | movement timing chart of the visual guidance chip | tip in FIG. It is an operation | movement timing chart of the visual guidance chip | tip in FIG. It is an operation | movement timing chart of the visual guidance chip | tip in FIG. It is an operation | movement timing chart of the visual guidance chip | tip in FIG. It is an operation | movement timing chart of the visual guidance chip | tip in FIG. It is an operation | movement timing chart of the visual guidance chip | tip in FIG. It is an operation | movement timing chart of the visual guidance chip | tip in FIG. It is an operation | movement timing chart of the visual guidance chip | tip in FIG. It is a basic process flowchart in the functional block diagram of the visual guidance chip | tip in FIG. It is a basic process flowchart in the functional block diagram of the visual guidance chip | tip in FIG. It is a basic process flowchart in the functional block diagram of the visual guidance chip | tip in FIG. It is a basic process flowchart in the functional block diagram of the visual guidance chip | tip in FIG. It is a basic process flowchart in the functional block diagram of the visual guidance chip | tip in FIG. It is a basic process flowchart in the functional block diagram of the visual guidance chip | tip in FIG. It is a basic process flowchart in the functional block diagram of the visual guidance chip | tip in FIG. It is a basic process flowchart in the functional block diagram of the visual guidance chip | tip in FIG. It is a basic process flowchart in the functional block diagram of the visual guidance chip | tip in FIG. It is a basic process flowchart in the functional block diagram of the visual guidance chip | tip in FIG. It is a basic process flowchart in the functional block diagram of the visual guidance chip | tip in FIG. It is a basic process flowchart in the functional block diagram of the visual guidance chip | tip in FIG. It is drawing which shows an example in the case of describing the connection relation of CAD data by data. It is drawing which showed the example of things, such as a mobile phone, as an example of movable imaging | photography object. It is drawing which shows the typical shape of FIG. It is drawing which shows the concept of the two objects which are moving. It is a basic example of the 16th and 17th means of this invention, and is a figure which shows several examples at the time of applying to the various test | inspection apparatus at the time of taking a nuclear power plant as an example. It is drawing which shows the concept of the as-built three-dimensional CAD data of the automatically generated periphery part of a pool bottom face weld line, taking a crawler type VT inspection apparatus as an example. It is a system configuration block diagram showing an example of a crawler type VT inspection device. It is drawing which shows the image when the CAD data which registered the pool and the weld line beforehand as CAD data, and the CAD data of the as-built weld line produced | generated by image processing are displayed in an overlapping manner. It is drawing which shows an example of the movement locus | trajectory data file which inputs an instruction | indication to a main control routine. It is drawing which shows the relationship between a world coordinate system and the robot coordinate system of a ROV main body. It is drawing which shows the example of the file which registers the control logic of robots, such as ROV. It is drawing which shows the concept of the CAD data which registers the concept of a world coordinate system and a ROV coordinate system, and basic environments, such as a pool, beforehand. It is drawing which shows an example of a definition file of basic environment data or shape data of ROV itself. It is a flowchart of the basic processing of the main control routine. It is a flowchart of the basic processing of the main control routine. 3 is a flowchart of basic processing of an image processing routine. 3 is a flowchart of basic processing of an image processing routine. It is a conceptual diagram of the monitoring service provision system which shows a fundamental example of the 21st-24th means of this invention. FIG. 129 is a block diagram illustrating a basic example of a specific hardware configuration of the general monitoring center in FIG. It is a flowchart which illustrates the basic program of the monitoring service operated with a PC terminal. It is a flowchart which illustrates the basic program of the monitoring service operated with a PC terminal. It is a flowchart which illustrates the basic program of the monitoring service operated with a PC terminal. It is a flowchart which illustrates the basic program of the monitoring service operated with a PC terminal. It is drawing explaining an example of the terminal suitable for the monitoring service. FIG. 136 is a functional block diagram of the terminal of FIG. 135. It is a functional block diagram which shows the humanoid robot which is a fundamental example of the 18th means of this invention. It is a flowchart of the process which the main circuit part of the humanoid robot shown in FIG. 137 performs. It is a flowchart of the process which the main circuit part of the humanoid robot shown in FIG. 137 performs. It is a flowchart of the process which the main circuit part of the humanoid robot shown in FIG. 137 performs. It is a flowchart of the process which the main circuit part of the humanoid robot shown in FIG. 137 performs. It is a flowchart of the process which the main circuit part of the humanoid robot shown in FIG. 137 performs. It is a flowchart of the process which the main circuit part of the humanoid robot shown in FIG. 137 performs. It is a flowchart of the process which the main circuit part of the humanoid robot shown in FIG. 137 performs. It is a flowchart of the process which the main circuit part of the humanoid robot shown in FIG. 137 performs. It is a flowchart of the process which the main circuit part of the humanoid robot shown in FIG. 137 performs. It is a flowchart of the process which the main circuit part of the humanoid robot shown in FIG. 137 performs. It is a flowchart of the process which the main circuit part of the humanoid robot shown in FIG. 137 performs.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10, 10a, 10b ... Imaging device, O1, O2 ... Viewpoint of imaging device, 30 ... Acceleration sensor, 40 ... Gyro sensor, 100 ... Visual information processing device (processing device), 200 ... Storage device, 300 ... Another CPU, etc. , 400 ... Driver, 401 ... Actuator, 21r ... Reactor, 21A ... Underwater ROV, 21B ... Crawler travel type VT device, 21a ... Arm mechanism, 21C ... Suspended camera, 21e ... Marker line such as laser, 23e ... Main Control routine, 23g ... image processing routine, Ow ... world coordinate system, Or ... robot coordinate system, 34b ... terminal of monitoring service system, 901 ... object in environment (imaging object), 909 ... weld line as imaging object, 34c ... Mobile phone of the applicant who receives the monitoring service, 34d ... Suspicious person, 35b ... Small CCD, 35m ... Small directional microphone, 6a ... sensor data control circuit, 36b ... Sensor signal processing LSI, 36c ... microcomputer, 36d ... image data recording device, 36e ... wireless communication device, 37y ... control forecasting simulation section.

Claims (25)

In a visual information processing apparatus including an image information processing apparatus that inputs image information obtained by photographing an object with a movable imaging apparatus and generates CAD data of the object, and a storage device that stores the generated CAD data ,
The image information processing apparatus includes an update target determination processing function for determining whether or not the generated new CAD data is an update target of the already generated CAD data, and the update of the existing CAD data and the new CAD data to be updated. A CAD data accuracy comparison processing function for comparing the accuracy, and a CAD data update processing function for updating the existing CAD data with the new CAD data when the accuracy of the new CAD data is higher than the accuracy of the existing CAD data to be updated,
The function to generate new-generation CAD data is a stable measurement among the points measured in stereo by associating the feature points in the image information of a plurality of imaging devices captured at the same time from the feature parts of the image information. Visual information characterized in that it is realized by executing a method for obtaining three-dimensional coordinate data in such a manner that corresponding points in image information due to movement of the imaging device are related by tracking at least three points. Processing equipment. In a visual information processing apparatus including an image information processing apparatus that inputs image information obtained by photographing an object with a movable imaging apparatus and generates CAD data of the object, and a storage device that stores the generated CAD data ,
The image information processing apparatus distinguishes the outline of the CAD data from the actual line, and the function of generating the CAD data associates the feature points in the image information of a plurality of image capturing apparatuses captured at the same time. The stereo measurement is performed from the characteristic part of the image information, and at least three of the points measured in stereo are tracked to follow the corresponding points in the image information due to the movement of the imaging device. A visual information processing apparatus which is realized by executing a method for obtaining dimensional coordinate data. In a visual information processing apparatus including an image information processing apparatus that inputs image information obtained by photographing an object with a movable imaging apparatus and generates CAD data of the object, and a storage device that stores the generated CAD data ,
The image information processing apparatus is configured so that the CAD data has information related to the time when the CAD data is generated, and the function of generating the CAD data is a feature point in the image information of a plurality of imaging devices captured at the same time. The corresponding points in the image information due to the movement of the imaging device are related by performing stereo measurement from the characteristic part of the image information and performing stereo measurement and tracking and tracking at least three stable points among the stereo measured points. Thus, the visual information processing apparatus is realized by executing a method for obtaining three-dimensional coordinate data.   The feature point position in the image information changes depending on the moving object, or the feature point position in the image information changes because the image pickup apparatus moves, or is obtained from each image pickup apparatus by shooting with a plurality of image pickup apparatuses. Corresponding point search for associating feature points whose feature points change in image information is performed by converting image information obtained from the imaging device into two-dimensional CAD data composed of color boundary line segments connecting the feature points. A visual information processing apparatus characterized in that, by converting, a similarity between feature parts in an area excluding at least a background area divided by a color boundary line segment is compared and a higher one is narrowed down as a corresponding point candidate.   5. The similarity of feature parts to be compared is obtained by calculating the similarity of a vector up to another feature point subtracted from the corresponding point candidate, or the similarity between the vector and the color information of the feature point at the tip of the vector. A visual information processing apparatus characterized in that it is combined with a method of narrowing down the corresponding similarity as a corresponding point.   6. The visual image processing device according to claim 4, wherein the color boundary line segment is extracted by connecting inflection points of image information such as a luminance signal as the color boundary line segment. In a visual information processing apparatus including an image information processing apparatus that inputs image information obtained by photographing an object with a movable imaging apparatus and generates CAD data of the object, and a storage device that stores the generated CAD data ,
The image information processing apparatus includes an update target determination processing function for determining whether or not the generated new CAD data is an update target of the already generated CAD data, and the update of the existing CAD data and the new CAD data to be updated. A CAD data accuracy comparison processing function for comparing the accuracy, and a CAD data update processing function for updating the existing CAD data with the new CAD data when the accuracy of the new CAD data is higher than the accuracy of the existing CAD data to be updated,
A method for extracting feature points in image information directly or indirectly by capturing image information obtained from an imaging device uses both feature points of line segments and feature points of closed-loop figures as feature points. A visual information processing apparatus characterized by being extracted. In a visual information processing apparatus including an image information processing apparatus that inputs image information obtained by photographing an object with a movable imaging apparatus and generates CAD data of the object, and a storage device that stores the generated CAD data ,
The image information processing apparatus includes an update target determination processing function for determining whether or not the generated new CAD data is an update target of the already generated CAD data, and the update of the existing CAD data and the new CAD data to be updated. A CAD data accuracy comparison processing function for comparing the accuracy, and a CAD data update processing function for updating the existing CAD data with the new CAD data when the accuracy of the new CAD data is higher than the accuracy of the existing CAD data to be updated,
A visual information processing apparatus characterized in that a shooting position of a camera of an imaging apparatus and image information (texture information) are recorded in association with CAD data. In a visual information processing apparatus including an image information processing apparatus that inputs image information obtained by photographing an object with a movable imaging apparatus and generates CAD data of the object, and a storage device that stores the generated CAD data ,
The image information processing apparatus includes an update target determination processing function for determining whether or not the generated new CAD data is an update target of the already generated CAD data, and the update of the existing CAD data and the new CAD data to be updated. A CAD data accuracy comparison processing function for comparing the accuracy, and a CAD data update processing function for updating the existing CAD data with the new CAD data when the accuracy of the new CAD data is higher than the accuracy of the existing CAD data to be updated,
A visual information processing apparatus having a function of obtaining a position and / or orientation of a moving imaging device and a function of outputting the obtained position information and / or orientation information.   10. The image information processing apparatus according to claim 1, wherein the image information processing apparatus receives a search command signal for CAD data, and outputs a search function for searching for CAD data based on the command signal and information on the searched CAD data. A visual information processing apparatus having a function.   11. The visual information processing apparatus according to claim 10, further comprising a command information input interface for designating a range for generating the three-dimensional CAD data.   12. The visual information processing apparatus according to claim 10, wherein the CAD data search command information includes a two-dimensional search command.   The search function according to claim 10, wherein the search function is a vector similarity to another feature point subtracted from a corresponding point candidate of 2D or 3D CAD data to be searched, or a vector similarity and a vector tip A visual information processing apparatus characterized in that it is realized in combination with a method of using the similarity of color information possessed by feature points and narrowing down the corresponding points as corresponding points.   14. The method according to claim 10, wherein at least a part shape of three-dimensional CAD data and a connection condition between each part are provided as a database in advance, and a part is determined according to a predetermined rule of the part connection condition based on the information of the database at the time of searching. A visual information processing apparatus characterized in that search is performed by generating search CAD data by changing the combination condition.   15. The visual information processing according to claim 10, further comprising an interface for inputting a determination function for the searched object, command information for using the determination function, and an interface for outputting information of the determination result. apparatus. In mobile inspection equipment,
A means for generating the three-dimensional data of the mobile environment in advance or while moving, a functional means for obtaining the position and / or posture of the moving inspection apparatus, and the three-dimensional CAD data A mobile inspection apparatus comprising functional means for recording inspection result data in association with a part of three-dimensional CAD data corresponding to an inspection target part specified from position information of an imaging apparatus.   In Claim 16, the function which sets the planned movement path | route of a movement inspection apparatus previously, The function which calculates | requires the difference | error of the planned movement path | route preset and an actual movement path | route from the positional information on the imaging device in three-dimensional CAD data, A movement characterized by providing a function for setting a rule for performing movement control of the movement inspection apparatus so as to reduce the error, and a control unit for executing movement control of the movement inspection apparatus based on the rule of the movement control. Inspection device.   A visual information processing device is provided that receives the image information of the imaging device and processes the input image to generate CAD data of an object existing in the moving space, and generates three-dimensional CAD data of the environment in which the robot operates to generate CAD data. A means for specifying the current position of the robot in the robot, a means for specifying the current posture and operation state of the robot from various sensor signals from the robot, and the three-dimensional CAD data and CAD data of the environment in which the robot operates. Means for reflecting the current position of the robot, the current posture and motion state of the robot to the simulation model of the robot, means for simulating the movement of the robot, and inputting the motion command signal of the robot based on the movement plan to the simulation means To determine whether the simulation result has reached a predetermined evaluation level And a means for correcting an operation command signal to the robot based on the simulation result when the simulation result does not reach the predetermined evaluation level, and an operation command to the robot having the best evaluation level under the predetermined condition. A control apparatus for a robot, wherein a signal is output to the robot.   A robot comprising the control device according to claim 18.   A manipulator, a mobile robot, a character reading device, an object recognition device, a measurement device, a CAD data generation device, a navigation device, an inspection device, comprising the visual information processing device according to claim 1. Systems such as picking devices, autonomous control robots, monitoring devices, automobiles, ships and airplanes. In a monitoring service providing system that centrally monitors information sent from one or more terminals at an information processing center,
A monitoring service providing system which searches for a monitoring target user according to a request from the user, tracks the monitoring target user during monitoring, and continuously monitors the visual monitoring user and its surroundings. In a monitoring service providing system that centrally monitors information sent from one or more terminals at the center,
Each terminal is provided with a function of recording predetermined information in a predetermined period in association with time, and recording data at a predetermined time of a predetermined terminal can be taken out by a request from a user. Monitoring service provision system.   23. The monitoring service providing system according to claim 21, further comprising a state expression means for notifying a person around the user that the monitoring service is being performed during monitoring.   24. The monitoring service provision according to claim 21, further comprising means for providing information to a user or a user and a person in the vicinity when a situation to inform the user occurs during monitoring. system.   In a terminal device for acquiring sensor information, a plurality of sensor installation ports are provided in advance, and one or a plurality of sets of sensors can be connected to a predetermined installation port when assembling the terminal. A terminal device for acquiring sensor information characterized by.

Images