JP6289844B2 - Diagnostic device and diagnostic method - Google Patents

Description

  Embodiments described herein relate generally to a diagnostic apparatus and a diagnostic method.

  There is known a method of using input information from an operator in order to detect a failure or malfunction of a sensor based on image information.

  In addition, in order to capture images for OCR and video coding, there is also a method of using a scanner that measures the shape of an object, adjusts the top and side surfaces of the courier service according to the position, and develops it into a flat image.

JP2013-103162A

  Since the former method relies on the subjective judgment of the operator, the judgment criteria for sensor failure and malfunction are different, which may lead to erroneous detection.

  In the latter method, when the correction image creation fails, the plane development view is distorted, and automatic coding by OCR as well as video coding that teaches the classification destination visually is impossible. There is a case. Further, even if a sensor such as a scanner continues to be incompatible, the failure may not be automatically determined and the device may be operated without knowing the failure state.

  An object of the present invention is to provide a diagnostic apparatus and a diagnostic method that are excellent in diagnosis of an image sensor.

  The diagnostic apparatus according to the embodiment includes an input unit, an image development unit, a diagnostic unit, and an output unit. The input means inputs image data acquired by one or more image sensors. The image expansion unit generates a developed image from an image of a rectangular parallelepiped object included in the image data. The diagnostic means diagnoses the image sensor based on the developed image. The output means outputs a diagnosis result.

1 is a system configuration diagram illustrating an example of an image processing system according to an embodiment. It is a flowchart which shows an example of the basic process of the image processing system which concerns on embodiment. It is a flowchart which shows an example of the basic process at the time of sensor abnormality diagnosis (at the time of failure diagnosis) of the image processing system which concerns on embodiment. It is a figure which shows an example of the expansion | deployment image when an image sensor and another sensor are normal. It is a flowchart which shows an example of the basic process of sensor failure analysis. It is a figure which shows the 1st example of the expansion | deployment image when abnormality has generate | occur | produced in the sensor 103. FIG. It is a figure which shows the 2nd example of the expansion | deployment image when abnormality has generate | occur | produced in the image sensor 103. FIG. It is a figure which shows an example of the output result when the image sensor 103 is normal. It is a figure which shows an example of the output result when abnormality has generate | occur | produced in the image sensor 103. FIG.

  Hereinafter, embodiments will be described with reference to the drawings.

  FIG. 1 is a system configuration diagram illustrating an example of an image processing system according to the embodiment, FIG. 2 is a flowchart illustrating an example of basic processing of the image processing system according to the embodiment, and FIG. It is a flowchart which shows an example of the basic process at the time of sensor abnormality diagnosis (at the time of failure diagnosis) of the image processing system concerned.

  For example, the image processing system shown in FIG. 1 is part of a video coding system. The video coding system reads an image of mail or courier, detects a destination information area from the image, recognizes an address or the like included in the destination information area, and classifies the mail or courier based on the recognition result. On the other hand, when the address or the like cannot be recognized, the video coding system displays the destination information area on the display, receives the input of the address or the like by the operator, and sorts the mail or the courier based on the input address or the like.

  As shown in FIG. 1, the image processing system includes a belt conveyor 102, an image sensor 103, other sensors 104 (distance, angle sensor, etc.), a sensor information input unit 105, an image development unit 106, a development image analysis unit 107, and a sensor. A failure diagnosis unit 108, a diagnosis result output unit 109, an image display unit 110, a diagnosis result display unit 111, a warning display unit 112, and a failure information database 113 are provided.

  As shown in FIG. 1, the belt conveyor 102 conveys a rectangular parallelepiped mail or courier 101, and the image sensor 103 and other sensors 104 photograph the mail or courier 101 from multiple directions. For example, the first image sensor 103 captures a mail or courier 101 from the first direction and outputs a first image corresponding to the mail or courier 101. The second image sensor 103 captures the postal matter or the courier 101 from the second direction, and outputs a second image corresponding to the postal matter or the courier 101. In this embodiment, a case of photographing a rectangular parallelepiped mail or courier 101 will be described. However, the abnormality diagnosis of the present embodiment can also be applied to a case of photographing a cylindrical mail or courier 101. .

  The image developing unit 106 generates a developed image from an image acquired by shooting. For example, the image developing unit 106 generates a developed image from the first image and the second image. Alternatively, the image developing unit 106 generates a developed image based on the image acquired by the image sensor 103 and the distance information acquired by another sensor 104 (for example, a distance sensor).

  The developed image analysis unit 107 analyzes the developed image. The sensor failure diagnosis unit 108 diagnoses an abnormality (failure, failure) of the sensor based on the developed image. The diagnosis result output unit 109 outputs (displays) a diagnosis result of sensor abnormality (failure, malfunction). The image display unit 110 displays a developed image, and also displays a restored image (a restored image corresponding to the postal matter or the courier 101) generated based on the developed image. The diagnosis result display unit 111 displays a diagnosis result of sensor abnormality (failure, malfunction). The warning display unit 112 displays an abnormality when it is diagnosed that an abnormality (failure or malfunction) has occurred in the sensor.

  Next, each process by the image processing system will be described with reference to FIGS.

  As shown in FIG. 2, the sensor information input unit 105 acquires image information and other information from the image sensor 103 and other sensors 104 (ST201). Here, the acquired continuous image information may be an image taken from the image sensor 103 or a continuously developed image that has already undergone image processing. When the input image is a continuous image taken from the image sensor 103, the image expanding unit 106 expands the image. Next, the sensor failure diagnosis unit 108 performs sensor diagnosis on the input continuously developed image (ST202). If the diagnosis result is normal (ST203, YES), the result is output (ST205). If the diagnosis is abnormal (ST203, NO), a warning is issued and the result is output (ST205).

  As shown in FIG. 3, the image development unit analysis unit 107 analyzes the input continuous development image (ST301), confirms the result of the analysis (ST302), and determines that it is normal (ST303, YES), “ The diagnosis result is output as “normal” (ST305). If it is determined as abnormal (ST303, NO), the cause of the failure is analyzed (ST304), and the diagnosis result including the analysis result is output (ST305).

  This will be described in more detail below.

  The image development unit 106 inputs a developed image of the rectangular parallelepiped mail or the courier 101 conveyed by the belt conveyor 102 to the developed image analysis unit 107. The developed image analysis unit 107 compares the developed image with a preset criterion, and the sensor failure diagnosis unit 108 estimates a sensor abnormality (failure, failure) based on the comparison result.

  Judgment criteria can be changed and added depending on the object. When targeting a rectangular parallelepiped, the length of the opposite side of the object (opposite sides) is expanded in parallel, each corner of the surface is a right angle, or the intersection line of adjacent surfaces is expanded. Whether they match may be set as a criterion.

  FIG. 4 is a diagram illustrating an example of a developed image when the image sensor 103 and the other sensors 104 are normal. As shown in FIG. 4, when the image sensor 103 and other sensors 104 are normal, a developed image 101c is obtained. As criteria for diagnosing sensor abnormalities (failures, malfunctions) from the developed image, the upper and lower sides of surface A are parallel to the left and right sides, and the right side of surface B and the upper side of surface C are the same length. Etc. may be set. If the parameters of the image sensor are known, a three-dimensional image of the object can be restored from the developed image. A determination criterion may be set for whether or not the image sensor 103 and the other sensors 104 operate normally for the restoration result of the three-dimensional image. For example, as a result of restoration from the developed images of the adjacent surfaces, whether or not they intersect accurately and whether or not the dimensions of the restored rectangular parallelepiped are within the normal range may be set as determination criteria.

  As a method for restoring the object from the developed image, 3 from the images taken from the plurality of image sensors 103 may be restored by using the input developed image and the calibration information of the image sensor 103 and other sensors 104. A three-dimensional image may be restored using the developed image and other information from the sensor 104. As an example, epipolar geometry can be calculated by detecting corresponding points (eight points or more) on two images taken from two image sensors 103, and a three-dimensional image is restored. Can do. Further, the image sensor 103 and other sensors 104 (distance, angle, etc.) can be restored. Other restoration techniques may be used.

  When a sensor failure or failure is estimated from the developed image, the sensor failure cause and failure level can be estimated based on the failure information database 113 constructed in advance. When the result of analyzing the developed image in ST302 and ST303 in FIG. 3 is confirmed, and it is determined that some failure has occurred in the sensor 103 and other sensors 104, the sensor failure diagnosis unit 108 The cause of failure of the sensor 104 is analyzed.

  FIG. 5 is a flowchart illustrating an example of a basic process of sensor failure analysis. The sensor failure diagnosis unit 108 executes processing for analyzing the failure state (ST501), refers to the image characteristics at the time of failure extracted from the developed image and the pre-built sensor failure information database 113, and the sensor 103 and other sensors. The cause of failure 104 is estimated, the estimated cause of failure is output (ST502), and the estimated failure level is output (ST503).

  FIG. 6 is a diagram illustrating a first example of a developed image when an abnormality occurs in the sensor 103. The sensor failure diagnosis unit 108 can diagnose that one of the image sensors 103 has a failure from the developed image 101 d created from the images taken by the two image sensors 103. From the image information and the sensor information, it can be seen that the result restored from the A side, the B side and the D side is normal, and the result restored by the combination of the A side, the B side and the C side is abnormal. Therefore, it can be presumed that some trouble has occurred in the image sensor 103 that has photographed the A surface, the B surface, and the C surface. Also, image features extracted from the developed image (each corner of the surface is not a right angle, the angle between the B surface and the C surface is not a right angle, character feature extraction failed from the C surface, and the C surface image is deformed) Etc.) and the failure information database 113, the cause of failure of the failure sensor can be estimated.

  FIG. 7 is a diagram illustrating a second example of a developed image when an abnormality occurs in the image sensor 103. Since the image characteristics of the developed images in FIG. 6 and FIG. 7 are different, even if it can be estimated that a failure has occurred in the same sensor, it can be estimated that the cause of the failure is different and the level of the failure is different. Even if it is possible to detect that the image sensor 103 is defective, when the image feature information extracted from the developed image cannot be collated with the information stored in the failure information database 113, the failure information database 113 is determined by the user's judgment and analysis. Can be added to.

  FIG. 8 is a diagram illustrating an example of an output result when the image sensor 103 is normal. When the developed image shown in FIG. 8 is input, the image display unit 110 displays the restored result at the lower left of the screen. FIG. 8 shows an example in which four surfaces are expanded and the four surfaces can be restored. The image display unit 110 can display the displayed restoration result at a different angle by a terminal operation such as a user's cursor or key, and can visually check the estimation result of the failure or malfunction of the sensor.

  FIG. 9 is a diagram illustrating an example of an output result when an abnormality occurs in the image sensor 103. The sensor failure diagnosis unit 108 refers to the sensor failure information database 113 constructed based on the developed image, so that the warning display unit 112 determines the estimated sensor failure, failure failure cause, level, and sensor. A warning can be issued accordingly. Further, the developed image and the three-dimensional restored image that issued the warning, the estimated cause of the failure of the sensor, the level, and the like are stored, and the sensor failure diagnosis unit 108 updates (reconstructs and adds information) the failure information database 113. Can do.

  An image sensor 103 that captures an object, an object developing unit 106, a sensor failure diagnosing unit 108 that diagnoses a sensor failure / failure based on the developed image, and a warning when a sensor is diagnosed as malfunctioning The display unit 112 and the like can operate independently.

  Further, the screen configuration shown in FIG. 8 is an example, and each element of the image configuration shown in FIG. 8 may be calculated by the same calculation device, or calculated by another calculation device. There may be. The computing device may operate separately from the image sensor 103 or may be incorporated as a part of the image sensor.

  The present embodiment will be summarized below.

(1) (Sensor fault diagnosis based on developed images)
The system according to the present embodiment is a sensor failure diagnosis device that is a part of a device that conveys a rectangular parallelepiped object that should be classified based on destination information, the image information and the sensor that photographed the object, and photographed the object Specific information, and create a developed image from the image of the target object, analyze the developed image, diagnose sensor failures and defects based on the developed image, and display the diagnosis results of the failure and defects. The developed image and the restored image are displayed, and a warning is displayed when it is diagnosed that the sensor is malfunctioning.

(2) (Criteria for sensor failure based on developed images)
Furthermore, the system according to the present embodiment, when diagnosing a sensor failure or malfunction, with a developed image created from an image obtained by photographing an object, by collating the developed image with a judgment criterion set in advance, Estimate sensor failure or malfunction.

(3) (Criteria: Parallel or right angle of the side of the developed image with respect to the rectangular parallelepiped)
Furthermore, when the system according to the present embodiment targets a rectangular parallelepiped object, the predetermined criterion is that the side of the developed image is a straight line, the opposite side is parallel, and the angle of each corner is It is close to a right angle. By comparing the image feature extracted from the developed image with the criterion, the sensor failure or malfunction is estimated.

(4) (Judgment criteria: Sensor failure diagnosed from the result of restoration based on the developed image)
Furthermore, in the system according to the present embodiment, when the cuboid object is restored from the developed image, the determination criterion set in advance determines that the restoration result is a cuboid, and the restored cuboid dimension is within the normal range. By doing so, sensor failure and malfunction are estimated.

(5) (Building sensor failure information database based on developed images)
Further, in the system according to the present embodiment, the failure diagnosis unit for diagnosing a sensor failure or failure based on the developed image uses the sensor failure information database constructed based on the developed image, and stores the accumulated failure information. Based on the above, sensor failure and malfunction are estimated.

(6) (Estimated faulty sensor based on developed image)
Furthermore, the system according to the present embodiment estimates a sensor that has failed from the developed image by referring to a sensor failure information database constructed based on the developed image.

(7) (Estimated failure cause of failure sensor based on developed image)
Furthermore, the system according to the present embodiment estimates the cause of the failure of the sensor that has caused the failure or failure from the image characteristics of the developed image by referring to the sensor failure information database constructed based on the developed image.

(8) (Estimated failure level of failure sensor based on developed image)
Furthermore, the system according to the present embodiment estimates the failure level of the sensor that has failed or malfunctioned from the image features of the developed image by referring to the sensor failure information database constructed based on the developed image.

(9) (A sensor is diagnosed by the user's visual inspection for the result restored based on the developed image and added to the failure information database)
Furthermore, the system according to the present embodiment diagnoses a sensor in which a failure or failure has occurred by visual observation of an operator, and adds the developed image feature to the sensor failure information database.

(10) (The result restored from the developed image is displayed on the screen and can be confirmed at different angles by user operation)
Furthermore, the system according to the present embodiment can display the restoration result of the object restored from the developed image on the screen, and can check at different angles by the user's operation. Is estimated.

(11) (Warning according to sensor failure cause, level, and sensor based on the developed image)
Furthermore, the system according to this embodiment refers to the sensor failure information database constructed based on the developed image, and warns the estimated sensor failure and failure information according to failure cause, level, and sensor. Put out.

(12) (Save warning image and restored 3D image)
Furthermore, the system according to the present embodiment stores the developed image and the three-dimensional restored image, the estimated cause of failure of the sensor, the level, and the like when a warning is issued after detecting a failure or malfunction of the sensor, Reconstruct database and add / modify information.

(13) (System configuration)
The system according to the present embodiment includes an image sensor that captures an object, an image development unit,
A failure diagnosis unit for diagnosing a failure or failure of the sensor based on the developed image and a warning display unit for diagnosing that a failure has occurred in the sensor are integrated.

  According to the above-described embodiment, a case where a sensor malfunctions or malfunctions is automatically detected for a rectangular parallelepiped mail conveyed by the belt conveyor 102 or a developed image created from an image taken with respect to the courier 101. Can be diagnosed. In addition, it is possible to automatically diagnose the cause and level of failure of the sensor in which an abnormality has occurred from the image features extracted from the developed image, and issue a warning.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and the equivalents thereof.

DESCRIPTION OF SYMBOLS 102 ... Belt conveyor, 103 ... Image sensor, 104 ... Other sensors, 105 ... Sensor information input part, 106 ... Image expansion | deployment part, 107 ... Development | deployment image analysis part, 108 ... Sensor failure diagnosis part, 109 ... Diagnosis result output part, DESCRIPTION OF SYMBOLS 110 ... Image display part, 111 ... Diagnosis result display part, 112 ... Warning display part, 113 ... Failure information database

Claims (15)

Input means for inputting image data acquired by one or more image sensors;
Image developing means for generating a developed image from an image of a rectangular parallelepiped object included in the image data;
Diagnostic means for diagnosing the image sensor based on the developed image;
An output means for outputting a diagnosis result;
A diagnostic device comprising: The input means inputs specific information of the image sensor,
The diagnostic apparatus according to claim 1, wherein the output unit outputs a diagnostic result corresponding to unique information of the image sensor.   The diagnostic device according to claim 1, wherein the output unit displays the developed image and a restored image of the object restored from the developed image.   The diagnostic apparatus according to claim 1, wherein the diagnostic unit diagnoses the image sensor based on a collation result between the developed image and a determination criterion. The determination criteria include a first reference indicating that a side included in the expanded image is a straight line, a second reference indicating that opposing sides included in the expanded image are parallel, and the expanded image. A third criterion indicating that the angle formed by each side is a right angle;
The diagnostic apparatus according to claim 4, wherein the diagnosis unit diagnoses the image sensor based on a result of collation between image feature information included in the developed image and the first, second, and third criteria.   The diagnostic apparatus according to claim 1, wherein the diagnostic unit diagnoses the image sensor based on a restored image of the object restored from the developed image.   The diagnostic apparatus according to claim 1, wherein the diagnosis unit estimates an abnormality of the image sensor based on an abnormality database of the image sensor constructed based on an abnormally developed image and the developed image. The input means inputs image data acquired by a plurality of image sensors,
The image development means generates a developed image from the image data acquired by the plurality of image sensors,
The diagnostic apparatus according to claim 7, wherein the diagnosis unit detects an abnormal image sensor from the plurality of image sensors based on the abnormality database and the developed image.   The diagnostic apparatus according to claim 7, wherein the diagnosis unit estimates a cause of abnormality of the image sensor based on the abnormality database and the developed image.   The diagnostic apparatus according to claim 8, wherein the diagnosis unit estimates an abnormality level of the image sensor based on the abnormality database and the developed image.   The diagnostic apparatus according to claim 7, wherein the diagnosis unit adds feature information of the developed image to the abnormality database in response to an instruction to add to the abnormality database. The diagnostic apparatus according to claim 3 , wherein the output unit changes the display angle in response to a display angle changing operation and displays the restored image. The input means inputs image data acquired by a plurality of image sensors,
The diagnostic means detects the cause and level of abnormality of each image sensor based on the abnormality database,
8. The diagnostic apparatus according to claim 7, wherein the output means displays an abnormality cause and an abnormality level of each image sensor.   The diagnostic apparatus according to claim 7, wherein the diagnostic unit updates the abnormality database based on the developed image from which an abnormality diagnosis is derived. Input image data acquired by one or more image sensors,
A developed image is generated from an image of a rectangular parallelepiped object included in the image data,
Diagnosing the image sensor based on the developed image;
A diagnostic method that outputs diagnostic results.

Images