JP6400260B1 - Data processing apparatus, programmable display and data processing method - Google Patents

Abstract

A data processing unit (14) for extracting a plurality of feature images on which a monitoring target is displayed from images constituting the video data, and a plurality of feature images extracted from the images constituting the video data by the data processing unit (14) And a display generation unit (16) for generating a composite image. The monitoring target is an object that moves at a monitoring location, and is, for example, a product that is conveyed by a belt conveyor in a process of producing the product. According to this configuration, it is possible to confirm the monitoring target without taking time.

Description

  The present invention relates to a data processing device, a programmable display, and a data processing method for processing video data.

  2. Description of the Related Art Conventionally, there is a technique for presenting a plurality of images taken at different times in a hierarchical manner and monitoring the movement of a subject that dynamically changes over a long period of time. In order to perform monitoring efficiently, it is desired to shorten the reproduction time.

  For example, Patent Document 1 discloses an image storage device that determines whether to capture an input image based on a monitoring target, and stores an image determined to be captured, thereby controlling an imaging interval so as to capture only the monitoring target. Is disclosed.

JP 2000-224542 A

  However, in the image storage device of Patent Document 1, although the reproduction time is shortened, the state of the monitoring target cannot be confirmed unless all the captured images are reproduced, particularly when the imaging time is long. , It takes time to check the monitoring target.

  The present invention has been made in view of the above, and an object of the present invention is to obtain a data processing apparatus that can shorten the time for checking a monitoring target.

In order to solve the above-described problems and achieve the object, the present invention provides a data processing unit for extracting a plurality of feature images on which a monitoring target is displayed from images constituting video data, and video data by the data processing unit. And a display generation unit that generates a composite image by combining a plurality of feature images extracted from the images constituting the image , and the data processing unit displays the state of the monitoring target at the time when the image constituting the video data was captured The acquisition unit that acquires the event information shown from the external device, and the event information acquired by the acquisition unit includes alarm information generated in response to detection of an abnormality or failure of the device controlled by the external device, based on this alarm information, select the image constituting the video data, Ru and an extractor for extracting feature image from the selected image.

  The data processing apparatus according to the present invention has an effect that the time for checking the monitoring target can be shortened.

The figure which shows the structure of the data processor in embodiment of this invention The figure which shows an example of the synthesized image in embodiment The figure which shows the structure of the data processing part in embodiment The figure which uses for description about the specific example which extracts the feature image from the image which comprises the video data in embodiment The flowchart which shows the procedure which acquires the event information in embodiment The flowchart which shows the procedure which extracts the feature image from the image which comprises the video data in embodiment The figure which uses for the description about the procedure which selects the event information in embodiment The figure which uses for description about the procedure which adds an event information image to the characteristic image in embodiment The flowchart which shows the procedure from the video data selection in the embodiment until the composite image is displayed The figure which shows the structure of the display production | generation part of the data processor in embodiment The figure which shows an example of the screen which selects the video data in embodiment The figure which shows an example of a structure of the video data in embodiment The figure which shows an example of the feature image extracted from the image by the extraction part in embodiment The figure which shows an example which adds an event information image to a feature image by the addition part in embodiment The figure which shows an example of the synthesized image in embodiment The figure which shows an example of a structure of the video data in embodiment The figure which shows an example of a structure of the video data in embodiment The figure which shows an example of the feature image extracted from the image by the extraction part in embodiment The figure which shows an example of the feature image extracted from the image by the extraction part in embodiment The figure which shows an example which adds an event information image to a feature image by the addition part in embodiment The figure which shows an example which adds an event information image to a feature image by the addition part in embodiment The figure which shows an example of the synthesized image in embodiment The figure which shows the structural example of the hardware of the data processor in embodiment

  Below, a data processor, a programmable indicator, and a data processing method concerning an embodiment of the invention are explained in detail based on a drawing. Note that the present invention is not limited to the embodiments.

Embodiment.
FIG. 1 is a diagram showing a configuration of a data processing device 1 according to an embodiment of the present invention. The data processing device 1 is realized by a programmable display. The programmable display device includes a display unit that displays an image, an operation unit that receives a user operation, a connection unit that connects an external device, and a storage unit that stores data. It is an electronic operation indicator that inputs values to an external device.

  The data processing device 1 extracts a feature image on which a monitoring target is displayed from a plurality of images constituting video data recorded for a predetermined time by an imaging device, and synthesizes and combines the extracted feature images. Generate an image. In the embodiment of the present invention, the monitoring target is an object, equipment, or the like that moves at the monitoring location. In the embodiment of the present invention, the feature image is described as being the same size as the image constituting the video data, but may be the same size as the monitoring target displayed in the image. That is, the characteristic image may be a part of the video data and the image on which the monitoring target is displayed divided into a plurality of areas. Hereinafter, a specific operation and configuration of the data processing apparatus 1 will be described.

  The data processing device 1 includes an operation unit 11 that receives a user operation, an external device connection unit 12 to which the external device 2 is connected, and an imaging device connection unit 13 to which the imaging device 3 is connected.

  The data processing apparatus 1 includes a data processing unit 14 that extracts a feature image that is an image of a portion where a monitoring target is displayed from an image that constitutes video data, and a storage unit 15 that stores the video data and the feature image. Prepare.

  The data processing apparatus 1 is generated by a display generation unit 16 that generates a composite image by combining a plurality of feature images extracted from each of a plurality of images constituting video data by the data processing unit 14, and a display generation unit 16. And a display unit 17 for displaying the synthesized image.

  The operation unit 11 includes a keyboard or a touch panel, and accepts user operations. Specifically, the operation unit 11 receives an operation for inputting additional information and an operation for selecting video data to be reproduced. The operation unit 11 generates an operation signal corresponding to the received operation, and outputs the generated operation signal to the data processing unit 14.

  The external device connection unit 12 is an interface to which the external device 2 is connected. The external device connection unit 12 is configured by, for example, a USB (Universal Serial Bus) connector or an RS-232C connector.

  The external device 2 is, for example, a PLC (Programmable Logic Controller) that controls an industrial machine or a sensing device that inspects a product. The PLC holds a device ID (Identification) stored in an industrial machine to be controlled. The device ID is information for identifying an industrial machine.

  The PLC controls the operation of the connected industrial machine. The PLC also monitors the state in which the industrial machine is operating. When the PLC detects an abnormality or failure in the industrial machine, the PLC generates alarm information and outputs the generated alarm information to the data processing device 1. The alarm information includes details of the abnormality or failure and time information when the abnormality or failure occurs.

  In addition, the sensing device performs, for example, an appearance inspection for inspecting the appearance of a product, or a position inspection for inspecting whether or not a part of the product is arranged at a desired position. The sensing device inspects the product, generates sensor information that is a result of the inspection, and outputs the generated sensor information to the data processing device 1. The sensor information includes an inspection result and inspection time information.

  The external device connection unit 12 outputs information input from the external device 2 to the data processing unit 14. The information input from the external device 2 is a device ID and alarm information when the external device 2 is a PLC. The information input from the external device 2 is sensor information when the external device 2 is a sensing device. Hereinafter, the device ID, alarm information, and sensor information are referred to as event information. Further, the event information may be, for example, information indicating a movement status of a product conveyed by a belt conveyor, information on a setup change operation, or the like in a process of producing a product. The information indicating the movement status of the product conveyed by the belt conveyor is information indicating the product delivery time and storage time, or information indicating the time when the product has passed a predetermined location. Further, the information on the work for changeover is information indicating a change in parameters set in the machine equipment.

  Note that the event information is given as information related to the images constituting the video data. In other words, the event information is information obtained corresponding to the time at which a certain image is taken and the factors that make up the video data. The image is stored in an image file format including a header part, a payload part, an index part, and the like. The event information is stored in, for example, an index part.

  The imaging device connection unit 13 is an interface to which the imaging device 3 is connected. The imaging device connection unit 13 is, for example, a USB connector or a communication connector for Ethernet (registered trademark).

  The imaging device 3 outputs the captured video data to the data processing device 1 after imaging at regular time intervals or after imaging with a trigger signal transmitted from the imaging device connection unit 13. The imaging device connection unit 13 outputs the video data input from the imaging device 3 to the data processing unit 14.

  The data processing unit 14 receives video data from the imaging device connection unit 13 and extracts a feature image, which is an image on which a monitoring target is displayed, from an image constituting the video data. Details of the procedure for extracting the feature image from the image constituting the video data will be described later.

  The video data is input from the imaging device 3 in a digital format, but may be input in an analog format. When the video data is input in the analog format, the imaging device connection unit 13 converts the analog format video data into the digital format video data by A / D (Analog to Digital) conversion.

  The display generation unit 16 generates a composite image by combining the feature image, which is the image of the portion where the monitoring target extracted by the data processing unit 14 is displayed, and the image where the background of the monitoring target is displayed. To do.

  For example, as illustrated in FIG. 2, the display generation unit 16 generates a composite image D by combining the feature image a2 in which the monitoring target B1 is displayed and the background image a1 in which the background of the monitoring target is displayed. To do.

  This embodiment can be applied to the field of monitoring a production line when products are produced in a factory. Thus, for example, as shown in FIG. 2, the monitoring target is a product to be produced, and the background image is a production line such as a belt conveyor.

  The display unit 17 is configured by a liquid crystal display device or an organic EL (Electro Luminescence) display device. The operation unit 11 and the display unit 17 may be configured by an operation display unit in which an operation function for receiving an operation by a user and a display function for displaying an operation screen are integrated.

  In the present embodiment, the data processing apparatus 1 selects video data and generates a composite image by synthesizing feature images extracted from each of a plurality of images constituting the selected video data. Images can be shown in a single composite image.

  The user can grasp the movement status of the monitoring target without replaying all the video data by confirming the composite image in which a plurality of feature images are combined. I can do it.

  Here, a specific configuration of the data processing unit 14 will be described. FIG. 3 is a diagram illustrating a configuration of the data processing unit 14. FIG. 4 is a diagram for explaining a specific example of extracting a feature image from images constituting video data. The data processing unit 14 acquires event information indicating a monitoring target state when the video data is captured or after the video data is captured, and the event information acquired by the acquisition unit 21 based on the event information acquired by the acquisition unit 21. An extraction unit 22 that selects an image constituting the video data and extracts a feature image from the selected image;

  Specifically, the extraction unit 22 selects an image captured at the same time as the time when the event information is acquired. That is, the extraction unit 22 searches for an image based on the time when the event information is acquired, and selects an image from the searched images. In addition, when the extraction unit 22 performs imaging using the trigger signal transmitted from the imaging device connection unit 13, the extraction unit 22 may search for an image based on the trigger signal and select an image from the searched images. As illustrated in FIG. 4, the extraction unit 22 extracts a feature image a1 that is an image of a portion where the monitoring target B1 is displayed from the selected image A1. Further, the extraction unit 22 outputs the video data, the feature image, and the event information to the storage unit 15. The storage unit 15 stores video data, feature images, and event information in association with each other.

  Here, the procedure for acquiring event information by the acquisition unit 21 will be described with reference to the flowchart shown in FIG.

  In step S1, the operation unit 11 accepts setting information. The setting information is information indicating the timing for acquiring event information. The timing at which the acquisition unit 21 acquires event information can be arbitrarily set according to the content of the setting information. For example, the acquisition timing may be a time at which imaging of the video data by the imaging device 3 is started, a time at which the imaging of the video data is performed, a time at which imaging of the video data is completed, a predetermined cycle, a multi-bit condition, It is done. The multi-bit condition refers to a result obtained by monitoring a change in a bit in a rising state and a falling state of a plurality of bits set in advance in the external device 2 such as a PLC and performing a logical operation using the plurality of bits. Is a condition that determines the timing for acquiring event information. For example, the external device 2 such as a PLC performs the logical operation described above, and the acquisition unit 21 of the data processing device 1 sets the timing for acquiring event information based on the result of the logical operation acquired from the external device 2 such as the PLC. It may be determined. The acquisition unit 21 of the data processing device 1 receives a plurality of bits from the external device 2 such as a PLC, performs a logical operation using the received plurality of bits, and based on the result of the logical operation, event information You may determine the timing which acquires. The operation unit 11 outputs setting information to the acquisition unit 21.

  In step S <b> 2, the acquisition unit 21 outputs setting information to the storage unit 15.

  In step S3, the acquisition unit 21 determines whether it is time to acquire event information based on the setting information. If the acquisition unit 21 determines that it is time to acquire event information (Yes in step S3), the process proceeds to step S4, and if it is determined that it is not time to acquire event information (No in step S3). Repeats the process of step S3.

  In step S <b> 4, the acquisition unit 21 requests the external device 2 to transmit event information, and acquires event information from the external device 2.

  Next, a procedure for extracting a feature image from an image constituting video data by the extraction unit 22 will be described with reference to a flowchart shown in FIG.

  In step S <b> 11, the imaging device 3 outputs video data to the extraction unit 22 via the imaging device connection unit 13. The video data is composed of a plurality of images.

  In step S <b> 12, the extraction unit 22 selects an image constituting the video data based on the event information acquired by the acquisition unit 21.

  In step S <b> 13, the extraction unit 22 identifies an image area with movement in the image based on the selected image. The image is composed of a plurality of pixels.

  Here, there is a method of using a motion vector as described in, for example, Japanese Patent Application Laid-Open No. 54-124927, for specifying an image area having motion in an image.

  In the method described in Japanese Patent Laid-Open No. 54-124927, a block a having an arbitrary size is defined in an image A constituting video data, and the same size as that of the block a is set at an arbitrary position in an image B immediately before the image A. The block b is determined, and the difference between the luminance value of the pixel constituting the block b and the luminance value of the pixel constituting the block a is calculated. Further, the location of the block b is sequentially changed in the image B, and the difference between the luminance value of the pixel constituting the block b and the luminance value of the pixel constituting the block a at the changed location is calculated. The block b having the smallest difference among the calculated differences is specified. It can be estimated that the identified block b is the same image portion as the block a of the image A. Then, a motion vector is calculated based on the difference between the identified position vector of the block b and the position vector of the block a.

  The extraction unit 22 identifies an image area with motion based on the motion vector. For example, when the magnitude of the calculated motion vector is larger than a certain value, the extraction unit 22 may specify that the image area is in motion. The identified image area includes a monitoring target.

  In step S14, the extraction unit 22 extracts a feature image from the image based on the identified image region.

  In the above description, the operation of extracting the feature image from the image corresponding to the timing at which the event information is acquired has been described. However, the operation is not limited to this operation. For example, as a modification of the present embodiment, the data processing apparatus 1 selects event information, selects an image based on the selected event information, extracts a feature image from the selected image, and extracts the extracted feature image. The composition which synthesizes and generates a composition picture may be sufficient. Specifically, a list of event information acquired by the acquisition unit 21 is stored in the storage unit 15. The display unit 17 displays a list of event information. Based on the list of event information displayed on the display unit 17, the user operates the operation unit 11 to select arbitrary event information. The extraction unit 22 extracts a corresponding image based on the selected event information.

  The operation unit 11 accepts selection of event information acquired by the acquisition unit 21. FIG. 7 shows a state where a plurality of event information, that is, a list of event information stored in the storage unit 15 is displayed on the display unit 17. The imaging device 3 transmits to the data processing device 1 information on the time at which the video data was captured, information on the location where the imaging device 3 is installed, and a unique ID assigned to the imaging device 3 together with the video data. May be. The display unit 17 displays not only the event information but also information on the time when the image data was captured, information on the location where the imaging device 3 is installed, and a unique ID assigned to the imaging device 3. It's okay.

  Specifically, the user operates the operation unit 11 based on the screen displayed on the display unit 17 to select one or more event information.

  The extraction unit 22 selects an image constituting the video data based on the event information received by the operation unit 11, and extracts a feature image from the selected image.

  Specifically, the extraction unit 22 reads the video data associated with the selected event information from the storage unit 15, and selects an image from the images constituting the read video data based on the event information. The feature image is extracted from the selected image. The extraction unit 22 outputs the extracted feature image to the display generation unit 16.

  The extraction unit 22 may be configured to read a feature image corresponding to the selected event information directly from the storage unit 15 and output the read feature image to the display generation unit 16.

  The display generation unit 16 combines a plurality of feature images to generate a combined image.

  Therefore, since the data processing apparatus 1 generates a composite image in which a plurality of feature images are combined based on event information arbitrarily selected by the user, the movement status of a specific monitoring target can be grasped and specified. It is possible to confirm the monitoring target in a short time.

  The acquisition unit 21 acquires processing information for processing the feature image. Specifically, the user operates the operation unit 11 to select video data, and sets processing information for the selected video data. The acquisition unit 21 acquires the processing information set from the operation unit 11. The processing information may be input from a terminal device that is the external device 2 connected to the external device connection unit 12. The acquisition unit 21 outputs the acquired processing information to the storage unit 15. The storage unit 15 stores the processing information in association with the corresponding video data.

  The processing information is information indicating the content of processing the monitoring target of the feature image by image processing. The processing information includes one or more of a density correction value for correcting the density of the monitoring target, a color correction value for correcting the color tone of the monitoring target, and a value indicating the magnitude of the motion vector.

  The data processing unit 14 includes a processing unit 23 that processes the feature image based on the processing information.

  Here, a specific operation of the processing unit 23 will be described. When the processing unit 23 receives selection of video data to be played back by the operation unit 11, the processing unit 23 reads out a feature image and processing information associated with the video data from the storage unit 15.

  When the processing information includes a density correction value, the processing unit 23 corrects the density of the monitoring target based on the density correction value.

  Further, when the processing information includes a color tone correction value, the processing unit 23 corrects the color tone to be monitored based on the color tone correction value.

  Further, when the processing information includes a value indicating the magnitude of the motion vector, the processing unit 23 selects a feature image of the motion vector having a size exceeding the value from the feature images.

  The data processing unit 14 reads event information from the storage unit 15, converts the read event information into an image to generate an event information image, and adds the event information image to the feature image monitoring target. And an adding unit 25.

  For example, the event information is stored in the index part that constitutes the image as described above. The image generation unit 24 reads event information from an index unit that constitutes an image, converts the read event information into an image, and generates an event information image. The adding unit 25 adds the event information image to the feature image extracted from the image from which the event information has been read.

  Specifically, when the selection of video data to be reproduced is received by the operation unit 11, the image generation unit 24 reads event information associated with the video data from the storage unit 15. The image generation unit 24 converts event information, which is text data, into an image, and generates an event information image.

  In addition, the acquisition unit 21 acquires instruction information for instructing a method for adding an event information image to the feature image monitoring target. The adding unit 25 adds the event information image to the feature image monitoring target based on the instruction information.

  For example, when the instruction information is “add event information image above the monitoring target”, the adding unit 25 adds the event information image C above the monitoring target B as shown in FIG. The position where the event information image is added is not limited to the upper side of the monitoring target, but may be the right side of the monitoring target, the left side of the monitoring target, the lower side of the monitoring target, or the like, or may be superimposed on the front of the monitoring target. . Further, FIG. 8 shows an event information image including an event ID and time information, but it is an example, and other information may be included. In addition, the user may arbitrarily select information included in the event information image by operating the operation unit 11.

  Here, a procedure from when video data is selected by the operation unit 11 to when a composite image is displayed on the display unit 17 will be described with reference to a flowchart shown in FIG.

  In step S21, the operation unit 11 receives selection of video data. Specifically, the user operates the operation unit 11 to select one or a plurality of video data from a plurality of video data displayed on the display unit 17.

  In step S <b> 22, the processing unit 23 reads out the feature image, event information, and processing information associated with the received video data from the storage unit 15.

  In step S23, the processing unit 23 processes the feature image based on the processing information. When the processing information includes a density correction value, the processing unit 23 corrects the density of the monitoring target based on the density correction value. When the processing information includes a color tone correction value, the processing unit 23 corrects the color tone to be monitored based on the color tone correction value. When the processing information includes a value indicating the magnitude of the motion vector, the processing unit 23 selects a feature image corresponding to a motion vector having a size exceeding the value.

  When the processing information includes all of the density correction value, the tone correction value, and the value indicating the magnitude of the motion vector, the adding unit 25 corresponds to the feature image corresponding to the motion vector having a magnitude exceeding the value. Select. Then, the adding unit 25 corrects the density of the monitoring target of the selected feature image based on the density correction value. The adding unit 25 corrects the color tone of the selected feature image to be monitored based on the color tone correction value.

  In step S24, the image generation unit 24 reads event information associated with the video data that has been selected in the process of step S21 from the storage unit 15, and converts the read event information into an image to generate an event information image. To do.

  In step S25, the adding unit 25 adds the event information image to the feature image monitoring target. When the instruction information is included in the processing information, the adding unit 25 adds the event information image to the monitoring target of the feature image based on the instruction information. Note that when the instruction information is not included in the processing information, the adding unit 25 adds the event information image to a predetermined position. The predetermined position is, for example, the right side of the monitoring target.

  Further, in the process of step S21, when selection of a plurality of video data is received by the operation unit 11, the process from step S22 to step S25 is repeated for each received video data.

  In step S <b> 26, the display generation unit 16 generates a composite image by combining the feature image to which the event information image has been added in the process of step S <b> 25 and the image in which the background to be monitored is displayed.

  Therefore, the data processing apparatus 1 selects the video data and generates a composite image by synthesizing the feature image to which the event information image is added from the images constituting the selected video data, so that the event information image is added. The feature images thus obtained can be shown together in one composite image.

  By confirming the composite image, the user can grasp the movement status of the monitoring target without reproducing all the video data, so that the monitoring operation for the monitoring target can be completed in a short time.

  In the present embodiment, the operation of generating the composite image after adding the event information image to the feature image has been described. However, the present invention is not limited to this operation. The display generation unit 16 may generate a composite image by combining a plurality of feature images, and then add an event information image to the feature image included in the composite image. Specifically, in this case, the data processing apparatus 1 includes a display generation unit 31 that generates a composite image as illustrated in FIG. The display generating unit 31 combines a plurality of feature images to generate a combined image, and an adding unit 25 that adds an event information image to the feature images included in the combined image combined by the combining unit 32. Is provided. That is, in the configuration example illustrated in FIG. 10, the data processing unit 14 does not include the adding unit 25.

  Next, a procedure until video data is selected, an event information image is added to the feature image, and a composite image is generated will be described.

  FIG. 11 shows a state in which a plurality of sample images of video data are displayed on the display unit 17. A sample image of video data is an image generated by reducing the size of one image constituting video data.

  The user operates the operation unit 11 based on the sample image displayed on the display unit 17 to select one or a plurality of video data. In the following description, it is assumed that the video data E1 is selected.

  As shown in FIG. 12, the video data E1 is composed of a plurality of images A11, A12, A13, and video data in which the monitoring targets B1, B2, B3 are conveyed on the belt conveyor X is captured. It is. Note that the monitoring targets B1, B2, and B3 are the same products that are conveyed on the belt conveyor X.

  FIG. 13 shows a feature image group E1 ′ composed of a plurality of feature images. As illustrated in FIG. 13, the extraction unit 22 extracts a feature image a11 that is an image of the portion where the monitoring target B1 is displayed from the image A11 based on the event information, and the monitoring target from the image A12 based on the event information. A feature image a12 that is an image of a portion where B2 is displayed is extracted, and a feature image a13 that is an image of a portion where the monitoring target B3 is displayed is extracted from the image A13 based on the event information.

  FIG. 14 shows a feature image group E1 ′ composed of a plurality of feature images to which event information images are added. As illustrated in FIG. 14, the adding unit 25 adds the event information image C1 to the monitoring target B1 of the feature image a11, adds the event information image C2 to the monitoring target B2 of the feature image a12, and monitors the feature image a13. An event information image C3 is added to B3.

  The display generation unit 16 generates a composite image D by combining the feature image a11, the feature image a12, the feature image a13, and the image on which the monitoring target background is displayed. As shown in FIG. 15, the composite image D includes a monitoring target B1 to which the event information image C1 is added, a monitoring target B2 to which the event information image C2 is added, and a monitoring target B3 to which the event information image C3 is added. And are included.

  Therefore, the data processing apparatus 1 selects the video data, and generates a composite image by combining the feature image in which the event information image is added to the monitoring target from the image constituting the selected video data. Can be shown together in one composite image.

  By confirming the composite image, the user can grasp the movement status of the monitoring target without reproducing all the video data, so that the monitoring operation for the monitoring target can be completed in a short time.

  Next, a procedure until a composite image is generated after a plurality of video data is selected will be described. In the following, although the imaging angle of the imaging device 3 when imaging the monitoring target is the same, the video data E2 and the video data E3 each imaging the monitoring target conveyed on different belt conveyors are selected. A case will be described. As shown in FIG. 16, the video data E2 is composed of a plurality of images A11, A12, and A13, and an image is taken of how the monitoring targets B11, B12, and B13 are conveyed from the belt conveyor X1 to the belt conveyor X2. Video data. Note that the monitoring targets B11, B12, and B13 are the same products that are transported on the belt conveyors X1 and X2.

  As shown in FIG. 17, the video data E3 is composed of a plurality of images A21, A22, A23, and an image is taken of how the monitoring targets B21, B22, B23 are conveyed from the belt conveyor X3 to the belt conveyor X2. Video data. Note that the monitoring targets B21, B22, and B23 are the same products that are transported on the belt conveyors X2 and X3.

  FIG. 18 shows a feature image group E2 ′ composed of a plurality of feature images. As illustrated in FIG. 18, the extraction unit 22 extracts a feature image a11 that is an image of a portion where the monitoring target B11 is displayed from the image A11 based on the event information, and extracts the monitoring target from the image A12 based on the event information. A feature image a12 that is an image of the portion where B12 is displayed is extracted, and a feature image a13 that is an image of the portion where the monitoring target B13 is displayed is extracted from the image A13 based on the event information.

  FIG. 19 shows a feature image group E3 ′ composed of a plurality of feature images. Further, as illustrated in FIG. 19, the extraction unit 22 extracts a feature image a21 that is an image of the portion where the monitoring target B21 is displayed from the image A21 based on the event information, and from the image A22 based on the event information. A feature image a22 which is an image of a part where the monitoring target B22 is displayed is extracted, and a feature image a23 which is an image of a part where the monitoring target B23 is displayed is extracted from the image A23 based on the event information.

  FIG. 20 shows a feature image group E2 ′ composed of a plurality of feature images to which event information images are added. As illustrated in FIG. 20, the adding unit 25 adds the event information image C11 to the monitoring target B11 of the feature image a11, adds the event information image C12 to the monitoring target B12 of the feature image a12, and monitors the feature image a13. An event information image C13 is added to B13.

  FIG. 21 shows a feature image group E3 ′ composed of a plurality of feature images to which event information images are added. As illustrated in FIG. 21, the adding unit 25 adds the event information image C21 to the monitoring target B21 of the feature image a21, adds the event information image C22 to the monitoring target B22 of the feature image a22, and monitors the feature image a23. An event information image C23 is added to B23.

  The display generation unit 16 synthesizes the feature image a11, the feature image a12, the feature image a13, the feature image a21, the feature image a22, the feature image a23, and the image on which the monitoring target background is displayed. Thus, the composite image D is generated. In the composite image D, as shown in FIG. 22, the monitoring target B11 to which the event information image C11 is added, the monitoring target B12 to which the event information image C12 is added, and the monitoring target B13 to which the event information image C13 is added. The monitoring target B21 to which the event information image C21 is added, the monitoring target B22 to which the event information image C22 is added, and the monitoring target B23 to which the event information image C23 is added are included.

  In the above description, the procedure for generating one composite image D from the two video data E2 and E3 has been described. However, one composite image may be generated from three or more video data.

  Therefore, the data processing apparatus 1 selects a plurality of video data, and generates a composite image by synthesizing the feature image to which the event information image is added from the images constituting the selected plurality of video data. The feature images of the video data can be shown together in one composite image.

  By checking the composite image, the user can grasp the movement status of the monitoring target without reproducing each of the plurality of video data, so that the monitoring target can be confirmed in a short time.

  In addition, since the data processing apparatus 1 synthesizes composite images generated from a plurality of video data, there is no need to display the plurality of video data in parallel, and the display unit 17 can be reduced in size. Can be scaled down.

  In addition, since the data processing apparatus 1 generates a composite image by adding event information to the feature image extracted from the video data, it is possible to easily grasp the difference and situation for each video data.

  Moreover, since the event information is displayed on the feature image, the data processing apparatus 1 can easily grasp the situation. In the example shown in FIG. 22, all of the event information images are added to the feature image, but the present invention is not limited to this. For example, when part or all of the event information image is hidden on the display screen, and the cursor that moves on the display screen according to the operation of the pointing device such as a mouse is superimposed on the feature image, the event information image May be displayed in a pop-up form.

  FIG. 23 is a diagram illustrating a hardware configuration example of the data processing apparatus 1. The data processing apparatus 1 is a computer and includes a communication circuit 101, a processor 102, a memory 103, a display unit 104, and an input unit 105. The external device connection unit 12 and the imaging device connection unit 13 illustrated in FIG. 1 are realized by the communication circuit 101.

  The data processing unit 14 and the display generation unit 16 illustrated in FIG. 1 are realized by a program stored in the memory 103 being executed by the processor 102. The storage unit 15 illustrated in FIG. 1 is realized by the memory 103.

  The processor 102 is, for example, a CPU or a microprocessor, and is a processing circuit. The memory 103 is also used as a storage area when a program is executed by the processor 102.

  The operation unit 11 illustrated in FIG. 1 is realized by the input unit 105. The display unit 17 illustrated in FIG. 1 is realized by the display unit 104. The input unit 105 is a keyboard, a mouse, or the like. The display unit 104 is a display, a monitor, or the like. The display unit 104 and the input unit 105 may be realized by a touch panel in which these are integrated.

  The configuration described in the above embodiment shows an example of the contents of the present invention, and can be combined with another known technique, and can be combined with other configurations without departing from the gist of the present invention. It is also possible to omit or change the part.

  DESCRIPTION OF SYMBOLS 1 Data processing device, 2 External device, 3 Imaging device, 11 Operation part, 12 External device connection part, 13 Imaging device connection part, 14 Data processing part, 15 Storage part, 16, 31 Display production | generation part, 17 Display part, 21 Acquisition unit, 22 extraction unit, 23 processing unit, 24 image generation unit, 25 addition unit.

Claims (9)

A data processing unit for extracting a plurality of feature images in which a monitoring target is displayed from images constituting video data;
A display generation unit configured to generate a composite image by combining a plurality of the feature images extracted from the image constituting the video data by the data processing unit ;
The data processing unit
An acquisition unit for acquiring event information indicating a state of the monitoring target at a time when an image constituting the video data is captured;
The event information acquired by the acquisition unit includes alarm information generated in response to detection of abnormality or failure of the device controlled by the external device, and the video data is based on the alarm information. select the image constituting the data processing apparatus according to claim Rukoto a extracting unit that extracts the feature image from the selected image. An operation unit that accepts selection of the event information acquired by the acquisition unit;
The extraction section, based on the event information the operation unit accepts, select the images forming the image data, the selected image according to claim 1, characterized in that extracting the feature image Data processing device. The acquisition unit acquires processing information for processing the feature image,
Wherein the data processing unit, on the basis of the processing information, the data processing apparatus according to claim 1 or 2, characterized in that it comprises a processing unit for processing the feature image. The data processing unit
An image generation unit that converts the event information acquired by the acquisition unit into an image to generate an event information image;
The data processing apparatus according to claim 1 , 2 or 3 , further comprising an adding unit that adds the event information image to the feature image. The data processing unit
An acquisition unit for acquiring event information indicating a state of the monitoring target at a time when an image constituting the video data is captured;
An image generation unit that converts the event information acquired by the acquisition unit into an image to generate an event information image;
The data processing apparatus according to claim 1, further comprising: an adding unit that adds the event information image to the feature image. The data processing unit
An acquisition unit for acquiring event information indicating a state of the monitoring target at a time when an image constituting the video data is captured;
An image generation unit that converts the event information acquired by the acquisition unit into an image and generates an event information image;
The data processing apparatus according to claim 1, wherein the display generation unit includes an addition unit that adds the event information image to the feature image included in the composite image. The acquisition unit acquires instruction information indicating a method for adding the event information image to the feature image;
The additional part, based on the instruction information, the data processing device according to the event information image to claim 4, 5 or 6, characterized in that to add to the feature images. Programmable display device, characterized in that it comprises a data processing device according to any one of claims 1 to 7. A data processing step of extracting a plurality of feature images in which a monitoring target is displayed from images constituting video data;
A display generation step of generating a composite image by combining a plurality of the feature images extracted from the image constituting the video data by the data processing step ;
The data processing step includes
An acquisition step of acquiring event information indicating a state of the monitoring target at a time when an image constituting the video data is captured;
The event information acquired by the acquisition step includes alarm information generated in response to detection of abnormality or failure of the device controlled by the external device, and the video data is based on the alarm information. select the image constituting the data processing method comprising Rukoto a extraction step of extracting the feature image from the selected image.

Images