JP6944201B2 - Electronic circuits, hardware components, edge processing systems, edge computing systems, identification methods, identification programs - Google Patents

Description

The present disclosure relates to electronic circuits and the like.

In recent years, research and development of technologies related to the "Internet of Things" (IoT) have been promoted. For example, Patent Document 1 below discloses an edge computing system that appropriately controls data communication between an edge device and an edge server to achieve both real-time performance and storage of all data.

Japanese Unexamined Patent Publication No. 2018-195175 (published on December 6, 2018)

In the above-mentioned conventional technique, when the discriminative model is relearned, it is necessary to relearn the model on the cloud server and then re-incorporate the relearned model into the terminal installed in the operating environment. However, there is a problem that the work of transmitting and receiving data between the operating environment and the cloud server and re-embedding the relearned model in the terminal requires a large amount of man-hours.

One aspect of the present disclosure is, for example, to provide an electronic circuit or the like that enables model re-learning in a stand-alone manner.

The electronic circuit according to one aspect of the present disclosure includes an acquisition unit that acquires time-series data recorded at a fixed point as input from a data recording device, and a reference that appears at an invariant position in a plurality of data constituting the time-series data. The estimation unit that estimates the pattern using the trained model stored in the storage unit and the identification pattern that appears at different positions in a plurality of data are identified based on the estimated reference pattern, and the identification result is obtained. An output identification unit and a re-learning unit that retrains a trained model using at least a part of the acquired time series data in order to re-estimate the reference pattern so as to follow changes in the identification environment. I have.

The identification method according to one aspect of the present disclosure includes an acquisition process of acquiring time-series data recorded at a fixed point as input from a data recording device, and a reference that appears at an invariant position in a plurality of data constituting the time-series data. Based on the estimation process that estimates the pattern using the trained model stored in the storage unit and the estimated reference pattern, the identification patterns that appear at different positions in multiple data are identified, and the identified results are obtained. The output identification process and the re-learning process of retraining the trained model using at least part of the acquired time series data to re-estimate the reference pattern to follow changes in the identification environment. Including, the electronic circuit executes.

The identification program according to one aspect of the present disclosure has an acquisition function for acquiring time-series data recorded at a fixed point as input from a data recording device, and appears at invariant positions in a plurality of data constituting the time-series data. Based on the estimation function that estimates the reference pattern using the trained model stored in the storage unit and the estimated reference pattern, the identification patterns that appear at different positions in the plurality of data are identified and the said. The trained model is re-learned using at least a portion of the acquired time-series data in order to re-estimate the reference pattern to follow changes in the identification environment with an identification function that outputs the identification results. Realize the re-learning function to learn on the computer.

It is a schematic diagram which shows the configuration example of the edge computing system which concerns on Embodiment 1. FIG. It is a block diagram which shows an example of the main part structure of the electronic circuit and the camera which concerns on 1st Embodiment. It is a flowchart which shows an example of the process executed by the said electronic circuit. It is a schematic diagram which shows the configuration example of the edge computing system which concerns on 2nd Embodiment. It is a block diagram which shows an example of the main part structure of the edge terminal and the camera which concerns on 2nd Embodiment.

[Embodiment 1]
The first embodiment will be described with reference to FIGS. 1 to 3.

(Outline of electronic circuit)
An outline of the electronic circuit 1a according to the first embodiment will be described with reference to FIG. In the first embodiment, the operation and the like are shown by taking as an example the case where the electronic circuit 1a is applied to the task of identifying a defective product from a moving image (time series image) of the product taken in the manufacturing process of the product. explain.

FIG. 1 is a schematic diagram showing a configuration example of an edge computing system 4a. As illustrated in FIG. 1, the edge computing system 4a is, for example, a system including a camera 2 that captures a moving image of a monitored object at a fixed point and a production line 3a that is the monitored object.

The camera 2 includes an electronic circuit 1a. Here, the electronic circuit 1a is hardware designed to identify whether or not the product 5a manufactured on the production line 3a is a defective product from a moving image. The electronic circuit 1a sequentially acquires time-series images constituting the moving image as input, and outputs an identification result by processing these with a plurality of functional modules mounted on the electronic circuit 1a.

Since the product 5a to be identified moves within the monitoring area (the range in which the camera 2 can shoot a moving image), the position where the product 5a appears in each image is different. On the other hand, since the background to be non-identified is stationary, the position where the background appears in each image is substantially the same.

However, if there is a difference in the properties (for example, brightness, color, etc.) of each image that captures the background, the difference may cause deterioration of the identification accuracy. For example, if there is a difference between the properties of the learning image used for learning in the development environment and the properties of the identification image actually obtained in the identification environment due to different video shooting conditions, the accuracy of the identification deteriorates. do. In addition, if the identification environment changes during video recording (for example, changes in brightness / shadow due to lighting, vibration generated in the camera that shoots the video, etc.), the properties of each image that composes the video will differ. Therefore, the accuracy of identification also deteriorates.

Therefore, the electronic circuit 1a needs to calibrate each image so as to follow the change in the identification environment and cancel the difference. That is, the model must be retrained using the images actually obtained in the identification environment.

Therefore, the electronic circuit 1a first considers a stationary region (appearing at an invariant position) in each image as a background to be a non-identification target, and estimates the region using a trained model. Next, the electronic circuit 1a regards the region with movement (appearing at a different position) as the product 5a to be identified based on the estimated region, and uses a method different from the method using the trained model. Identify product 5a (eg, estimate using a model different from the model used to estimate the background).

That is, the electronic circuit 1a is necessary for the learning / estimation by separating the background (reference pattern) and the product (identification pattern) 5a and limiting the target of learning / estimation using the model to the static background. Narrow down the input information that becomes. As a result, the calculation load required for the learning / estimation can be reduced. Therefore, the electronic circuit 1a can both estimate the background and relearn the model.

In other words, the electronic circuit 1a can relearn the model only by the electronic circuit 1a (standalone) in the identification environment even if its computational resources are poor. Therefore, the performance of the model can be kept high even in the actual environment without spending the work man-hours required for re-learning.

Further, since the calculation load is small, the electronic circuit 1a can operate with low power. Therefore, the electronic circuit 1a can be easily incorporated not only in the camera 2 but also in various devices such as routers, drones, and industrial devices.

The configuration of the edge computing system 4a is not limited to the configuration illustrated in FIG. The edge computing system 4a may further include, for example, a computer that receives the identification result from the camera 2.

(Electronic circuit and camera configuration)
An example of the configuration of the electronic circuit 1a and the camera 2 will be described with reference to FIG. FIG. 2 is a block diagram showing an example of the main configuration of the electronic circuit 1a and the camera 2.

First, the configuration of the electronic circuit 1a will be described. The electronic circuit 1a is a circuit that identifies a product (pattern) 5a from a moving image (time series data) 6 recorded at a fixed point. The electronic circuit 1a can be realized, for example, as a hardware component having a predetermined interface. Alternatively, it can be realized by a circuit such as a programmable logic device (PLD). In this case, for example, a field-programmable gate array (FPGA) may be used as the PLD. The specific configuration for realizing the electronic circuit 1a is not limited to these examples.

As illustrated in FIG. 2, the electronic circuit 1a can include, for example, an acquisition unit 11, an estimation unit 12, an identification unit 13, a determination unit 14, a relearning unit 15, and a storage unit 16. Further, the electronic circuit 1a may further include, for example, a comparison unit, a specific unit, and the like.

The acquisition unit 11 acquires the moving image 6 as an input from the camera 2. For example, the acquisition unit 11 can acquire a plurality of images (data) 7 constituting the moving image 6 in real time, and can sequentially output the acquired images 7a to the estimation unit 12.

Further, for example, the acquisition unit 11 relearns an image (re-learning) used for re-learning the trained model 9 from an image acquired from the time when the previous background was estimated to the time when the next background is estimated. Data) 7b can be sampled. In this case, the acquisition unit 11 outputs the re-learning image 7b to the re-learning unit 15. Further, the acquisition unit 11 can change, for example, the amount of information contained in each image according to the estimation of the background and the identification of the product 5a.

The estimation unit 12 estimates the backgrounds appearing at invariant positions in the plurality of images 7a constituting the moving image 6 by using the trained model 9 stored in the storage unit 16. The estimation unit 12 outputs the estimated result as an estimation result 8a to the identification unit 13.

Further, for example, when the trained model 9 is relearned, the estimation unit 12 can re-estimate the background using the retrained trained model 9. Further, the estimation unit 12 can set the background by calibrating each image so as to cancel the change in the identification environment by using, for example, the trained model 9.

The identification unit 13 identifies the products 5a that appear at different positions in the plurality of images based on the estimated background, and outputs the identified results. The identification unit 13 can output the identified result as the identification result 8b to the determination unit 14 and the transmission unit 22.

Further, the identification unit 13 can, for example, re-identify the product 5a with reference to the re-estimated background, and output the re-identified result. Further, the identification unit 13 can track the product 5a moving in the monitoring area and identify the product 5a by using, for example, a prediction model that can predict the change of the time series stored in the storage unit 16.

The determination unit 14 determines the necessity of re-learning the background. The determination unit 14 outputs the determination result as the determination result 8c to the relearning unit 15.

The re-learning unit 15 re-learns the trained model 9 using at least a part of the acquired moving image in order to re-estimate the background so as to follow the change in the identification environment. The re-learning unit 15 may relearn the trained model 9 only when the determination unit 14 determines that re-learning is necessary.

The storage unit 16 is a storage module composed of an arbitrary recording medium, and stores the learned model 9. Note that FIG. 2 illustrates a configuration in which the electronic circuit 1a includes the storage unit 16. However, the storage unit 16 may be an external storage device communicatively connected to the electronic circuit 1a.

The comparison unit can compare the first background estimated at the first time with the second background estimated at the second time. In this case, the determination unit 14 can determine the necessity of re-learning by, for example, confirming whether or not the cumulative difference obtained by the comparison satisfies a predetermined condition.

The identification unit can identify the difference between the third background estimated at the third time and the image acquired at the fourth time. In this case, the identification unit 13 executes identification of the product 5a based on the third background, for example, with respect to the identified difference. Then, the identification unit 13 outputs the identification result as the identification result 8b to the determination unit 14.

Next, the configuration of the camera (data recording device, edge terminal) 2 will be described. The camera 2 is a device that captures a moving image of a production line (monitoring target) 3a as a data environment that is a source of an image from a fixed point. The camera 2 includes an electronic circuit 1a, an input interface 21, a recording unit 21, a transmitting unit 22, and a communication interface 23.

The input interface 21 is a contact point for the camera 2 to take in information on the identification environment in order to shoot a moving image of the identification environment. Input interface 21 may, for example, can be constituted by a circuit arbitrary.

The recording unit 21 records the moving image 6 and gives the moving image 6 to the electronic circuit 1a as an input. As a result, the acquisition unit 11 can acquire the moving image 6 as an input from the camera 2.

The transmission unit 22 outputs the identification result 8b input from the identification unit 13 to the communication interface 23.

The communication interface 23 communicates with the outside via a communication network according to a predetermined communication method, and transmits the identification result 8b input from the transmission unit 22 to the external computer. The communication interface 23 only needs to have an essential function of enabling communication with an external device, and the communication line, communication method, communication medium, and the like are not limited.

(Model learning method)
An example of a method in which the electronic circuit 1a (re-learning unit 15) learns a model will be described.

The type of the trained model 9 is not limited as long as it is a model whose background can be estimated. Here, the learning model 9 is preferable as long as it is a model with a low calculation cost required for learning / estimation. This is because, as described above, the model can be easily retrained.

In particular, the position where the background is reflected in each image is substantially the same, and the change in the image in which the background is reflected (difference in the properties of each image in which the background is reflected, which occurs due to the change in the identification environment) can be expressed. Given its importance, a model that can express the background with a small number of explanatory variables and parameters by feature extraction focusing on the change is preferable.

For example, it is preferable to model the background using sparse modeling (a learning model utilizing sparseness) and train the model. The electronic circuit 1a may use, for example, non-negative matrix factorization (NMF) as a learning method of the model. According to the learning method, information can be extracted by utilizing the sparsity hidden in the event, so that the electronic circuit 1a can extract one of the important aspects of the event (true information to be acquired) from a small amount of data. .. In this way, by adopting a model with a low calculation cost required for learning / estimation, the electronic circuit 1a can relearn the model in real time.

Further, the electronic circuit 1a may partially relearn the trained model 9 by updating only the parameters that require relearning among the parameters included in the trained model 9, for example.

For example, when non-negative matrix factorization (NMF) is used, the electronic circuit 1a is a parameter included in the background dictionary that corresponds to a changed part in the image (that is, the product 5a as an identification pattern). Only can be updated.

Further, the electronic circuit 1a may dynamically change the frequency of re-learning the trained model 9 according to, for example, the speed at which the product 5a changes its position. Specifically, the electronic circuit 1a measures the speed at which the re-learning image 7b changes by mode-decomposing the re-learning image 7b in the time direction and extracting a frequency component. As a result, the speed at which the background changes and the speed at which the product 5a changes can be obtained independently.

Then, the electronic circuit 1a can dynamically change the frequency of re-learning according to the speed at which the product 5a changes. Thereby, when it is necessary to identify an identification target having a relatively small movement, for example, when the product 5a slowly moves in the monitoring area, the identification target can be accurately identified.

The specific learning method described above is merely an example, and is not limited to this. The electronic circuit 1a can use various learning methods depending on the applied task.

(Method of determining the necessity of re-learning)
An example of a method in which the electronic circuit 1a (determination unit 14) determines the necessity of re-learning will be described.

The electronic circuit 1a determines that the background needs to be relearned, for example, when at least one of the following two conditions is satisfied. The conditions are not limited to the following two. Additions and changes can be made as appropriate according to the situation in which the electronic circuit 1a is applied.
(1) When the accumulation of background subtraction exceeds a predetermined threshold before the lapse of a predetermined time (2) When the object contained in the background changes

The above (1) corresponds to a case where the identification environment changes relatively slowly, for example, when the sunlight irradiating the background changes with the passage of time.

The electronic circuit 1a extracts a region (background region) presumed to be the background in each image from each image. Next, the difference (background difference) between the first background area at a certain time and the second background area at the next time is calculated, and it is determined whether or not the background difference exceeds a predetermined threshold value. ..

If it exceeds, the electronic circuit 1a determines that re-learning is necessary. If it does not exceed, the electronic circuit 1a calculates the background subtraction between the second background region and the third background region at the next time, and sums the previous background subtraction and the background subtraction calculated this time ( Cumulative) is calculated. Then, it is determined whether or not the accumulation exceeds the predetermined threshold value.

If it exceeds the limit, the electronic circuit 1a (determination unit 14) determines that re-learning is necessary. If it does not exceed, the accumulation is calculated in the same manner, and it is determined whether or not the accumulation exceeds the predetermined threshold value.

The electronic circuit 1a can determine that re-learning is necessary after the lapse of the predetermined time, even if the cumulative total does not exceed the predetermined threshold value. Further, a predetermined time and a predetermined threshold value can be appropriately set according to the task of applying the electronic circuit 1a.

In (2) above, for example, the equipment included in the production line 3 is changed by a human being, or the bicycle parked on the bicycle falls down due to the wind, and the object estimated as a part of the background is changed. Corresponds to the case.

This corresponds to the case where the cumulative total suddenly exceeds the predetermined threshold value before the predetermined time elapses in the case of the above (1). That is, it means that the currently estimated background may have changed to a completely different background. Therefore, the electronic circuit 1a determines that re-learning is necessary.

The specific determination method described above is merely an example, and is not limited to this. The electronic circuit 1a can use various determination methods depending on the application task.

(Image sampling method for re-learning)
An example of a method in which the electronic circuit 1a (acquisition unit 11) samples the re-learning image 7b will be described.

When the electronic circuit 1a determines that re-learning is necessary, the electronic circuit 1a can re-learn the trained model 9 by using all or a part of the images acquired from the time when the background was learned last time to the present.

When a part is used, for example, the electronic circuit 1a can randomly sample a predetermined number of images from the plurality of images and use the sampled images as the re-learning image 7b for re-learning. As a result, the electronic circuit 1a can reduce the computational load of relearning and improve the real-time nature of processing. The number of images to be sampled can be appropriately set according to the task of applying the electronic circuit 1a.

As described above, when the re-learning image 7b is prepared by sampling, the electronic circuit 1a weights the currently estimated background with, for example, the number of sampled images, and is sampled with the weighted background. The trained model 9 can be retrained based on the image. As a result, the trained model 9 can be retrained so as to reflect minute changes in the background from only the currently estimated background and the newly acquired image.

Alternatively, when the re-learning image 7b is used by sampling, the electronic circuit 1a can relearn the trained model 9 by using, for example, only the region of the sampled image in which the background difference exceeds the threshold value. As a result, the electronic circuit 1a can reduce the computational load of relearning and improve the real-time nature of processing.

Here, it is assumed that the electronic circuit 1a retrains the trained model 9 using a plurality of images acquired in a short time. Under this assumption, for example, assuming that the production line 3 is temporarily stopped, the product 5a is stopped in the monitoring area, so that the product 5a, which is originally an identification target, is also learned as a background. There is a possibility.

Therefore, if it is desired to identify the stopped product 5a as well, it is preferable that the electronic circuit 1a relearns the trained model 9 using a plurality of images acquired within a sufficiently long time. However, since there are many images included in a long time, if the electronic circuit 1a relearns the model using all the images, the calculation load becomes high and the real-time property of the processing may be impaired.

In order to overcome this problem, re-learning by the above sampling is also effective. As a result, the electronic circuit 1a can improve the real-time property while maintaining the estimated quality of the background.

The specific sampling method described above is merely an example, and is not limited to this. The electronic circuit 1a can use various sampling methods depending on the application task.

As described above, the electronic circuit 1a may, for example, sample the training image 7b, partially relearn the trained model 9, or dynamically change the frequency of relearning. The computational load of re-learning can be reduced and the real-time processing can be improved. As a result, the electronic circuit 1a can relearn the background of the image standalone in the identification environment even if its computational resources are poor.

(Background estimation method)
An example of a method in which the electronic circuit 1a (estimating unit 12) estimates the background will be described.

The electronic circuit 1a estimates the background to appear in the image using the trained model 9. For example, when non-negative matrix factorization (NMF) is used, the electronic circuit 1a can derive the background of the image as an estimation result from the learned basis functions (background dictionary). Further, the electronic circuit 1a can derive, for example, an average value for each pixel of a plurality of images as a background estimation result.

The specific estimation method described above is merely an example, and is not limited to this. The electronic circuit 1a can use various estimation methods depending on the application task.

(Identification method of identification pattern)
An example of a method in which the electronic circuit 1a (identification unit 13) identifies the product 5a will be described.

The electronic circuit 1a calculates, for example, the squared error between the currently estimated background and the most recently acquired image. Then, a region composed of pixels having an error larger than a predetermined threshold value is specified, and if the area of the region is larger than the predetermined area, it is determined that there is a high possibility that the product 5a to be identified exists in the region. do. The predetermined threshold value and the predetermined area can be appropriately set according to the task of applying the electronic circuit 1a.

Next, the electronic circuit 1a identifies the product 5a in the above region using, for example, an identification model. At this time, the identification model may be a model different from the trained model 9 for estimating the background, and may be trained in advance so that the product 5a can be identified. Further, since the electronic circuit 1a does not have to relearn the identification model, the identification model may be a more complicated model than the trained model 9.

When the identification target cannot be predicted in advance (that is, when it is desired to identify an arbitrary object moving in the monitoring area), as described above, a model capable of identifying the identification target cannot be constructed. Therefore, for example, the electronic circuit 1a sets the above-mentioned region as it is as an identification target.

In general, even if only the product 5a is identified, the calculation load of the identification process for the entire image is high, so that the real-time property may be impaired. Therefore, by limiting the identification area to the specified area, the electronic circuit 1a can improve the real-time property while maintaining the identification accuracy.

The specific identification method described above is merely an example, and is not limited to this. The electronic circuit 1a can use various identification methods depending on the application task.

(Tracking of identification pattern)
An example of a method in which the electronic circuit 1a (identification unit 13) tracks the product 5a will be described.

Once the product 5a is identified, the electronic circuit 1a can track the product 5a using a predictive model that can predict changes over time. The prediction model may be, for example, a Kalman filter, a particle filter, a hidden Markov model, or the like, which sequentially Bayesian estimates the state transitions of hidden variables. Alternatively, it may be a model that estimates the output for the input as a prediction, such as generalized linear regression or neural network.

When the electronic circuit 1a identifies the product 5a at a certain time, the position of the product 5a in the image obtained at the next time can be estimated using the prediction model. The electronic circuit 1a can obtain a position where the product 5a has moved by, for example, storing an image of a region including the product 5a at the time of identification and performing pattern matching on the vicinity of the predicted position.

For example, when the product 5a cannot be detected by the pattern matching, the electronic circuit 1a may use the estimated amount of the prediction model as it is and consider it as the moving position of the product 5a. Then, the electronic circuit 1a can determine that the product 5a is out of the monitoring area when the undetectable state continues for a predetermined time.

In this way, after identifying the product 5a, by tracking the product 5a using a prediction model, the amount of calculation required for identification (for example, even when the identification accuracy is lowered due to an external factor). The accuracy of the identification can be maintained while reducing the number of items and increasing the real-time property.

The specific tracking method described above is merely an example, and is not limited to this. The electronic circuit 1a can use various tracking methods depending on the applied task.

(How to change the amount of information)
An example of a method in which the electronic circuit 1a (acquisition unit 11) changes the amount of information in the image will be described.

In general, the identification accuracy is higher when a high-resolution image is used than when a low-resolution image is used. However, the higher the resolution, the heavier the calculation load, which may impair the real-time property.

Therefore, the electronic circuit 1a holds, for example, a high-resolution image and a low-resolution image. Alternatively, the electronic circuit 1a may acquire the image in high resolution and reduce the resolution of the acquired high resolution image (that is, the amount of information in the image can be changed). At this time, the electronic circuit 1a can estimate the background using the low-resolution image. Then, the electronic circuit 1a can identify the product 5a by referring to the same region of the high-resolution image with respect to the region where the product 5a is determined to exist by calculating the background subtraction, for example.

In this way, by roughly identifying the location of the identification target using the low-resolution image and executing the identification process only from the corresponding area of the high-resolution image, high-precision identification can be performed without impairing the real-time property. realizable. The above method of changing the image resolution (information amount) is particularly suitable for a task such as abnormality detection of the product 5a described in the present embodiment, in which a minute abnormality needs to be detected in a short time. It is valid.

The specific change method described above and the estimation / identification method using the change method are merely examples, and are not limited thereto. The electronic circuit 1a can use various methods depending on the application task.

(Process executed by electronic circuit 1a)
FIG. 3 is a flowchart showing an example of processing executed by the electronic circuit 1a. In the following description, the “-step” in parentheses represents each step included in the identification method.

First, the acquisition unit 11 acquires the moving image 6 from the camera 2 as an input (step 1, hereinafter, "step" is abbreviated as "S". Acquisition step). Next, the estimation unit 12 estimates the backgrounds appearing at invariant positions in the plurality of images constituting the moving image 6 using the trained model 9 stored in the storage unit 16 (S2, estimation step).

Then, the identification unit 13 identifies the products 5a appearing at different positions in the plurality of images based on the estimated background, and outputs the identified result as the identification result 8b (S3, identification step). Finally, the trained model 9 is retrained using at least a part of the acquired moving image in order to re-estimate the background so as to follow the change of the discrimination environment (S4, re-learning step).

The electronic circuit 1a determines whether or not the entire processing is completed (S5), and if it is determined that the entire processing is completed (YES in S5), the electronic circuit 1a ends the entire processing. If it is determined that the process has not been completed (NO in S5), the process of the acquisition unit 11 is repeated again.

The identification method may optionally include not only the above-mentioned processing described above with reference to FIG. 3 but also processing executed in each part included in the electronic circuit 1a. For example, the identification method can appropriately include a process executed by a determination unit, a comparison unit, and a specific unit.

(Effects of electronic circuits)
The electronic circuit 1a has an effect that the background of the image can be relearned standalone in the identification environment even if the computational resources are poor. Therefore, the effect that the performance of the model can be kept high even in the actual environment can be further achieved without spending the work man-hours required for re-learning. Further, since the calculation load is small, the electronic circuit 1a can be operated with low power, which is an additional effect. It can be presumed that all the methods that can relearn the background so as to follow the change of the identification environment in the edge terminal with poor computational resources are included in the scope of the present disclosure.

[Embodiment 2]
The second embodiment will be described with reference to FIGS. 4 to 5. In the second embodiment, only the configuration added to the above-described first embodiment and the configuration different from the configuration of the first embodiment will be described. That is, all the configurations described in the first embodiment can be included in the second embodiment (and vice versa). Further, the definitions of the terms described in the first embodiment are the same in the second embodiment.

In the second embodiment, the usage rate of the parking lot (the ratio of the number of identified cars to the number of cars that can be parked) is monitored by identifying the car from the moving image of the parking lot. The operation of an electronic circuit or the like will be described by taking a case as an example.

(Overview of edge terminals)
An outline of the edge terminal 1b according to one aspect of the present disclosure will be described with reference to FIG. FIG. 4 is a schematic diagram showing a configuration example of the edge computing system 4b. As illustrated in FIG. 4, the edge computing system 4b is a system including, for example, an edge terminal 1b, a camera 2 that shoots a moving image of a monitoring target at a fixed point, and a parking lot 3b that is the monitoring target. The edge terminal 1b and the camera 2 are communicably connected, and a moving image taken by the camera 2 is given as an input of the edge terminal 1b.

Similar to the electronic circuit 1a, the edge terminal 1b first considers a stationary region (parking lot) in each image as a background to be non-identified, and estimates the region using a trained model. Next, the edge terminal 1b regards the region with movement as the vehicle 5b to be identified with reference to the estimated region, and identifies the vehicle 5b by a method different from the method using the trained model.

That is, the edge terminal 1b is necessary for the learning / estimation by separating the background (reference pattern) and the automobile (identification pattern) 5b and limiting the target of learning / estimation using the model to the static background. Narrow down the input information that becomes. As a result, the calculation cost required for the learning / estimation can be reduced. Therefore, the edge terminal 1b can both estimate the background and relearn the model.

In other words, the edge terminal 1b can relearn the background of the image standalone in the identification environment even if its computational resources are poor. Therefore, the performance of the model can be kept high even in the actual environment without spending the work man-hours required for re-learning.

In other words, the edge terminal 1b can separate the truly changed image area (automobile) based on the background relearned on the terminal side, so that the adaptability and robustness to noise and environmental changes can be improved.

The configuration of the edge computing system 4b is not limited to the configuration illustrated in FIG. For example, the edge computing system 4b may further include a computer that receives the identification result from the camera 2.

(Configuration of edge terminal)
An example of the configuration of the edge terminal 1b will be described with reference to FIG. FIG. 5 is a block diagram showing an example of a main part configuration of the edge terminal 1b. The edge terminal 1b is an information processing device that identifies an automobile (identification pattern) 5b from a moving image 6 recorded at a fixed point. The edge terminal 1b may be any specific device as long as it can execute the identification program.

Here, the identification program is a program that causes the edge terminal 1b to perform the same function as that realized by the electronic circuit 1a. The identification program is, for example, in a multifunction device (for example, a computer) including a memory and one or a plurality of processors capable of executing one or a plurality of programs stored in the memory. , Which can be carried out particularly favorably. However, the specific configuration for realizing the edge terminal 1b is not limited to these examples.

The edge terminal 1b may include, for example, a control unit 10, a transmission unit 22, and a communication interface 23.

The control unit 10 comprehensively controls various functions of the edge terminal 1b. The control unit 10 includes an acquisition unit 11, an estimation unit 12, an identification unit 13, a determination unit 14, a re-learning unit 15, and a storage unit 16. Further, the control unit 10 can further include a comparison unit and a specific unit (the comparison unit and the specific unit are not shown). Since the functions of the respective parts included in the control unit 10 are the same as the functions of the respective parts included in the electronic circuit 1a, the description thereof will be omitted. Further, since the transmission unit 22 and the communication interface 23 provided in the edge terminal 1b have the same functions as the functions of the respective units provided in the camera 2, the description thereof will be omitted.

(Effects of edge terminals)
The edge terminal 1b can relearn the background of the image standalone in the identification environment even if its computational resources are poor. Therefore, there is an effect that the performance of the model can be kept high even in the actual environment without spending the work man-hours required for re-learning.

[Other Embodiments]
The present disclosure is also applicable to tasks other than the task of identifying defective products described in the first embodiment and the task of calculating the parking lot usage rate described in the second embodiment. For example, it can be widely applied to general identification tasks such as visual inspection of buildings, visitor statistics, and traffic monitoring.

For example, the electronic circuit 1a can be incorporated into a drone, cracks, floats, peeling, dirt, etc. can be detected from an image of a building taken aerial by the drone, and a portion requiring repair can be specified.

In addition, by implementing this disclosure, it is possible to monitor various targets such as large-scale commercial facilities, factory equipment, illegal intrusions, equipment malfunctions, shopping malls, retail stores, cafes, and events. This allows, for example, to automatically generate statistics on the number of customers at any time of the day, allowing the administrator to analyze and predict the number of visitors.

Furthermore, the present disclosure can also be implemented in the fields of voice recognition and signal processing. That is, the electronic circuit 1a and the edge terminal 1b, for example, acquire voice from a microphone (time-series data) as input, and are invariant in a plurality of phonemes (that is, data including gain, frequency, etc.) constituting the voice. The background sound (reference pattern) that appears at each position is estimated using the trained model stored in the storage unit, and the estimated background sound is used as a reference for different positions in multiple phonemes. At least a portion of the acquired speech in order to identify the particular speech (identification pattern) that appears, output the identified result, and re-estimate the background sound to follow changes in the identification environment. Can be used to retrain the trained model.

Similarly, the electronic circuit 1a and the edge terminal 1b acquire, for example, a control signal (time series data) from a control device as an input, and data including a plurality of signals (that is, torque, vibration, etc.) constituting the control signal. ), The standard signals (reference patterns) that appear at the invariant positions are estimated using the trained model stored in the storage unit, and the estimated standard signals are used as references at different positions in a plurality of signals. At least a portion of the acquired signal is used to identify the specific signal (identification pattern) that appears, output the identified result, and re-estimate the standard signal to follow changes in the identification environment. The trained model can be retrained.

[Example of realization by software]
The control block of the electronic circuit 1a and the edge terminal 1b may be realized by software using a controller such as a CPU (Central Processing Unit). That is, the electronic circuit 1a and the edge terminal 1b are a CPU that executes an instruction of an identification program that is software that realizes each function, and a ROM (Read) in which the identification program and various data are readablely recorded by a computer (or CPU). It is equipped with a Only Memory) or a storage device (these are referred to as "recording media"), a RAM (Random Access Memory) for developing the identification program, and the like. Then, when the computer (or CPU) reads the identification program from the recording medium and executes it, an example of the object according to one aspect of the present disclosure is achieved. As the recording medium, a "non-temporary tangible medium", for example, a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. Further, the identification program may be supplied to the computer via an arbitrary transmission medium (communication network, broadcast wave, etc.) capable of transmitting the identification program. One aspect of the present disclosure can also be realized in the form of a data signal embedded in a carrier wave, in which the identification program is embodied by electronic transmission.

Specifically, the identification program according to the embodiment of the present disclosure is an identification program that identifies a pattern from time-series data recorded at a fixed point, and is provided with a computer in each part of the electronic circuit 1a or the edge terminal 1b. It can be realized as an identification program to function as.

The identification program can be implemented in any programming language. For example, the identification program can be implemented using a script language such as ActionScript or JavaScript (registered trademark), an object-oriented programming language such as Objective-C or Java (registered trademark), or a markup language such as HTML5. Further, it is provided with an information processing terminal (for example, electronic circuit 1a, edge terminal 1b) having each part that realizes each function realized by the identification program, and each part that realizes the remaining functions different from each of the above functions. Edge processing systems and edge computing systems, including cameras and the like, also fall within the scope of this disclosure.

[Additional notes]
In the electronic circuit according to one aspect of the present disclosure, the estimation unit re-estimates the reference pattern using the re-learned trained model, and the identification unit uses the re-estimated reference pattern as a reference. As a result, the identification pattern can be re-identified and the re-identified result can be output.

The electronic circuit according to one aspect of the present disclosure further includes a determination unit that determines the necessity of re-learning of the reference pattern, and the re-learning unit determines that re-learning is necessary, the learned model. Can be relearned.

The electronic circuit according to one aspect of the present disclosure further includes a comparison unit that compares the first reference pattern estimated in the first time with the second reference pattern estimated in the second time. The determination unit can determine whether or not the re-learning is necessary by confirming whether or not the cumulative difference obtained by the comparison satisfies a predetermined condition.

The electronic circuit according to one aspect of the present disclosure further includes a specific unit that identifies a difference between the third reference pattern estimated in the third time and the data acquired in the fourth time, and the identification unit. Can execute the identification of the identification pattern based on the estimated third reference pattern for the identified difference and output the result of the identification.

In the electronic circuit according to one aspect of the present disclosure, the acquisition unit relearns the trained model from the data acquired from the time when the previous reference pattern was estimated to the time when the next reference pattern is estimated. The re-learning data used for can be sampled.

In the electronic circuit according to one aspect of the present disclosure, the identification unit can track the identification pattern and identify the identification pattern by using a prediction model that can predict changes in time series stored in the storage unit. ..

In the electronic circuit according to one aspect of the present disclosure, the estimation unit uses the trained model to calibrate the plurality of data so as to cancel the change in the identification environment, thereby performing the reference pattern. To set.

In the electronic circuit according to one aspect of the present disclosure, the re-learning unit partially re-learns the trained model by updating only the parameters that require re-learning among the parameters included in the trained model. learn.

In the electronic circuit according to one aspect of the present disclosure, the acquisition unit changes the amount of information contained in each data according to the estimation of the reference pattern and the identification of the identification pattern.

In the electronic circuit according to one aspect of the present disclosure, the relearning unit dynamically changes the frequency of relearning according to the speed at which the identification pattern changes its position.

In the electronic circuit according to one aspect of the present disclosure, the time series data is a moving image composed of a plurality of images, the identification pattern is an identification target accompanied by movement in the moving image, and the reference pattern is the said reference pattern. It is a static background in the video.

The hardware component according to one aspect of the present disclosure includes a predetermined interface and the electronic circuit. Further, the edge terminal according to one aspect of the present disclosure includes the hardware component.

The edge processing system according to one aspect of the present disclosure includes the edge terminal and a data recording device that records the time series data and inputs the time series data to the edge terminal connected communicably as an input. ..

The data recording device according to another aspect of the present disclosure includes a communication interface that is communicably connected to an external computer, and a transmission unit that transmits an identification result output from the electronic circuit to the computer via the communication interface. Is further equipped.

The electronic circuit according to another aspect of the present disclosure can also be expressed as follows. That is, an electronic circuit that identifies a pattern from data using a model, and an acquisition unit that sequentially acquires data including an identification pattern that changes relative to a reference pattern with the passage of time in chronological order, and the reference. By separating the estimation unit that estimates the reference pattern, and the reference pattern and the identification pattern using the learned model stored in the storage unit so that the identification pattern can be easily separated from the pattern. An electronic circuit including an identification unit that identifies the identification pattern, and the trained model is a model that can dynamically estimate the reference pattern so that the identification pattern is highlighted. Can be expressed.

Further, the electronic circuit according to another aspect of the present disclosure can also be expressed as follows. That is, it is an electronic circuit that identifies a pattern from data using a model, and estimates a reference pattern using an acquisition unit that acquires time-series data from a data recording device and a learned model stored in the storage unit. The learning model includes an estimation unit and an identification unit that identifies the identification pattern by separating the reference pattern and the identification pattern, and the trained model follows the change in the identification environment with the passage of time. Is a model that can be relearned using the time series data that reflects the change so that can be dynamically estimated, and the relearning enhances the accuracy of the dynamic estimation of the reference pattern and the reference. It can also be expressed as an electronic circuit characterized by learning that improves the performance of identifying the identification pattern from the pattern.

Further, the electronic circuit according to another aspect of the present disclosure can also be expressed as follows. That is, an electronic circuit that identifies an object from a moving image composed of a plurality of images, and a storage unit that stores a learned model that estimates a static (not the object of the identification) background in the moving image, and the storage unit. An acquisition unit that sequentially acquires images constituting a moving image, an estimation unit that estimates the background using the stored learned model, and a motion in the moving image based on the estimated background. Re-learning the trained model using an identification unit that outputs the identification result of the accompanying object and at least one image obtained by the progress in order to re-estimate the background according to the progress of the moving image. It can also be expressed as an electronic circuit characterized by having a learning unit.

Further, the electronic circuit according to another aspect of the present disclosure can also be expressed as follows. That is, it is a digital circuit including a memory and a plurality of modules, in which the memory stores a trained model for estimating a reference pattern, and the digital circuit acquires time-series data as input from a data recording device. An estimation module that estimates an acquisition module, a reference pattern that appears at an invariant position in a plurality of data constituting the time-series data, using a trained model stored in the memory, and the estimated reference pattern. In order to identify the identification pattern that appears at different positions in the plurality of data and output the identification result, and to re-estimate the reference pattern so as to follow the change of the identification environment. It can also be expressed as a digital circuit including a re-learning module that retrains the trained model by using at least a part of the acquired time-series data.

Further, the electronic circuit according to another aspect of the present disclosure is an estimation unit that estimates a reference pattern included in the time series data by using an acquisition unit that acquires time series data from a fixed point and a learning model that utilizes sparsity. It can also be expressed as an electronic circuit including an output unit that identifies a dynamic identification pattern with reference to the reference pattern and outputs the identification result.

The present disclosure is not limited to the above-described embodiment, and various modifications can be made within the scope of the claims, and the embodiment obtained by appropriately combining the technical means disclosed in each of the different embodiments. Is also included in the technical scope of the present disclosure. Furthermore, new technical features can be formed by combining the technical means disclosed in each embodiment.

The present disclosure describes any electronic circuit, such as an integrated circuit, digital circuit, analog circuit, mixed signal circuit, hardware component, programmable logic device, computer, smartphone, tablet terminal, camera, microphone, computer system, etc. It can be widely applied to any device including, and any information processing device.

1a: Electronic circuit, 1b: Edge terminal, 2: Camera (data recording device), 3a: Production line (data environment), 3b: Parking lot (data environment), 4a: Edge computing system, 4b: Edge computing system 5, 5a: Product (identification pattern), 5b: Automobile (identification pattern), 6: Video (time series data), 7a: Image (data), 7b: Re-learning image (re-learning data), 8a: Estimated result , 8b: Identification result, 8c: Judgment result, 9: Learned model, Control unit 10, 11: Acquisition unit, 12: Estimate unit, 13: Identification unit, 14: Judgment unit, 15: Relearning unit, 16: Memory Unit, 21: Input interface, 21: Recording unit, 22: Transmission unit, 23: Communication interface

Claims (9)

An acquisition unit that acquires a video shot at a fixed point as input from a data recording device,
A trained model in which a plurality of learning images included in a moving image taken in advance at a fixed point and a background appearing at an invariant two-dimensional position in the plurality of learning images are learned in advance, and a trained model.
A plurality of images constituting a moving acquired after the training by the acquisition unit, an estimation unit which to extract each background region which is a region corresponding to the background by the learned model,
With reference to each of the extracted background areas, a two-dimensional position where the identification target is different to move within the shooting range of the moving image in a region where the difference from each background region is larger than a predetermined value in the plurality of images. an identification unit, respectively identified as the identification area appearing, and outputs the result of the identification in,
The first background region extracted by the estimation unit in the first time is compared with the second background region extracted by the estimation unit in the second time after the first time. Comparison section and
A determination unit that determines whether the accumulation of background subtraction, which is the difference obtained by the comparison, exceeds a predetermined threshold value.
When it is determined that the cumulative background subtraction exceeds the threshold value, the moving image acquired by the acquisition unit from the time when the background was learned last time to the time when the cumulative background subtraction exceeds the threshold value. using at least a part, an electronic circuitry and a re-learning unit that is relearning the learned model. The estimation unit re-extracts each of the background regions using the retrained trained model.
The electronic circuit according to claim 1, wherein the identification unit re-identifies each of the identification areas with reference to each of the re-extracted background areas, and outputs the result of the re-identification. The acquisition unit obtains re-learning data used for re-learning the trained model from the plurality of images acquired from the time when the background was learned last time to the time when the cumulative background subtraction exceeds the threshold value. The electronic circuit according to claim 1 or 2, wherein the electronic circuit is sampled. The acquisition unit acquires two images, a high-resolution image and a low-resolution image, for each of the plurality of images.
The estimation unit extracts each of the background regions from the low-resolution image, and then extracts the background region.
The electronic circuit according to any one of claims 1 to 3 , wherein the identification unit identifies each identification region from the high-resolution image. The electronic circuit according to any one of claims 1 to 4.
With a given interface,
The electronic circuit is a hardware component that is a programmable logic device that implements a function of operating the electronic circuit as each of the parts. An edge terminal having the hardware component according to claim 5 and
An edge processing system comprising the data recording device that records the moving image and feeds the moving image to the hardware component provided in the edge terminal as an input. The edge processing system according to claim 6 and
Wherein in said moving image recorded by the data recording apparatus, the edge computing system including a photographing object constituting at least a part of the front dress Jing. The acquisition process of acquiring a video shot at a fixed point as input from a data recording device,
A trained model in which a plurality of learning images included in a moving image taken in advance at a fixed point and a background appearing at an invariant two-dimensional position in the plurality of learning images are learned in advance, and a trained model.
A plurality of images constituting a moving acquired after the training by the obtaining step, an estimating step to extract each background region which is a region corresponding to the background by the learned model,
With reference to each of the extracted background areas, a two-dimensional position where the identification target is different to move within the shooting range of the moving image in a region where the difference from each of the background regions is larger than a predetermined value in the plurality of images. an identification step of identifying a respective identification area, and outputs the result of the identification appearing respectively,
The first background region extracted by the estimation step in the first time is compared with the second background region extracted by the estimation step in the second time after the first time. Comparison process and
A determination step of determining whether the accumulation of background subtraction, which is the difference obtained by the comparison, exceeds a predetermined threshold value, and
When it is determined that the cumulative background subtraction exceeds the threshold value, the moving image acquired by the acquisition step from the time when the background was learned last time to the time when the cumulative background subtraction exceeds the threshold value. At least a portion with, and a re-learning process that is relearning the learned model identification method an electronic circuit executes. An acquisition function that acquires a video shot at a fixed point as input from a data recording device ,
A trained model in which a plurality of learning images included in a moving image taken in advance at a fixed point and a background appearing at an invariant two-dimensional position in the plurality of learning images are learned in advance, and a trained model.
A plurality of images constituting a moving acquired after the training by the acquisition function, and the estimated function to extract each background region which is a region corresponding to the background by the learned model,
With reference to each of the extracted background areas, a two-dimensional position where the identification target is different to move within the shooting range of the moving image in a region where the difference from each background region is larger than a predetermined value in the plurality of images. an identification function are identified as the identification area appearing, and outputs the result of the identification in,
The first background area extracted by the estimation function in the first time is compared with the second background area extracted by the estimation function in the second time after the first time. Comparison function and
A determination function for determining whether the accumulation of background subtraction, which is the difference obtained by the comparison, exceeds a predetermined threshold value, and
When it is determined that the cumulative background subtraction exceeds the threshold value, the moving image acquired by the acquisition function from the time when the background was learned last time to the time when the cumulative background subtraction exceeds the threshold value. using at least a part, a re-learning function and that is re-learning the learned model identification program for realizing on a computer.

Images