WO2023236789A1

WO2023236789A1 - Monitoring system

Info

Publication number: WO2023236789A1
Application number: PCT/CN2023/096516
Authority: WO
Inventors: 萨里尼约瑟夫; 姚晨; 杉山聪
Original assignee: 索尼半导体解决方案公司; 萨里尼约瑟夫
Priority date: 2022-06-07
Filing date: 2023-05-26
Publication date: 2023-12-14
Also published as: CN117255173A

Abstract

The present invention relates to a monitoring system. The monitoring system according to the present invention may comprise: a camera, which captures an image of a monitoring area and stores the image; and at least one graphic marker, which is mounted on at least one storage position in the monitoring area, wherein each of the at least one graphic marker is invisible to the camera when a target object is located at the corresponding storage position; and the camera comprises a circuit, which is configured to: capture an image of the monitoring area at a predetermined frequency, determine whether the graphic marker is comprised in each captured image, and send a determination result to the outside of the camera. By means of the present invention, the power consumption of a camera and the dependence of a monitoring system on a network are significantly reduced, the amount of images/video data to be processed by the camera and the complexity of a computation model are reduced, the speed and accuracy of data transmission are improved, and personal privacy can be securely protected.

Description

surveillance system

References to related applications

This application claims the rights and interests of Chinese Patent Application No. 202210636273.

Technical field

The invention relates to a monitoring system, which includes a smart camera with image processing function and at least one graphic mark. The monitoring system can monitor targets in the monitoring area of various application places.

Background technique

Current surveillance systems generally include cameras and external processing units that communicate with the cameras, such as edge computers, cloud processors, and various smart terminals such as mobile phones. In such a monitoring system, the camera sends real-time captured images to the processing unit, which displays and processes the images. In sales venues, some intelligent monitoring systems can process images through edge computing and obtain the sales status of goods.

However, there are many problems with these monitoring systems. First, the camera needs to capture and store a large amount of image/video data, and it needs to be sent to the processing unit, significantly increasing the camera's power consumption. Moreover, data transmission is limited by the bandwidth of wired or wireless networks, making the transmission quality unstable or even low. Secondly, the data transmitted by the camera may contain images of people, exacerbating the risk of personal privacy leakage.

In addition, the image processing involved in current surveillance systems is performed in a processing unit outside the camera. For example, in automatic monitoring systems such as smart sales stores/vending machines and smart warehouse monitoring, cloud processors are used to process images sent by cameras. This will also cause problems such as delays, losses and even errors in data transmission due to poor network transmission quality, which will lead to a decrease in the accuracy of image processing results.

Furthermore, traditional automatic surveillance systems all use detection methods that directly detect and analyze target objects in surveillance images/videos. This detection method requires a large amount of image data and calculations, and requires the establishment and training of complex computational models.

Contents of the invention

In order to solve the above technical problems. The present invention proposes a monitoring system, which includes a smart camera with image processing function and at least one graphic mark. Smart cameras are capable of processing images inside the camera and sending only the processing results rather than the complete image to the outside of the camera. In this way, the power consumption of the camera and the monitoring system's dependence on the network are significantly reduced, and the speed and accuracy of data transmission are improved. Problems such as data packet loss, transmission errors, data delays, etc. of traditional monitoring systems will not occur. At the same time, personal privacy can also be safely protected by sending only the processing results as feature data instead of image data. In addition, the present invention uses at least one graphic mark to monitor the target, which greatly reduces the amount of image/video data that needs to be processed, and also significantly reduces the complexity of the calculation model.

The monitoring system according to the present invention may include: a camera that captures an image of a monitoring area and stores the image; and at least one graphic marker installed to at least one placement position in the monitoring area, wherein the at least Each of a graphical marker is invisible to the camera when the target object is located at the corresponding position, wherein the camera includes circuitry configured to: capture the monitoring at a predetermined frequency image of the area, determine whether the graphic mark is included in each captured image, and send the determination result to the outside of the camera.

Preferably, the circuit of the camera is further configured to extract object information from the image of the monitoring area and send the object information to the outside of the camera.

Preferably, each of the at least one graphical mark has a selected color that is different from the color of the target object to be placed on the corresponding said storage location.

According to the monitoring system of the present invention, a plurality of graphic marks can be installed to a plurality of storage positions in the monitoring area respectively, and the plurality of graphic marks are different from each other. For example, multiple graphic indicia may have different patterns, shapes, and/or colors.

Preferably, the circuitry of the camera is further configured to associate each image marker with a corresponding storage location and/or an object to be placed at the corresponding storage location.

Preferably, the circuit of the camera is further configured to adjust the camera according to the recognition accuracy of the at least one graphic mark in the multiple images of the monitoring area by capturing and analyzing multiple images of the monitoring area. shooting parameters.

Preferably, in order to determine and/or inspect the target object, the circuit of the camera is further configured to: intercept an area of interest from the first image of the monitoring area, where the area of interest contains the object of interest located at the corresponding object location, A second image of the area of interest is captured, and the target object information of the target of interest is extracted from the second image.

Similarly, in order to determine the graphic mark, preferably, the circuit of the camera is further configured to: intercept an area of interest from the first image of the monitoring area, the area of interest containing the graphic mark of interest in the at least one graphic mark, A second image of the area of interest is captured, and it is determined whether the graphic marker of interest is included in the second image.

Preferably, the circuit of the camera is further configured to receive an external signal, and when the external signal indicates that someone enters the monitoring area, the camera starts to capture images of the monitoring area at the predetermined frequency. Optionally, the circuit of the camera is also configured to detect whether someone enters the monitoring area by detecting and extracting facial features from images of the monitoring area.

Preferably, the circuitry of the camera is further configured to, when determining that the graphic mark is included in a captured image, send the determination result together with a fill signal indicating an empty storage location. Preferably, the circuit of the camera is further configured to send the determination result and a filling signal indicating an empty storage position when it is determined that the graphic mark is included in a predetermined number of consecutive captured images.

The camera of the monitoring system according to the present invention includes an integrated chip, and the circuit of the above-mentioned camera is arranged on the integrated chip. A memory is also included in the integrated chip, and the memory stores a predetermined algorithm for image processing of the circuit.

Detailed and preferred embodiments of the present invention will be described below with reference to the accompanying drawings.

Description of the drawings

Figure 1 is a schematic diagram showing a first embodiment of a monitoring system according to the present invention.

Figure 2 is a schematic diagram illustrating one embodiment of graphical markings of the monitoring system according to the present invention.

FIG. 3 is a schematic diagram showing one embodiment of the chip structure of the camera of the monitoring system according to the present invention.

Figure 4 is a schematic diagram showing a second embodiment of the monitoring system according to the present invention.

Figure 5 is a schematic diagram showing a third embodiment of the monitoring system according to the present invention.

FIG. 6 is a schematic diagram illustrating an embodiment of a camera of the surveillance system recognizing and extracting facial features of a human face according to the present invention.

7 is a schematic diagram showing an embodiment of images captured by a camera of the monitoring system according to the present invention.

8 is a schematic diagram illustrating an embodiment in which a camera of a surveillance system determines graphic markers in a surveillance area according to the present invention.

FIG. 9 is a schematic diagram showing another embodiment of a camera of a surveillance system determining graphic markers in a surveillance area according to the present invention.

FIG. 10 is a flowchart illustrating an embodiment of a camera of a surveillance system determining whether a graphic mark exists in an image according to the present invention.

FIG. 11 is a schematic diagram showing the working flow of the camera of the monitoring system according to the present invention.

Detailed ways

Hereinafter, preferred embodiments of the present disclosure are explained in detail with reference to the accompanying drawings. Unless otherwise stated, the same reference numbers refer to the same or equivalent parts.

Figure 1 shows a first embodiment of the monitoring system of the present invention. As shown in Figure 1, the camera 1 of the monitoring system is installed in the monitoring area and can perform panoramic monitoring of the monitoring area. A plurality of target objects 2 are arranged in the monitoring area, and the target objects 2 are placed on corresponding storage positions 3 respectively. A supply source 8 of target objects is also arranged in the monitoring area, which is used to replenish the target objects 2 to the corresponding storage position 3 in time when needed. For example, the monitoring area can be a vending machine, and the target object 2 is a commodity. Based on the real-time monitoring of the camera 1, after receiving the filling instruction, the vending machine automatically fills the corresponding storage position with the target object 2 stored in the supply source 8. Alternatively, the monitoring area can also be a storage area such as a warehouse, where the target object 2 is the stored goods and the storage location 3 is the inventory location in the warehouse. Based on the real-time monitoring of the camera 1, after receiving the filling instruction, the target object 2 in the supply source 8 is automatically replenished to the corresponding storage position 3. Of course, the monitoring area is not limited to this. The monitoring system of the present invention can be applied to any area that has monitoring requirements for target items. place.

The monitoring system also includes at least one graphic marker 4 . In the embodiment shown in FIG. 2 , a plurality of graphic marks 4 are respectively installed on corresponding target object storage positions 3 . For example, the graphic mark 4 can be adhered to the storage position 3 by means of an adhesive without any modification of the storage position 3 being necessary. Or alternatively, the graphic mark can be embedded in the corresponding storage position 3 through mechanical means. This method can reliably prevent the graphic mark from falling off or falling off due to operating errors.

As shown in the upper image of FIG. 2 , when the target object 2 is placed on the corresponding object position 3 according to the predetermined layout, the graphic mark 4 is invisible to the camera 1 . For example, the graphic mark 4 can be adhered to the surface of the object placement position 3 so that the target object 2 will block the graphic mark 4 after it is placed. As shown in the lower image of FIG. 2 , when the target object 2 is not on the corresponding storage position 3 , the graphic mark 4 on the storage position 3 is exposed and can then be captured by the camera 1 . The camera 1 may determine that the corresponding storage position is empty based on detecting the graphic mark 4 in the captured image, and then send the determination result to the outside of the camera, such as an external processing unit. The external processing unit sends instructions to the monitoring area based on the received judgment result that there is an empty level (empty level 3), and the monitoring area performs corresponding measures according to the instructions, such as filling the empty level, or sending out an alarm signal, etc.

FIG. 2 shows that multiple target objects 2 and multiple graphic marks 4 are respectively placed in multiple target object placement positions 3 . Each graphic mark 4 is different, so that when the camera 1 recognizes the graphic mark 4, it can simultaneously determine the specific location of the corresponding object location and the corresponding target object information (eg, type/category, etc.). In this regard, the circuit included in the camera 1 can pre-associate each graphic mark 4 with information such as the corresponding object position 3 and/or the type, position coordinates and other information of the target object 2 .

Multiple graphic marks can be distinguished by graphics, shape, and/or color. Of course, the present invention is not limited to this. Any graphic markings that are distinguishable from each other and thereby enable the camera to distinguish them from the image can be used in the present invention. At the same time, each graphic mark should also be distinguished from the corresponding target object, so that the camera can accurately identify the graphic mark and the target object respectively. For example, the image mark may include one or more colors that are different from the colors contained in the target object. Specifically, as shown in the lower image of FIG. 2 , the graphic mark 4 has a lighter color (white in the figure) than the target 2 and has diagonal stripes. The camera 1 can easily and quickly detect the graphic mark in the captured image by recognizing the graphic and the different colors, thereby determining the empty storage position. For ease of illustration and understanding, Figure 2 only shows graphic markers and targets distinguished by a single shade of color and a simple stripe pattern. It is easy to understand that graphic marks can be constructed in a variety of ways, including but not limited to one or a combination of graphics, shapes, colors, special symbols, etc.

The camera 1 used in the monitoring system of the present invention will be described in detail below. Camera 1 is a smart camera with its own image processing function. The interior of the camera 1 includes an integrated circuit chip, which may have a stacked structure as shown in FIG. 3 , such as a stacked complementary metal oxide semiconductor (CMOS) image sensor chip. Specifically, the integrated chip is composed of a stacked substrate structure. A pixel array composed of a plurality of pixels is arranged on the first substrate in the stacked substrate. The pixel array converts the optical signal received by the camera lens into an electrical signal through photoelectric conversion, and transmits the electrical signal to the second substrate in the stacked substrate. superior. A memory and a processing circuit are arranged in the second substrate, and the second substrate is coupled to the first substrate (the coupling structure is not shown in the figure) to communicate with each other. Optionally, the memory may also be located on a third substrate (not shown) in the stacked substrate. That is, the pixel array, memory and processing unit can be located on different substrates. It should be understood that the above-mentioned stacked arrangement of multiple substrates is only exemplary. pixel arrays, pixel circuits, processing circuits, memories and other components can be designed on multiple substrates and stacked according to design requirements. The processing circuit includes, for example, a digital signal processor (DSP), which generates image data based on electrical signals and stores the image data in a memory. Predetermined algorithm models are also stored in the memory, and the processing circuit can perform various processing on the image based on these algorithm models, such as detecting whether there are graphic markers in the surveillance image, and different processing of the image as will be described in detail below.

As a result, the image processing of the surveillance system can be performed on the camera side, rather than in an external processor (edge computer, cloud server, mobile phone terminal, etc.) connected to the camera through wired or wireless connections. Of course, camera 1 can also access the network, for example, to obtain or update algorithm models, etc.

An example of image processing performed in Camera 1 is shown in Figure 8. FIG. 8 shows an example of an image of the monitoring area captured by the camera 1 . The left image in FIG. 8 shows the first image captured by the camera 1 at a certain moment. The circuit in the camera 1 processes the first image and initially detects the graphic marks 4 and 4'. After further calculation and processing, for example, comparing it with a pre-stored standard graphic mark image, it is determined that the graphic mark 4 is detected (for example, the probability greater than 90%). At the same time, the circuit determines that the probability that the pattern mark 4' is the pattern mark 4 is 90% or less. At this time, the circuit can intercept the area of interest containing the graphic mark 4' from the first image (as shown in the dotted box in the left image of Figure 8), and instruct the camera 1 to adjust the shooting parameters, such as focal length, resolution, exposure time and possible other parameters (white balance, noise reduction, etc.), and then shoot the area of interest containing the graphic mark 4' to obtain a second image with higher image quality, as shown in the right image of Figure 8. The circuit processes the clear second image, confirms that graphic mark 4 is detected, and then sends the judgment result to the outside of the camera.

The camera circuit can send only the judgment result to the outside. For example, in the embodiment shown in FIG. 8 , only the result of detecting two graphic marks 4 is sent to the external processing unit as feature data. Based on the detection results, the external processing unit determines that two vacant material levels appear in the monitoring area, and obtains the material location information and target object information associated with these two graphic marks 4. Then, the processing unit can issue corresponding instructions to the monitoring area, such as replenishing the target object 2 to the corresponding empty level, and/or causing the monitoring area to issue an alarm.

Compared with traditional surveillance systems that need to send a large amount of image/video data to the outside for processing and judgment by an external processing unit, the surveillance camera of the present invention only uses the detection results as feature data (for example, "m where target 2 is placed" bits and n bits found graphic mark 4" or "bit m and n bits are vacant material levels") are sent to the outside, which greatly reduces the amount of transmitted data, thereby significantly improving the quality and speed of transmission. Moreover, traditional surveillance systems monitor targets by detecting and judging target images in surveillance images. The detection and judgment of target images requires complex modeling and practice processes. In contrast, the present invention proposes to use graphic markers to indirectly monitor target objects by detecting graphic markers in monitoring images. By using simple graphical notations, the process of modeling and practicing computational models is greatly simplified, and the amount of data that needs to be processed is significantly reduced, thereby significantly improving image processing speed. Furthermore, compared with the traditional monitoring system, depending on the processing function of the monitoring camera of the present invention, the camera can adjust the shooting parameters and angles in real time and automatically as needed. The present invention only needs to arrange one camera in the monitoring area.

Optionally, in some cases, the camera can also send part of the image data outside the camera. For example, in some special cases, the camera cannot always accurately determine whether a graphic mark is detected (for example, when a predetermined number of shooting and detection are repeated, the probability is always below 90%, for example). At this time, the camera can send the second image as shown in Figure 8 and/or the area of interest image captured from the first image or the second image to the outside of the camera through a request from the external processing unit, for identification by the user or more complex The system makes a judgment. In this case, the amount of image data that needs to be sent is still much smaller than that of a traditional surveillance system, because the traditional surveillance system does not perform any processing on the captured images. Additionally optionally, it will be appreciated that the camera is also capable of sending complete image data external to the camera. In this case, the external processing unit can issue a complete data request to the camera.

In the manner described above, the camera 1 can use the graphic mark 4 to automatically, quickly and accurately determine whether the target object in the monitoring area exists (for example, whether it is sold, unloaded, stolen or accidentally dropped, etc.).

And, in a similar manner, camera 1 can also determine whether the layout of the target object is correct. Specifically, the camera 1 acquires a first image of the monitoring area, and the camera circuit intercepts an area of interest from the first image, where the area of interest contains at least one target object. The camera circuit processes the area of interest to obtain target information. For example, the camera circuit can identify the type/category of the target by comparing and matching the target in the image with pre-stored images of multiple targets. When the target object is a product with a machine-readable barcode such as a QR code or barcode, the camera circuit can also obtain information such as the type/type of the target object by identifying the QR code or barcode on the package of the target object. In addition, similar to the method shown in Figure 8, when the target information cannot be accurately obtained from the first image, the camera circuit can adjust the shooting parameters to obtain a second image containing the area of interest, and then obtain the second image with improved image quality. Extract target object information from the second image. Then, the camera circuit can compare the target object information with the pre-stored arrangement information to determine whether the target object is placed in the correct position. Optionally, the camera circuit can also send the target object information to an external processing unit, and the external processing unit performs subsequent processing and judgment.

Also similar to the processing method of the embodiment of FIG. 8 , the camera circuit can only extract and identify the target object information data (for example, "such and such Coke") from the image, instead of a huge amount of image data. Sent to the outside, thereby significantly reducing the transmission pressure on network bandwidth. Of course, the camera circuitry can also send partial or complete image data to the outside based on external requests.

In addition, when the camera is installed in the monitoring area, the camera circuit can adjust the shooting parameters in advance based on the recognition accuracy of the graphic marks in the monitoring area image for different monitoring areas, targets therein, and the layout of the targets. set up. Specifically, the camera can take multiple shots of the monitoring area with multiple sets of different shooting parameters in advance, and process and analyze the multiple sets of images obtained. The shooting parameters here include, for example, but are not limited to shooting angle, resolution, exposure time, white balance, noise reduction, etc. Based on the shooting parameters of image frames with higher recognition accuracy of graphic marks in multiple sets of images, the camera circuit determines a preferred shooting parameter range suitable for the monitoring area and its target objects. Therefore, based on the determined shooting parameter range, the camera circuit can set the shooting parameters of the monitoring area in advance. In this way, the judgment efficiency and accuracy during monitoring are further improved.

Figure 4 shows a second embodiment of a monitoring system according to the invention. In the second embodiment, the monitoring system is provided in a monitoring area of a vending store, for example. As shown in Figure 4, the camera 1 is arranged so that its viewing angle covers multiple targets 2 sales area and inventory area 5. The sales area includes a plurality of storage locations (not shown), and graphic marks (not shown) are arranged on each storage location. Graphical markings are similar to those described above for Figure 2. The monitoring area is also equipped with sensors (not shown) that monitor whether someone enters the monitoring area. Camera 1 is connected to this sensor and receives the signal from the sensor. When the sensor sends a signal that someone enters the monitoring area, the camera 1 ends the standby state, starts shooting the monitoring area at a predetermined frequency, and processes the images. Upon detecting one or more graphic markers, camera 1 sends the detection results to the external processing unit. Optionally, the camera 1 directly sends a filling signal to the inventory area 5 based on the detection result, instructing the inventory area 5 to fill the target object into the corresponding empty level. After confirming that the empty material position is filled, the camera 1 takes a picture of the inventory area 5 and processes the captured image to determine whether the supply source of the target object is empty. If it is determined that an empty supply source exists, the determination result is sent to the external processing unit. After receiving the reminder from the external processing unit, the user can replenish the supply source accordingly.

Figure 5 shows a third embodiment of a monitoring system according to the invention. The third embodiment differs from the second embodiment in that the camera 1 of the surveillance system is arranged to be able to simultaneously monitor the sales area, the storage area 5 and the entrance to the surveillance area. That is, camera 1 also monitors whether anyone enters the monitoring area.

Specifically, as shown in Figure 5, the camera takes pictures of the entrance to the surveillance area at a predetermined frequency, and determines whether someone has entered the surveillance area by detecting and extracting facial features in the image (i.e., the dotted facial feature map in Figure 6) district. Detecting facial features typically involves locating faces in images and detecting key facial structural features. First, the camera circuit uses, for example, a deep learning-based algorithm to locate the face in the image and obtain the face boundary points. Then, key facial structural features in the face area are detected within the face boundary, such as the mouth, right eyebrow, left eyebrow, right eye, left eye, nose, and mandible, and the feature points shown in Figure 6 are obtained. It should be understood that any known detection and extraction method of facial features can be selected as needed, and the relevant algorithms can be stored in the chip of the camera. Of course, when the camera is connected to the network, relevant calculations can also be performed through the cloud and the calculation results can be obtained.

The camera circuit determines that someone has entered the monitoring area based on the detected face facial feature data, and starts photographing the sales area at a predetermined frequency based on the determination. Alternatively or preferably, the camera circuitry may send the facial feature data externally. For example, as shown in Figure 6, the camera 1 sends the detected facial feature map to the processing unit 7 outside the camera through the network 6. The processing unit 7 and the external processing units mentioned in this article can be a remote computer as shown in Figure 6, or a cloud processor, an application program, or an application program on a mobile phone terminal, etc. As shown in Figure 6, what is transmitted between the camera 1 and the processing unit 7 through the network 6 is the facial feature map extracted by the camera 1. This data is greatly reduced whether compared with the original image data or the face image data. The amount of data transferred and the speed of transfer. In this way, the need to transmit images with huge amounts of data is avoided, while the privacy of consumers is also avoided.

In addition, in this regard, as shown in the left image of Figure 7, camera 1 may capture an image containing human hands. In some applications, camera 1 can process the human hands in the image, extract visual marks such as birthmarks or tattoos that the hands may contain, and send the visual marks to the outside. Thus, the external processing unit can assist in identifying the target object possessing the hand through the visual marker. Since the camera 1 processes the image and only sends the visual mark to the outside, the efficiency of the external processing unit in identifying the target object can be improved. On the contrary, in other application situations, when the target object is a group of people whose private information needs to be protected (for example, when the surveillance area is a sales place and the target object is ordinary consumers), when camera 1 detects human hands, it will not An image containing a human hand is sent externally. In this way, different purposes are achieved by setting up the camera 1 differently in different monitoring places/situations.

Furthermore, the right image of FIG. 7 shows an image captured by the camera 1 including the graphic mark 4 . As shown by the arrows in Figure 7, if the left image and the right image respectively appear in sequential order in the image frames of continuous shooting (for example, 15fps), the camera 1 determines that the corresponding target object 2 is taken away or sold. On the contrary, if A in Figure 9 and the right image in Figure 7 appear in sequentially captured image frames, the camera 1 determines that the target 2 may not have been taken away but accidentally slipped. In this way, the detection of human hands in the image can also assist the camera 1 in determining the status of commodities or goods in the monitoring area.

In addition, when the camera captures the image on the left as shown in Figure 7, the camera may not be able to accurately determine whether there is a graphic mark 4 in the image due to the occlusion of the human hand. At this time, the camera will wait for a certain period of time (to wait for the human hand to leave) before shooting again. If the camera is still unable to make a determination after a predetermined time, for example, a suspicious person remains in the monitoring area, the camera circuit will send a notification signal to the outside to alert the user.

FIG. 9 shows an embodiment in which the camera 1 detects the graphic mark 4 . A of FIG. 9 shows an image captured by the camera 1 when monitoring the monitoring area. If no graphic mark is detected in the image, the camera 1 does not send any signal or image to the outside. The camera 1 continues to operate, and when the image shown in B of FIG. 9 is captured, the camera circuit detects the graphic mark 4, and then only sends the detection result to the outside. Optionally, the camera circuitry sends the detection result externally together with a fill signal indicating an empty storage location. Additionally optionally, the camera circuitry sends the fill signal directly to the supply as shown in Figure 1 Source 8 or inventory area 5 as shown in Figures 4 and 5. After receiving the filling signal from the external or camera circuit, the supply source 8 or the inventory area 5 replenishes the target object 2 to the corresponding storage position. C in FIG. 9 shows an image of the target object 2 being filled captured by the camera 1. Based on this image and the subsequent image of the replenishment completed (A in FIG. 9), the camera 1 determines that the replenishment is successful. If the camera captures an image as shown in D in Figure 9 within a predetermined period of time after the filling signal is sent, it is judged that the replenishment has failed, and the judgment result is sent to the outside. At the same time, the camera 1 takes pictures of the supply source 8 or the inventory area 5, analyzes the pictures, determines whether the supply source of the corresponding target object in the supply source 8 or the inventory area 5 is empty, and sends the judgment result to the outside. If the camera 1 determines that the supply source of the corresponding target object is empty, optionally, a signal for replenishing the supply source can be sent to the outside at the same time. On the other hand, if camera 1 determines that the supply source of the corresponding target object is not empty, it may be that the automatic filling equipment fails or the target object to be filled is stuck, resulting in the failure to fill normally. At this time, the circuit of camera 1 Send a reminder signal to the outside, and the external processing unit can remind the user to check and repair.

It is worth pointing out that the graphic mark 4 will also be partially exposed and photographed and detected by the camera 1 at the moment shown in C of Figure 9 , but the camera 1 will not inappropriately send out a fill signal in such a situation. Based on the algorithm stored in camera 1, the camera will only send a judgment result (and optionally, a filling signal) that there is an empty material level when it detects a complete graphic mark 4 as shown in B of Figure 9 . Correspondingly, when the image as shown in C in FIG. 9 is captured outside the time sequence shown in ABC in FIG. 9 , for example, when the target object 2 is being taken out, the camera 1 will not erroneously send the judgment result that there is an empty object level. .

FIG. 10 shows a specific example of the camera circuit of the monitoring system according to the present invention determining whether there is a graphic mark in the image. As shown in Figure 10, when the camera circuit processes the captured image, if the possibility of detecting a graphic mark is judged to be 90% or above based on the algorithm model stored in the camera, it is determined that the graphic mark is detected; if it is determined that the graphic mark is detected When the probability of reaching a graphic mark is 30% or less, it is determined that no graphic mark is detected.

On the other hand, if it is determined that the possibility of detecting a graphic mark is greater than 30% and less than 90%, the camera circuit can adjust the shooting parameters and shoot again (count+=1) to analyze and analyze again based on the optimized image. Judge until it can be determined whether a graphic mark is detected. This step is repeated up to 3 times. If it is determined that a graphic mark is detected (ie, the probability is 90% and above) during the xth (x≤3, that is, count<=3 as shown in Figure 10) times of shooting and analysis, then enter Figure 11 Step S5 shown; if it is determined that no graphic mark is detected during the xth (x≤3) shooting and analysis (that is, the probability is 30% and below), then enter step S4 as shown in Figure 11 .

On the other hand, if after repeating 3 times it is still not possible to determine whether the graphic mark is detected (the probability falls between 30% and 90%) (i.e. count>3 shown in Figure 10), the camera circuit sends an external signal Alert signal. After receiving the reminder, the user can request the surveillance area image from the camera through the external processing unit, and the user will make manual judgment based on the partial/complete image of the surveillance sent by the camera. Alternatively, when the camera circuit cannot determine that the cause is a malfunction or obstacle, the user can go to the monitoring area for inspection.

It should be understood that the possibility values 90% and 30% shown in Figure 10 are only exemplary, and users can select other values as thresholds based on design needs, usage scenarios and other factors. For example, 80% and 20% etc.

Figure 11 shows an example of the working steps of the camera of the monitoring system according to the present invention. As shown in Figure 11, in step S1, it is determined whether someone enters the monitoring area. For example, in the embodiment shown in FIG. 4 , a sensor installed at the entrance of the monitoring area sends a signal to the camera 1 when it detects that someone enters the monitoring area. When this signal is not received, camera 1 is in standby mode. Or alternatively, as in the embodiment shown in Figure 5, when the camera 1 is arranged with a viewing angle covering the entrance of the monitoring area, the entrance of the monitoring area can be photographed, and whether someone enters the monitoring area can be determined by processing the image of the entrance of the monitoring area. district. The judgment method here is as described above based on Figure 6 .

After the camera 1 detects that someone enters the monitoring area or receives an external signal that someone enters the monitoring area, the camera 1 starts to photograph the monitoring area, as shown in step S2. In S2, camera 1 takes pictures of the monitoring area at a predetermined frequency. The camera circuit proceeds to step S3 when the graphic mark 4 may be detected in the captured image. In S3, the camera circuit determines whether a vacant level exists by determining whether a graphic mark is detected. When it is determined that the graphic mark 4 is not detected in one image currently captured, or preferably, when the graphic mark 4 is not detected in a plurality of consecutively captured images (a preset number), the camera circuit makes a decision that there is no vacancy. Judgment of material level, and then return to S1. On the other hand, in S3, when the camera circuit determines that there is a graphic mark 4 in an image currently captured, it determines that the storage position corresponding to the graphic mark 4 is empty. Alternatively or preferably, for example, in order to improve the accuracy of judgment, when the camera circuit detects the graphic mark 4 in multiple frames (a preset number) of images continuously captured, determine the object corresponding to the graphic mark 4. The position is empty, and information about the position and/or target associated with the empty position is obtained. When multiple graphic marks 4 are detected at the same time, the camera circuit determines the location information corresponding to each graphic mark 4 (for example, the specific location in the monitoring area) and/or the target object 2 information (e.g., type/type of target). In addition, in S3, the circuit of camera 1 may detect that the monitoring area is blocked by obstacles, such as consumers or other personnel wandering in front of the sales area, or the hands and other parts of consumers or other personnel acting as obstacles blocking camera 1 Partial perspective. At this time, as shown in Figure 10, when the camera still cannot determine whether there is a target or graphic mark in the captured image after repeated detection a predetermined number of times (for example, 3 times), the camera circuit will send a reminder signal to the outside to remind the user to deal with the obstacle. object/obstacle situation. Optionally, the camera can also send part of the captured images to the outside based on the user's request, and the user can make manual judgment.

When one or more graphic marks 4 are detected in S3, step S5 is entered. In S5, the camera circuit combines the determination result in S3 and the acquired feature information (for example, "the object location at a certain row and column where the target object The object position somewhere where the target Y is placed...is empty") is sent to the outside instead of sending the captured image to the outside. Optionally, the camera circuit simultaneously sends the fill signal to the outside together with the above determination result. After receiving the above determination result (and optionally, the filling signal), the external processing unit issues an instruction to instruct the monitoring area to fill the target object into the corresponding vacant material level. Alternatively, the camera circuit may directly send a filling signal to the supply source of the monitoring area (for example, the supply source 8 shown in Figure 1 or the inventory area 5 shown in Figure 4), instructing the supply source 8 or the inventory area 5 to The corresponding empty material level is filled with the corresponding target object. Next, proceed to step S6. In S6, wait for a predetermined time (for example, 2 seconds) to complete the target filling. Then proceed to step S7. In the S7, the camera circuit detects whether the empty level is actually filled. In this step, similarly to S3, it is determined whether a graphic mark is detected in the captured image. If graphic mark 4 is not detected, the camera circuit determines that the empty material level has been filled, and then returns to S1; if graphic mark 4 is detected, it proceeds to step S8. In S8, it is determined whether the corresponding target stored in the supply source 8 or the inventory area 5 is empty. Specifically, the camera 1 photographs the supply source 8 or the inventory area 5, and processes the photographed image to detect whether the supply source 8 or the inventory area 5 of the corresponding target object is empty. If it is determined that the supply source of the corresponding target object is empty, step S10 is entered. In S10, the camera circuit sends the judgment result to the outside. Optionally, the camera circuit sends an instruction to replenish the supply source to the outside together with the judgment result. After receiving a signal from the external processing unit to replenish the supply source, the user replenishes the supply source 8 or the inventory area 5 accordingly.

In S7, when the camera circuit cannot determine whether the vacant material position is filled (for example, the captured image is unclear, blocked by obstacles, etc.), step S9 is entered. In the S9, the camera circuit optimizes the shooting parameters and repeats shooting and judgment until it can accurately judge whether the empty position is filled. Similar to the embodiment shown in FIG. 10 , the steps may be repeated up to three times. If it can be determined whether the graphic mark is detected during the x (x≤3)-th shooting and analysis (thereby being able to determine whether the vacant material level is filled), enter S7; if it is still repeated after 3 times (i.e., x>3) If it cannot be determined whether the empty material level is filled, the camera circuit sends an external reminder. Users can go to the monitored area to check, for example, to clear obstacles.

The above processing steps are stored as a program in the memory of the integrated chip of the camera 1, and are used to instruct the camera to perform corresponding operations during the monitoring process when the processing circuit in the camera reads the program. It can be understood that the above program can also be stored in an external processing unit or a computer-readable storage medium, and is used to obtain the camera shots by the external processing unit/computer under special circumstances (for example, the camera circuit cannot operate the image normally). photos and process them outside the camera.

The above-mentioned image processing includes the processing of graphic marks, targets, and facial features in the image. The calculations involved can be performed using known algorithms for recognizing graphics or using known calculation methods. Modeling. I won’t go into details here.

Although the above embodiment is described by taking a monitoring system arranged in a vending machine/store as an example, the present invention is obviously not limited to this, but can be applied to any situation where there is a need to monitor target objects.

Claims

A monitoring system including:

a camera that captures images of the surveillance area and stores the images; and

At least one graphical mark is installed to at least one placement position in the monitoring area, wherein each of the at least one graphical mark is invisible to the camera when the target object is located at the corresponding placement position. visible,

Wherein, the camera includes a circuit configured as:

capturing images of the surveillance area at a predetermined frequency,

determine whether the graphic marker is included in each captured image, and

Send the determination results to the outside of the camera.
The monitoring system of claim 1, wherein the circuit is further configured to extract target information from the image of the monitoring area and send the target information to the outside of the camera.
The monitoring system of claim 1, wherein each of said at least one graphical mark has a selected color that is consistent with said item to be placed on the corresponding said storage location. The target objects are of different colors.
The monitoring system according to claim 1, wherein a plurality of graphic marks are respectively installed to a plurality of storage positions in the monitoring area, and the plurality of graphic marks are different from each other.
The monitoring system of claim 4, wherein the plurality of graphic markers have different patterns, shapes and/or colors.
The monitoring system of claim 4, wherein the circuit is further configured to associate each of the image markers with the corresponding storage location and/or the object to be placed at the corresponding storage location. Target objects are associated.
The monitoring system of claim 1, wherein the circuit is further configured to mark the plurality of images in the monitoring area according to the at least one graphic by capturing and analyzing the plurality of images of the monitoring area. The shooting parameters of the camera are adjusted based on the recognition accuracy.
The monitoring system of claim 2, wherein the circuit is further configured to:

An area of interest is intercepted from the first image of the monitoring area, where the area of interest includes an object of interest located at a corresponding object location,

capturing a second image of the area of interest, and

The target object information of the target object of interest is extracted from the second image.
The monitoring system of claim 1, wherein the circuit is further configured to:

A region of interest is intercepted from the first image of the monitoring area, and the region of interest contains a graphic mark of concern in the at least one graphic mark,

capturing a second image of the area of interest, and

It is determined whether the graphic mark of interest is included in the second image.
The monitoring system of claim 1, wherein the circuit is further configured to receive an external signal, and when the external signal indicates that someone enters the monitoring area, the camera begins to capture the predetermined frequency. images of the surveillance area, or,

Wherein, the circuit is further configured to detect whether someone enters the monitoring area by detecting and extracting facial features from the image of the monitoring area.
The monitoring system of claim 1 , wherein the circuit is further configured to, when it is determined that the graphic mark is included in a captured image, send the determination result together with a fill signal indicating an empty storage location.
The monitoring system of claim 1 , wherein the circuit is further configured to, when it is determined that the graphic mark is included in a predetermined number of consecutive captured images, transmit the determination result together with a message indicating an empty storage location. Fill signal.
The monitoring system according to any one of claims 1-12, wherein the camera includes an integrated chip, and the electronic Roads are arranged on the integrated chip, and the integrated chip further includes a memory storing a predetermined algorithm for image processing of the circuit.