CN108742159B - Intelligent control device and method for water dispenser based on RGB-D camera - Google Patents
Intelligent control device and method for water dispenser based on RGB-D camera Download PDFInfo
- Publication number
- CN108742159B CN108742159B CN201810306420.0A CN201810306420A CN108742159B CN 108742159 B CN108742159 B CN 108742159B CN 201810306420 A CN201810306420 A CN 201810306420A CN 108742159 B CN108742159 B CN 108742159B
- Authority
- CN
- China
- Prior art keywords
- water
- cup
- camera
- rgb
- information storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A47—FURNITURE; DOMESTIC ARTICLES OR APPLIANCES; COFFEE MILLS; SPICE MILLS; SUCTION CLEANERS IN GENERAL
- A47J—KITCHEN EQUIPMENT; COFFEE MILLS; SPICE MILLS; APPARATUS FOR MAKING BEVERAGES
- A47J31/00—Apparatus for making beverages
- A47J31/44—Parts or details or accessories of beverage-making apparatus
- A47J31/46—Dispensing spouts, pumps, drain valves or like liquid transporting devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
Abstract
The invention discloses an intelligent control device of a water dispenser based on an RGB-D camera and a control method thereof, wherein the control method comprises the following steps: an RGB-D camera, a control and information storage/processing unit and a power supply; the output end of the power supply is connected with the power supply input end of the control and information storage/processing unit and is used for directly or indirectly providing working power supply for the control and information storage/processing unit and the RGB-D camera; the control and information storage/processing unit is electrically connected with the RGB-D camera. According to the invention, the depth image of the water taking container (cup) obtained by the RGB-D camera is used for extracting characteristic information such as the position of the opening edge of the water taking container (cup) of the water dispenser, the height of the opening edge and the like, so that the functions of intelligently controlling the water dispenser to start and stop water outlet are realized.
Description
Technical Field
The invention belongs to the technical field of water dispensers, and particularly relates to an intelligent water outlet control device of a water dispenser and a control method thereof.
Background
With the continuous popularization of water dispensers, the functional requirements of consumers on the water dispensers are higher and higher. The existing water dispenser needs to finish quantitative water taking through key operation, cannot realize the intelligent control of the water taking process, and brings certain inconvenience to users.
Disclosure of Invention
The invention solves the technical problem of providing an intelligent control device and a control method for water outlet of a water dispenser, which judge whether a cup exists below a water outlet of the water dispenser and whether a cup opening edge is aligned with the water outlet by the technical means of an RGB-D camera, control to open or close the water flow of the water outlet of the water dispenser and intelligently control the water outlet quantity.
In order to realize the control device and the control method thereof, the technical scheme adopted by the invention is as follows:
an intelligent control device of a water dispenser based on an RGB-D camera comprises: an RGB-D camera, a control and information storage/processing unit and a power supply; the output end of the power supply is connected with the power supply input end of the control and information storage/processing unit and is used for directly or indirectly providing working power supply for the control and information storage/processing unit and the RGB-D camera; the control and information storage/processing unit is electrically connected with the RGB-D camera.
Further, in the intelligent control device for the water dispenser based on the RGB-D camera, the RGB-D camera is fixedly arranged on the body of the water dispenser and is close to the water outlet of the water dispenser; the RGB-D camera sets a shot target area, and the target area comprises a plurality of fixed mark points positioned on the water dispenser and used as mark points for establishing a three-dimensional coordinate system.
Further, the above-mentioned water dispenser intelligent control device based on RGB-D camera, RGB-D camera includes color camera and depth camera: the color camera shoots RGB color images, and the depth camera shoots depth images; the working principle of the depth camera is one of the following three principles: I) structured light measurement principle, II) binocular parallax measurement principle, III) TOF flight time measurement principle.
Furthermore, the intelligent control device of the water dispenser based on the RGB-D camera is characterized in that; the auxiliary lighting unit is used for providing working power supply for the auxiliary lighting unit; the auxiliary lighting unit is fixedly arranged on the water dispenser body at the upper part of the target area.
The intelligent control method of the water dispenser based on the RGB-D camera comprises the following steps:
s1, establishing a background model after registration of a target area: the target area has no foreign object, the control and information storage/processing unit synchronously obtains and processes a background color image and a depth image of the target area through the RGB-D camera, converts the images into a background point cloud image, registers the color image and the depth image, and establishes a corresponding relation between color image pixel points and depth information of the depth image, so as to obtain a pixel position and depth image transformation model of the color image; the control and information storage/processing unit obtains a pixel position and a depth transformation model according to registration, and the registration method comprises the following steps: adopting registration based on feature point matching, wherein feature points of a color image and a depth image synchronously obtained by an RGB-D camera are detected by using a scale invariant feature transform algorithm (SIFT) or other related feature point algorithms, and the feature points of the images are matched; removing the wrong matching point pairs of the images by using a random consensus algorithm (RANSAC) or other algorithms to obtain correctly matched characteristic point pairs; obtaining the corresponding relation between the correctly matched position and depth information of the feature point pair according to the position and the depth information of the correctly matched feature point pair, and calculating the pixel position and the depth transformation model between the two frames of images so as to obtain a registered depth image with higher resolution;
according to the known three-dimensional coordinate information of the positions of the mark points and the water outlet and the parameters of the RGB-D camera, establishing a post-registration background model of the target area, and storing the post-registration background model by the control and information storage/processing unit;
s2, extracting a foreground depth image after target area registration: placing a foreign object in a target area, under the condition that water does not exist at a water outlet of the water dispenser, synchronously acquiring a color image and a depth image of the target area in real time by the control and information storage/processing unit through an RGB-D camera, performing noise threshold processing, smoothing processing and the like on the color image and the depth image, performing registration in the step S1, realizing foreground and background segmentation, selecting an effective algorithm to extract a foreground contour, namely the contour of the foreign object in the target area, selecting the effective algorithm to extract feature points of the foreign object, and acquiring the foreign object such as a local area pixel height pole: three-dimensional coordinate information of characteristic points such as local area extremely high points, local area extremely low points and the like, pixel highest points, pixel lowest points and the like;
s3, judging whether the foreign object in the foreground depth image after registration is a cup: judging whether a pixel point subset with the following characteristics exists in the full set of all pixel points of the foreign object in the foreground depth image after registration: 1) the pixel point subset comprises the highest pixel point or the highest pixel point of a local area in the full set, 2) the projection points of the pixel points in the subset on an xy plane are distributed on a closed ring surrounded by a narrow band, the width of the narrow band is dW, and 3) the pixel point full set does not contain the height value of the pixel points of the subset, the projection points on the xy plane of which fall in the central area of the closed ring, on the z coordinate axis of a three-dimensional coordinate system, which is far smaller than the height value of the pixel points in the subset on the z coordinate axis of the three-dimensional coordinate system;
the pixel points in the subset comprise singular points which are compensated by means of mathematical fitting by means of noise elimination, smoothing processing and the like by using an effective algorithm in the process of extracting the registered foreground depth image, and the proportion of the number of the pixel points of the singular points in the subset is not more than a certain threshold value;
if the registered depth image of the foreign object meets the characteristics, determining the foreign object in the extracted registered foreground depth image as a cup with a complete cup rim and a certain depth;
the control and information storage/processing unit divides and extracts the cup rim and the depth image of the internal area of the cup from the registered depth image of the cup by using an effective algorithm to be used as a monitoring area;
s4, judging whether the rim of the cup mouth is aligned with the water outlet of the water dispenser: according to the established position parameters of the background model after the registration of the target area and the water outlet in the three-dimensional coordinate system, the control and information storage/processing unit judges whether the projection of the water outlet on the xy plane is in the closed ring defined by the projection of the cup opening along the corresponding pixel point on the xy plane, and the nearest distance from the edge of the closed ring exceeds a certain preset value L;
further, the control and information storage/processing unit obtains the aligning degree of the water outlet and the cup mouth along the central area through analysis and calculation according to the known coordinate information, and according to the aligning degree, the water outlet of the water dispenser is closed at different specific water level heights, and a specific amount of water is injected into the cup;
s5, injecting water into the cup by the water dispenser: through the steps, the control and information storage/processing unit determines that the projection of the water outlet on the xy plane is indeed located in the closed ring formed by the projections of the pixel points corresponding to the cup opening edge on the xy plane, and the nearest distance from the edge of the closed ring exceeds a certain preset value L, namely the water outlet of the water dispenser is indeed aligned with the opening edge of the cup, and the control and information storage/processing unit sends an instruction: opening water flow at a water outlet of the water dispenser and injecting water into the cup;
s6, detecting the height of the water level injected into the cup in real time: in the process of injecting water into a cup by a water dispenser, a control and information storage/processing unit controls an RGB-D camera to synchronously obtain a color image and a depth image of a monitoring area in real time at a speed of a certain number of frames per second, the color image and the depth image of the monitoring area are processed and registered in real time, the height value of a pixel point in the registered depth image corresponding to the internal point of a closed ring on an xy plane on a three-dimensional coordinate system z coordinate axis is analyzed and calculated, and the pixel point in the registered depth image corresponding to the internal point of the closed ring is the pixel point on the water surface of the cup in real time, so that the real-time height value of the water level plane in the cup is;
the process of processing, registering, analyzing and calculating the color image and the depth image of the water level plane in the cup by the control and information storage/processing unit comprises the following steps: noise caused by water flow flowing down from a water outlet of the water dispenser is filtered through an effective algorithm, and interference factors such as water splash caused by water flow falling and splashing and noise caused by water vapor are eliminated;
s7, completing the water taking process by using a cup: when the control and information storage/processing unit processes, registers, analyzes and calculates the water level in the cup to reach the preset specific water level height by synchronously and real-timely processing and registering the color image and the depth image of the monitoring area obtained by the RGB-D camera, the control and information storage/processing unit sends an instruction: closing the water outlet of the water dispenser, stopping injecting water into the cup, and stopping shooting the color image and the depth image;
s8, updating the background model after the registration of the target area: after the water taker removes the cup from the target area, no object or cup enters the target area again within a certain time, the control and information storage/processing unit starts the RGB-D camera to synchronously obtain a color image and a depth image of the multi-frame target area, an updated post-registration background model is established again for the target area according to the step S1, and the newly established post-registration background model is used for replacing the original background model stored in the control and information storage/processing unit to serve as a step S1 of taking water next time: and establishing a background model after the registration of the target area.
Further, in the above intelligent control method for water outlet of a water dispenser based on an RGB-D camera, the control and information storage/processing unit determines to turn on the auxiliary lighting unit according to a preset lighting condition of the target area, so as to improve the lighting condition of the target area and improve the image quality of the captured color image and depth image, for example: sharpness and detail resolution, etc.; the steps S1-S8 are similarly performed after the auxiliary lighting unit is turned on, wherein the control and information storage/processing unit issues an instruction to turn off the auxiliary lighting unit after the step S7 is completed.
Has the advantages that: the intelligent water dispenser realizes the intellectualization of the water taking process of the water dispenser, automatically discharges water according to different water taking quantity requirements of users, and prevents water overflow; meanwhile, the water is prevented from being discharged by mistake due to improper operation.
The invention uses the RGB-D camera, can ensure to obtain the depth image with high resolution, and is beneficial to the identification of the depth image.
Drawings
Fig. 1 is a schematic block diagram of a water dispenser intelligent control device based on an RGB-D camera without an auxiliary lighting unit.
FIG. 2 is a schematic diagram of a water dispenser structure of the water dispenser intelligent control device based on the RGB-D camera.
Fig. 3 is a schematic diagram of a module with an auxiliary lighting unit in the intelligent control device of the water dispenser based on the RGB-D camera.
FIG. 4 is a step diagram of the intelligent control method of the water dispenser based on the RGB-D camera.
FIG. 5 is a flow chart of the intelligent control method of the water dispenser based on the RGB-D camera without an auxiliary lighting unit.
Detailed Description
An intelligent control device of a water dispenser based on an RGB-D camera, the intelligent control device 1 of the water dispenser based on the RGB-D camera comprises: an RGB-D camera 2, a control and information storage/processing unit 3 and a power supply 4; the output end of the power supply 4 is connected with the power supply input end of the control and information storage/processing unit 3 and is used for directly or indirectly providing working power supply for the control and information storage/processing unit 3 and the RGB-D camera 2; the control and information storage/processing unit 3 is electrically connected to the RGB-D camera 2, as shown in fig. 1.
In the above intelligent control device for a water dispenser based on a depth camera, the RGB-D camera includes a color camera and a depth camera: the color camera shoots RGB color images, and the depth camera shoots depth images; the working principle of the depth camera is one of the following three principles: I) structured light measurement principle, II) binocular parallax measurement principle, III) TOF flight time measurement principle.
According to the intelligent control device of the water dispenser based on the RGB-D camera, the area which is possibly formed by the cup when a user takes water is a target area of the color image and the depth image shot by the RGB-D camera 2 below the water outlet of the water dispenser; the target area comprises a plurality of mark points with fixed relative positions with the water dispenser, which are used as three-dimensional coordinate system mark points and used for modeling the background of the target area through the obtained color image and depth image; the RGB-D camera 2 is fixedly arranged on the upper water dispenser body of the target area and close to the water dispenser water outlet, the position of the RGB-D camera 2 enables the RGB-D camera 2 to obtain a color image and a depth image of the whole target area, water is not prevented from being injected into a cup by the water dispenser, and the position of the RGB-D camera 2 relative to the water dispenser is fixed.
FIG. 2 is a schematic diagram of a water dispenser related to the RGB-D camera-based intelligent control device of the water dispenser; wherein: ABCD is a plane on which a base cup of a water taking area of the water dispenser is placed, P is a water outlet, o-xyz is a three-dimensional space coordinate system of the water dispenser, and ABCD is on an xy plane.
The intelligent control device of the water dispenser based on the RGB-D camera further comprises an auxiliary lighting unit 5, and as shown in figure 3, a power supply 4 provides a working power supply for the auxiliary lighting unit; the auxiliary lighting unit 5 is fixedly arranged on the upper water dispenser body of the target area, can irradiate the target area and does not obstruct the place where the water dispenser injects water into the cup.
The intelligent control method of the water dispenser based on the RGB-D camera comprises the following steps as shown in figure 4:
s1, establishing a background model after registration of a target area: the target area has no foreign object, the control and information storage/processing unit synchronously obtains a background color image and a depth image of the target area through the RGB-D camera and processes the background color image and the depth image, the images are converted into a background point cloud image, the color image and the depth image are registered, and the corresponding relation between the color image information and the depth image depth information is established, so that a pixel position and a depth image transformation model of the color image are obtained; the control and information storage/processing unit obtains a pixel position and a depth transformation model according to registration, and the registration method comprises the following steps: adopting registration based on feature point matching, wherein feature points of a color image and a depth image synchronously obtained by an RGB-D camera are detected by using a scale invariant feature transform algorithm (SIFT) or other related feature point algorithms, and the feature points of the images are matched; removing the wrong matching point pairs of the images by using a random consensus algorithm (RANSAC) or other algorithms to obtain correctly matched characteristic point pairs; obtaining the corresponding relation between the correctly matched feature point pairs according to the position and the depth information of the correctly matched feature point pairs, and calculating the pixel position and the depth transformation model between the color image and the depth image so as to obtain a registered depth image with higher resolution;
according to the known three-dimensional coordinate information of the positions of the mark points and the water outlet and the parameters of the RGB-D camera, establishing a post-registration background model of the target area, and storing the post-registration background model by the control and information storage/processing unit;
s2, extracting a foreground depth image after target area registration: when a foreign object is placed in a target area and no water is discharged from a water outlet of the water dispenser, the control and information storage/processing unit synchronously obtains a color image and a depth image of the target area in real time through an RGB-D camera, performs noise threshold processing (such as a median filtering method) and smoothing processing on the color image and the depth image, performs registration in the step S1 to realize foreground and background segmentation, selects and uses effective algorithms (such as watershed, seed search, background subtraction, binarization algorithm and the like) to extract the contour of the foreground, namely the contour of the foreign object in the target area, selects effective algorithms (such as ORB, SIFT, FAST, MSER, STAR and the like) to extract the feature points of the foreign object, and obtains the foreign object such as local area pixel height poles (local area extremely high points, local area extremely low points and the like), three-dimensional coordinate information of characteristic points such as a pixel highest point, a pixel lowest point and the like;
s3, judging whether the foreign object in the foreground depth image after registration is a cup: judging whether a pixel point subset with the following characteristics exists in the full set of all pixel points of the foreign object in the foreground depth image after registration: 1) the pixel point subset includes the highest pixel point in the full set or the local area pixel extremely high point, 2) the projection points of the pixel points in the subset on the xy plane are distributed on a closed ring surrounded by a narrow band, the width of the narrow band is dW, for example: dW =2mm, 3) the full set of pixel points does not include those pixel points in the subset whose projection points on the xy plane fall in the central region of the closed loop, whose height values on the z coordinate axis of the three-dimensional coordinate system are much smaller than those of the pixel points in the subset, for example: the difference between the height of the pixel points in the central area of the closed ring on the z coordinate axis of the three-dimensional coordinate system and the height of the pixel points in the subset on the z coordinate axis of the three-dimensional coordinate system is larger than 20 mm;
the pixel points in the subset include singular points which are compensated by means of mathematical fitting by means of noise elimination, smoothing processing and the like by using an effective algorithm in the process of extracting the registered foreground depth image, and the proportion of the number of the pixel points of the singular points in the subset does not exceed a certain threshold, such as: less than 5%;
if the registered depth image of the foreign object meets the characteristics, determining the foreign object in the extracted registered foreground depth image as a cup with a complete cup rim and a certain depth;
further, the control and information storage/processing unit 3 divides and extracts the cup rim and its internal region depth image as a monitoring region from the registered depth image of the cup by using an effective algorithm;
s4, judging whether the rim of the cup mouth is aligned with the water outlet of the water dispenser: according to the established position parameters of the registered background model of the target area and the water outlet in the three-dimensional coordinate system, the control and information storage/processing unit 3 judges whether the projection of the water outlet on the xy plane falls in the closed ring defined by the projection of the cup opening along the corresponding pixel point on the xy plane, and the nearest distance from the edge of the closed ring exceeds a certain preset value L, for example: l =5 mm;
further, the control and information storage/processing unit 3 obtains the degree of alignment of the water outlet and the cup mouth along the central area through analysis and calculation according to the known coordinate information, and according to the degree of alignment, the water outlet of the water dispenser is closed at different specific water level heights, and a specific amount of water is injected into the cup;
for example: if the water outlet is directly opposite to the central area of the rim of the cup opening, the difference between the height of the preset water level for injecting the cup and the height of the rim of the cup opening is 10 mm; if the projection of the water outlet on the xy plane is positioned in the middle position of the edge and the central area of the closed ring, the difference between the height of the preset water level for filling the cup and the height of the rim of the cup opening is 30 mm; by analogy, if the water outlet is far away from the central area of the rim of the cup mouth, the height of the water level injected into the cup is lower until the projection of the water outlet on the xy plane, and the nearest distance from a closed ring formed by the projection of the rim of the cup mouth on the xy plane is less than a certain set value L (assumed to be 5 mm), the water flow flowing out of the water outlet is considered not to fall into the cup safely and completely, and under the condition, the control and information storage/processing unit cannot open the water outlet to inject water into the cup;
s5, injecting water into the cup by the water dispenser: through the above steps, the control and information storage/processing unit 3 determines that the projection of the water outlet on the xy plane does fall inside the closed ring surrounded by the projection of the pixel point corresponding to the cup rim on the xy plane, and the distance closest to the edge of the closed ring exceeds a certain predetermined value L (for example: L =5 mm), that is, the water outlet of the water dispenser is indeed aligned with the rim of the cup, and the control and information storage/processing unit 3 sends an instruction: opening water flow at a water outlet of the water dispenser and injecting water into the cup;
s6, detecting the height of the water level injected into the cup in real time: in the process of filling water into the cup by the water dispenser, the control and information storage/processing unit 3 controls the RGB-D camera to synchronously obtain a color image and a depth image of the monitoring area in real time at a speed of a certain number of frames per second, processes and registers the color image and the depth image of the monitoring area in real time, analyzes and calculates the height value of pixel points in the registered depth image corresponding to the internal point of the closed ring on the xy plane on the coordinate axis z of the three-dimensional coordinate system, the pixel points in the registered depth image corresponding to the internal point of the closed ring are the pixel points on the real-time water surface of the cup, thereby obtaining the real-time height value of the water level plane in the cup,
the process of processing, registering, analyzing and calculating the color image and the depth image of the water level plane in the cup by the control and information storage/processing unit 3 comprises the following steps: noise caused by water flow flowing down from a water outlet of the water dispenser is filtered through an effective algorithm, and interference factors such as water splash caused by water flow falling and splashing and noise caused by water vapor are eliminated;
s7, completing the water taking process by using a cup: when the control and information storage/processing unit 3 processes, registers, analyzes and calculates the water level in the cup to reach a preset specific water level height by the synchronous real-time color image and depth image of the monitoring area obtained by the RGB-D camera, the control and information storage/processing unit 3 sends an instruction: closing the water outlet of the water dispenser, stopping injecting water into the cup, and stopping shooting the color image and the depth image;
s8, updating the background model after the registration of the target area: after the water taker removes the cup from the target area, no object or cup enters the target area again within a certain time, the control and information storage/processing unit 3 starts the RGB-D camera to obtain a color image and a depth image of the multi-frame target area synchronously, an updated post-registration background model is established again for the target area according to the step S1, and the newly established post-registration background model is used to replace the original background model stored in the control and information storage/processing unit 3, as a step S1 of taking water next time: and establishing a background model after the registration of the target area.
The implementation flow chart of the water taking process of the water dispenser is shown in fig. 5.
In the above intelligent control method for a drinking fountain based on RGB-D camera, the control and information storage/processing unit 3, according to different preset lighting conditions of the target area, for example: the auxiliary lighting unit 5 is judged to be turned on when the external light is too noisy and unstable, so as to improve the lighting condition of the target area and improve the quality of the obtained image, for example: sharpness and detail resolution, etc.; the steps S1-S8 are also performed after the auxiliary lighting unit 5 is turned on, wherein the control and information storage/processing unit 3 issues an instruction to turn off the auxiliary lighting unit 5 after the step S7 is completed.
Claims (4)
1. Water dispenser intelligent control device based on RGB-D camera, its characterized in that: the water dispenser water outlet intelligent control device (1) based on the RGB-D camera comprises: an RGB-D camera (2), a control and information storage/processing unit (3) and a power supply (4); the output end of the power supply (4) is connected with the power supply input end of the control and information storage/processing unit (3) and is used for directly or indirectly providing working power supply for the control and information storage/processing unit (3) and the RGB-D camera (2); the control and information storage/processing unit (3) is electrically connected with the RGB-D camera (2);
the RGB-D camera (2) is fixedly arranged on the water dispenser body and close to the water outlet of the water dispenser; the RGB-D camera (2) is used for setting a shot target area, and the target area comprises a plurality of fixed mark points positioned on the water dispenser and used as mark points for establishing a three-dimensional coordinate system; the RGB-D camera (2) is used for detecting the cup and detecting the height of the water level filled in the cup in real time;
among the RGB-D cameras are color cameras and depth cameras: the color camera shoots RGB color images, and the depth camera shoots depth images; the working principle of the depth camera is one of the following three principles: I) structured light measurement principle, II) binocular parallax measurement principle, III) TOF flight time measurement principle.
2. The intelligent control device of the water dispenser based on the RGB-D camera as claimed in claim 1, wherein: the auxiliary lighting unit (5) is also included, and the power supply (4) provides working power supply for the auxiliary lighting unit; the auxiliary lighting unit (5) is fixedly arranged on the water dispenser body at the upper part of the target area.
3. The intelligent control method of the water dispenser based on the RGB-D camera is characterized in that: the method comprises the following steps:
s1, establishing a background model after registration of a target area: the target area has no foreign object, the control and information storage/processing unit (3) synchronously obtains and processes a background color image and a depth image of the target area through the RGB-D camera (2), converts the images into a background point cloud image, registers the color image and the depth image, and establishes a corresponding relation between color image pixel points and depth information of the depth image so as to obtain a pixel position and depth image transformation model of the color image; the control and information storage/processing unit (3) obtains a pixel position and a depth transformation model according to registration, and the registration method comprises the following steps: adopting registration based on feature point matching, wherein feature points of a color image and a depth image synchronously obtained by an RGB-D camera are detected by using a scale invariant feature transform algorithm (SIFT) or other related feature point algorithms, and the feature points of the images are matched; removing the wrong matching point pairs of the images by using a random consensus algorithm (RANSAC) or other algorithms to obtain correctly matched characteristic point pairs; obtaining the corresponding relation between the correctly matched position and depth information of the feature point pair according to the position and the depth information of the correctly matched feature point pair, and calculating the pixel position and the depth transformation model between the two frames of images so as to obtain a registered depth image with higher resolution;
according to the known three-dimensional coordinate information of the positions of the mark points and the water outlets and the parameters of the RGB-D camera (2), establishing a background model after registration of the target area, and storing the background model after registration by the control and information storage/processing unit (3);
s2, extracting a foreground depth image after target area registration: placing a foreign object in a target area, under the condition that water does not exist at a water outlet of the water dispenser, synchronously acquiring a color image and a depth image of the target area in real time through the RGB-D camera (2) by the control and information storage/processing unit (3), performing noise threshold processing, smoothing processing and the like on the color image and the depth image, performing registration in the step S1, realizing foreground and background segmentation, selecting an effective algorithm to extract a foreground contour, namely the contour of the foreign object in the target area, selecting the effective algorithm to extract feature points of the foreign object, and acquiring the foreign object such as a local area pixel height pole: three-dimensional coordinate information of characteristic points such as local area extremely high points, local area extremely low points and the like, pixel highest points, pixel lowest points and the like;
s3, judging whether the foreign object in the foreground depth image after registration is a cup: judging whether a pixel point subset with the following characteristics exists in the full set of all pixel points of the foreign object in the foreground depth image after registration: 1) the pixel point subset comprises the highest pixel point or the highest pixel point of a local area in the full set, 2) the projection points of the pixel points in the subset on an xy plane are distributed on a closed ring surrounded by a narrow band, the width of the narrow band is dW, and 3) the pixel point full set does not contain the height value of the pixel points of the subset, the projection points on the xy plane of which fall in the central area of the closed ring, on the z coordinate axis of a three-dimensional coordinate system, which is far smaller than the height value of the pixel points in the subset on the z coordinate axis of the three-dimensional coordinate system;
the pixel points in the subset comprise singular points which are compensated by means of mathematical fitting by means of noise elimination, smoothing processing and the like by using an effective algorithm in the process of extracting the registered foreground depth image, and the proportion of the number of the pixel points of the singular points in the subset is not more than a certain threshold value;
if the registered depth image of the foreign object meets the characteristics, determining the foreign object in the extracted registered foreground depth image as a cup with a complete cup rim and a certain depth;
the control and information storage/processing unit (3) divides and extracts the cup rim and the depth image of the internal area of the cup from the registered depth image of the cup by using an effective algorithm to be used as a monitoring area;
s4, judging whether the rim of the cup mouth is aligned with the water outlet of the water dispenser: according to the established position parameters of the background model after the registration of the target area and the water outlet in the three-dimensional coordinate system, the control and information storage/processing unit (3) judges whether the projection of the water outlet on the xy plane falls into a closed ring formed by the projections of the pixel points corresponding to the cup opening on the xy plane, and the nearest distance from the edge of the closed ring exceeds a certain preset value L;
further, the control and information storage/processing unit (3) obtains the aligning degree of the water outlet and the cup mouth along the central area through analysis and calculation according to the known coordinate information, and according to the aligning degree, the water outlet of the water dispenser is closed at different specific water level heights, and a specific amount of water is injected into the cup;
s5, injecting water into the cup by the water dispenser: through the steps, the control and information storage/processing unit (3) determines that the projection of the water outlet on the xy plane does fall in the closed ring formed by the projections of the pixel points corresponding to the cup opening edge on the xy plane, the nearest distance from the edge of the closed ring exceeds a certain preset value L, namely the water outlet of the water dispenser is aligned with the opening edge of the cup, and the control and information storage/processing unit (3) sends an instruction: opening water flow at a water outlet of the water dispenser and injecting water into the cup;
s6, detecting the height of the water level injected into the cup in real time: in the process that a water dispenser injects water into a cup, a control and information storage/processing unit (3) controls an RGB-D camera (2) to synchronously obtain a color image and a depth image of a monitoring area in real time at the speed of a certain number of frames per second, the color image and the depth image of the monitoring area are processed and registered in real time, the height value of a pixel point in the registered depth image corresponding to the internal point of a closed ring on an xy plane on a three-dimensional coordinate system z coordinate axis is analyzed and calculated, and the pixel point in the registered depth image corresponding to the internal point of the closed ring is a real-time pixel point on the water surface of the cup, so that the real-time height value of the water level plane in the cup is obtained;
the process of processing, registering, analyzing and calculating the color image and the depth image of the water level plane in the cup by the control and information storage/processing unit (3) comprises the following steps: noise caused by water flow flowing down from a water outlet of the water dispenser is filtered through an effective algorithm, and interference factors such as water splash caused by water flow falling and splashing and noise caused by water vapor are eliminated;
s7, completing the water taking process by using a cup: when the control and information storage/processing unit (3) processes, registers, analyzes and calculates the water level in the cup to reach a preset specific water level height through the synchronous real-time color image and depth image of the monitoring area obtained by the RGB-D camera (2), the control and information storage/processing unit (3) sends an instruction: closing the water outlet of the water dispenser, stopping injecting water into the cup, and stopping shooting the color image and the depth image;
s8, updating the background model after the registration of the target area: after the water taker removes the cup from the target area, no object or cup enters the target area again within a certain time, the control and information storage/processing unit (3) starts the RGB-D camera (2) to synchronously obtain a color image and a depth image of the multi-frame target area, an updated registered background model is established again for the target area according to the step S1, and the newly established updated registered background model is used for replacing the original background model stored in the control and information storage/processing unit (3) and is used as the step S1 of taking water next time: and establishing a background model after the registration of the target area.
4. The intelligent control method of the water dispenser based on the RGB-D camera as claimed in claim 3, wherein: the control and information storage/processing unit (3) judges to turn on the auxiliary lighting unit (5) according to the preset lighting condition of the target area, so as to improve the lighting condition of the target area and improve the image quality such as the definition, detail resolution capability and the like of a shot color image and a shot depth image; after the auxiliary lighting unit (5) is turned on, the implementation steps include S1-S8, wherein after the step S7 is completed, the control and information storage/processing unit (3) issues an instruction to turn off the auxiliary lighting unit (5).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810306420.0A CN108742159B (en) | 2018-04-08 | 2018-04-08 | Intelligent control device and method for water dispenser based on RGB-D camera |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810306420.0A CN108742159B (en) | 2018-04-08 | 2018-04-08 | Intelligent control device and method for water dispenser based on RGB-D camera |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108742159A CN108742159A (en) | 2018-11-06 |
| CN108742159B true CN108742159B (en) | 2021-01-15 |
Family
ID=63981105
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810306420.0A Active CN108742159B (en) | 2018-04-08 | 2018-04-08 | Intelligent control device and method for water dispenser based on RGB-D camera |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108742159B (en) |
Families Citing this family (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109717749A (en) * | 2018-11-30 | 2019-05-07 | 青岛海尔智能技术研发有限公司 | A kind of Intelligent water injection device and method |
| CN111820753B (en) * | 2019-04-18 | 2022-12-27 | 佛山市美的清湖净水设备有限公司 | Water dispenser and control method and device thereof |
| CN110211093A (en) * | 2019-04-30 | 2019-09-06 | 上海工程技术大学 | A kind of water outlet control method for automatic drinking water apparatus |
| CN111568211A (en) * | 2020-02-28 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568212A (en) * | 2020-02-28 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568220A (en) * | 2020-02-28 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568196A (en) * | 2020-02-28 | 2020-08-25 | 佛山市云米电器科技有限公司 | Intelligent water outlet method, system and storage medium |
| CN111568235A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568237A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568239A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568232A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water outlet control method of water dispenser, water dispenser and computer readable storage medium |
| CN111568228A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water purifying and drinking machine control method, water purifying and drinking machine and readable storage medium |
| CN111568225A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568236A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568230A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water outlet control method of water dispenser, water dispenser and computer readable storage medium |
| CN111568244A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568238A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water outlet control method of water dispenser, water dispenser and computer readable storage medium |
| CN111568234A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water surface detection method, water dispenser and computer readable storage medium |
| CN111642975A (en) * | 2020-02-29 | 2020-09-11 | 佛山市云米电器科技有限公司 | Water outlet control method, water dispenser and computer readable storage medium |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105654481B (en) * | 2015-12-29 | 2019-03-08 | 深圳市未来媒体技术研究院 | The detection system of vascular exosmosis leakage situation based on depth camera |
| CN107872644B (en) * | 2016-09-23 | 2020-10-09 | 亿阳信通股份有限公司 | Video monitoring method and device |
| CN107240129A (en) * | 2017-05-10 | 2017-10-10 | 同济大学 | Object and indoor small scene based on RGB D camera datas recover and modeling method |
| CN107103311A (en) * | 2017-05-31 | 2017-08-29 | 西安工业大学 | A kind of recognition methods of continuous sign language and its device |
| CN107369158B (en) * | 2017-06-13 | 2020-11-13 | 南京邮电大学 | Indoor scene layout estimation and target area extraction method based on RGB-D image |
| CN107273866A (en) * | 2017-06-26 | 2017-10-20 | 国家电网公司 | A kind of human body abnormal behaviour recognition methods based on monitoring system |
| CN107456086A (en) * | 2017-08-25 | 2017-12-12 | 深圳市浮思特科技有限公司 | The water outlet Intelligent Measurement module and water outlet control method of view-based access control model detection |
| CN107578053B (en) * | 2017-09-25 | 2020-03-24 | 重庆虚拟实境科技有限公司 | Contour extraction method and device, computer device and readable storage medium |
| CN107730526A (en) * | 2017-09-25 | 2018-02-23 | 中国科学院声学研究所 | A kind of statistical method of the number of fish school |
-
2018
- 2018-04-08 CN CN201810306420.0A patent/CN108742159B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN108742159A (en) | 2018-11-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108742159B (en) | Intelligent control device and method for water dispenser based on RGB-D camera | |
| CN108427940B (en) | Intelligent control device and control method for water dispenser water outlet based on depth camera | |
| CN108921883B (en) | Water dispenser control device based on two-position depth image recognition and control method thereof | |
| CN107563347B (en) | Passenger flow counting method and device based on TOF camera | |
| CN103473542B (en) | Multi-clue fused target tracking method | |
| US9995578B2 (en) | Image depth perception device | |
| JP6577565B2 (en) | Method and apparatus for depth image processing | |
| CN102402686B (en) | A kind of registration number character dividing method based on connected domain analysis | |
| US9767568B2 (en) | Image processor, image processing method, and computer program | |
| CN106503707B (en) | Licence plate recognition method and device under a kind of infrared lighting condition | |
| CN104574285A (en) | Method for automatically removing image black eyes | |
| KR20160044316A (en) | Device and method for tracking people based depth information | |
| US20150302595A1 (en) | Method and apparatus for generating depth information | |
| CN110927174B (en) | White spirit segmentation detection device and method based on embedded module | |
| CN105160328B (en) | A kind of human body contour outline recognition methods based on binary image | |
| CN107025633B (en) | Image processing method and device | |
| CN105631868A (en) | Depth information extraction method based on image classification | |
| CN111568208A (en) | Water dispenser control method, water dispenser and computer readable storage medium | |
| CN111568244A (en) | Water dispenser control method, water dispenser and computer readable storage medium | |
| CN207636505U (en) | A kind of shoes mould surface damage detection device | |
| Kamboj et al. | Color Edge Detection in RGB Color Space Using Automatic Threshold Detection | |
| CN103366174B (en) | Method and system for obtaining image information | |
| CN111642975A (en) | Water outlet control method, water dispenser and computer readable storage medium | |
| CN109509245A (en) | A kind of style image generation method and device | |
| CN117770650A (en) | Improved water quantity control method and system for self-service equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20191211 Address after: 214031 No. 32 Wuai North Road, Liangxi District, Wuxi City, Jiangsu Province Applicant after: Xu Yu Address before: 311800 Jiefang Road 698, Dian Kou Town, Zhuji City, Shaoxing, Zhejiang Applicant before: Zhejiang an Jing Intelligent Technology Co., Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |