WO2018143550A1 - Appareil de notification de date d'expiration d'aliments stockés par une intelligence artificielle de lecture de caractères dans un réfrigérateur et procédé associé - Google Patents

Appareil de notification de date d'expiration d'aliments stockés par une intelligence artificielle de lecture de caractères dans un réfrigérateur et procédé associé Download PDF

Info

Publication number
WO2018143550A1
WO2018143550A1 PCT/KR2017/013434 KR2017013434W WO2018143550A1 WO 2018143550 A1 WO2018143550 A1 WO 2018143550A1 KR 2017013434 W KR2017013434 W KR 2017013434W WO 2018143550 A1 WO2018143550 A1 WO 2018143550A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
expiration date
box
food
refrigerator
Prior art date
Application number
PCT/KR2017/013434
Other languages
English (en)
Korean (ko)
Inventor
구영모
Original Assignee
휴먼식스 주식회사
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 휴먼식스 주식회사 filed Critical 휴먼식스 주식회사
Publication of WO2018143550A1 publication Critical patent/WO2018143550A1/fr

Links

Images

Classifications

    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
    • F25D29/00—Arrangement or mounting of control or safety devices
    • F25D29/005—Mounting of control devices
    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
    • F25D23/00—General constructional features
    • F25D23/02—Doors; Covers
    • F25D23/04—Doors; Covers with special compartments, e.g. butter conditioners
    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
    • F25D29/00—Arrangement or mounting of control or safety devices
    • F25D29/008—Alarm devices
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
    • G06Q10/087—Inventory or stock management, e.g. order filling, procurement or balancing against orders
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/254—Analysis of motion involving subtraction of images
    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
    • F25D2400/00—General features of, or devices for refrigerators, cold rooms, ice-boxes, or for cooling or freezing apparatus not covered by any other subclass
    • F25D2400/36—Visual displays
    • F25D2400/361—Interactive visual displays
    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
    • F25D2500/00—Problems to be solved
    • F25D2500/06—Stock management
    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
    • F25D2700/00—Means for sensing or measuring; Sensors therefor
    • F25D2700/02—Sensors detecting door opening
    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
    • F25D2700/00—Means for sensing or measuring; Sensors therefor
    • F25D2700/06—Sensors detecting the presence of a product
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00—Indexing scheme for image generation or computer graphics
    • G06T2210/12—Bounding box

Definitions

  • the present invention relates to an apparatus and a method for notifying the shelf life of food stored by a character reading artificial intelligence of a refrigerator to determine a shelf life of a food stored in a refrigerator and to inform a user of information about a food having an expiration date or that has passed.
  • a character reading artificial intelligence of a refrigerator to detect the expiration date through the character recognition from the image information photographed the area printed on the wrapping paper of the food stored in the refrigerator to inform the expiration date information to the user
  • the present invention relates to a shelf life notification device and a method thereof.
  • a refrigerator is partitioned into a plurality of storage compartments using a detachable shelf or the like so as to efficiently store food stored in the refrigerator according to the type or size.
  • household refrigerators have also become increasingly large in recent years, and in the refrigerators, foods to be stored for a long time or foods to be consumed within a few days are purchased and loaded in large quantities, and various types of foods are stored in each refrigerator and stored by the user. You can't remember all the information about the food, and if you're using more than one refrigerator, it's harder to remember.
  • Korean Patent Laid-Open Publication No. 10-2006-0111009 and Korean Patent Laid-Open Publication No. 10-2013-0030526 have installed a display device having a touch screen function on the outside of the refrigerator door to prevent this, without the user opening the refrigerator door and checking the inside. It is proposed to make it easy to check the food information, such as the type and quantity of food stored in the refrigerator, location information, expiration date, or to install a camera in the refrigerator to display the captured image of the stored food through the display device.
  • the image taken by the camera displays only the image information of the stored food, so that only the type of stored food can be checked, but the expiration date information has to be input by the user separately.
  • the present invention has been invented in view of the above-described circumstances, and the information on the food which has an expiration date or has passed due to the determination of the date of manufacture or the expiration date through image analysis from image signals photographing the food stored in each storage compartment of the refrigerator. It is a technical task of the present invention to provide an apparatus and a method for notifying the shelf life of food stored by a character reading artificial intelligence of a refrigerator to visually and visually notify a user.
  • the storage food shelf life notification device by the character reading artificial intelligence of the refrigerator according to an embodiment of the present invention
  • a plurality of cameras photographing food stored and installed in at least one of a left side, a right side, or an upper side of each storage compartment of the refrigerator to obtain a stored food image;
  • An expiration date extracting unit extracting an expiration date by analyzing the stored food image;
  • a database unit for storing the expiration date detected by the expiration date extraction unit together with the corresponding food information;
  • a control unit including the expiration date extracting unit, controlling the camera to obtain a stored food image, and then detecting the expiration date and storing the expiration date in a database unit;
  • a touch screen installed on an outer surface of the refrigerator door to display food information and an expiration date stored in the database unit to the user.
  • a door sensor for detecting the opening of the refrigerator door and a speaker for outputting an alarm sound are installed, and when the control unit detects the opening of the refrigerator door through the door sensor, the database unit is searched for information on foods that have an expiration date or have passed. It is characterized in that it is made to inform the user by outputting or displaying on the touch screen.
  • the control unit is further provided with a wired communication unit and a wireless communication unit connected to the Internet network, it characterized in that it is possible to transmit information about the expiration date of the stored food in the refrigerator to a user connected from a remote place.
  • the touch screen may be configured to modify food information stored in a database according to user input information.
  • the bounding box having less than similarity is less than and deleted from the loading list of the database, and if it is above a preset threshold or there is no pre-stored bounding box, it is determined to be loading and added to the loading list of the database. do.
  • the determining of the presence or absence of the motion may include determining an average of pixel difference values of red, green, and blue of the current image frame and the image frame immediately before the movement, and determining that there is motion when the average of the pixel difference values exceeds a preset threshold. do.
  • the two-dimensional bounding box is extracted from each opacity image obtained from the images captured by the upper, right, and left cameras, and the coordinate values of each corner are temporarily stored in the memory, and the two-dimensional bounding box is stored in the memory. It is characterized by converting to a three-dimensional bounding box by calculating in a union method.
  • the coordinates protruding to the left of the images taken by the upper, right, and left cameras are set to the x coordinate of the 3D bounding box and to the rightmost.
  • Set the extruded coordinates to the x2 coordinates of the three-dimensional bounding box set the highest projected coordinates to the y-coordinates of the three-dimensional bounding box, and set the lowest projected coordinates to the y2 coordinates of the three-dimensional bounding box.
  • the coordinates protruding toward the front are set to the z coordinates of the three-dimensional bounding box, and the coordinates protruding toward the rear to the z2 coordinates of the three-dimensional bounding box.
  • Dissimilarity in the step of determining whether the loading / unloading is characterized in that the more vertices converge to 0 by summing the distance of each vertex for the three-dimensional bounding box is determined to be similar.
  • the loading / unloading step is in the loading step, performing a single box extraction step of extracting areas estimated as a text printing area into a rectangular one by using a contour extraction algorithm from 'color + alpha image', and then It characterized in that to carry out the expiration date extraction step by analyzing the printing portion in the single box.
  • those whose width, height, or area of the single box are less than or equal to or above the predetermined lower threshold point or exceed the upper threshold point are determined as noise objects and excluded.
  • the noise objects are excluded based on the distance between the center points and the adjacent single boxes are grouped together below the preset threshold for the remaining single boxes to form a group box.
  • the box is characterized in that the noise object is excluded.
  • the direction of the group box is the horizontal type or the vertical type, and the order number is specified in order for the single box in each group box according to the corresponding directionality, and then from the first number box. Extracts the through line connecting the center point of each single box to the last single box in a straight line, calculates an average value of the direction angles of the extracted through lines, and compares the average value with a direction angle difference more than a preset threshold point.
  • the box is judged as a noisy object and excluded, and the group boxes having fewer than four number of boxes are characterized as performing a directional check to determine as a noisy object.
  • the expiration date extraction step is characterized by extracting the expiration date by determining whether the characters of the group box read is matched with a preset 8-digit year-date pattern, 6-digit year-date pattern, or 4-digit month-day pattern .
  • the date and date extracted from the group box are regarded as the expiration date of the loaded food. If two or more are elected, the latest date is considered as the expiration date and is stored in a database.
  • the date is regarded as the expiration date of the loaded food, and when two or more are selected, the maximum difference between the extracted dates is 6 months or more.
  • the date with the smallest date is considered as the expiration date, and the year of expiration date is considered to be the year + 1, while the maximum difference between the dates extracted from two or more group boxes is less than 6 months.
  • the maximum expiration date is regarded as the expiration date, and the year of expiration date is regarded as the current year, and is stored as a expiration date in a database.
  • a notification message for notifying eating or discarding of the corresponding food is output.
  • the control unit transmits a food image using a text message or 'color + alpha image' when receiving a request for transmission of food information stored in a database from an external terminal connected through a wired communication unit or a wireless communication unit.
  • the external terminal is connected to the control unit through an app, characterized in that for requesting the transmission of food information.
  • the apparatus for notifying the shelf life of food stored by the character reading artificial intelligence of the refrigerator of the present invention it is possible to determine the expiration date of the food stored in the refrigerator without the user's manual operation and inform the user. It is possible to prevent the deterioration due to the expiration date of the food and to check the expiration date without inputting the expiration date, which is convenient for use, and the expiration date of the food stored in the refrigerator can be checked through the internet network even in remote places. There is an effect that can prevent the purchase.
  • FIG. 1 is a block diagram showing a storage food shelf life notification device by the character reading artificial intelligence of the refrigerator according to an embodiment of the present invention.
  • FIG. 2 is a view showing a touch screen installation state of the refrigerator according to the present invention.
  • Figure 3 is a schematic view of the camera installation state of each compartment of the refrigerator according to the present invention.
  • FIG. 4 is a flow chart illustrating a method for notifying storage of food shelf life of an artificial intelligence refrigerator according to the present invention
  • FIG. 7 is a view for explaining an operation of extracting a changed image region in the present invention.
  • FIG. 8 is a view for explaining an operation of extracting an opacity image region in the present invention.
  • FIG. 11 is a view for explaining the operation of extracting a single box in the present invention.
  • FIG. 12 is a view illustrating an image in a state of excluding an image determined to be a noise object by comparing the width, height, and area of the single box in FIG. 11;
  • FIG. 13 is a view illustrating an image in a state in which an image having a non-adjacent center point distance is determined as a noise object in FIG. 12 and excluded;
  • FIG. 14 is a view for explaining an operation of extracting a group box from FIG. 13;
  • FIG. 15 is a view illustrating an image in a state in which a group box including less than four single boxes included in a group box is excluded in FIG. 14;
  • FIG. 15 is a view illustrating an image in a state in which a group box including less than four single boxes included in a group box is excluded in FIG. 14;
  • FIG. 16A shows an example of determining the order of the single box in the group box
  • Fig. 16B shows an example of extracting the penetration line in the group box
  • Fig. 16C shows the critical point of the through line angle.
  • FIG. 16 (d) is a view for explaining an operation of extracting a horizontally divided single sheet image based on a penetrating line angle;
  • 17 is a view illustrating an operation of displaying the food in the refrigerator so as to check the food through the smart phone app according to the present invention.
  • FIG. 1 to 3 shows a block diagram of a storage food shelf life notification device by the character reading artificial intelligence of the refrigerator according to an embodiment of the present invention.
  • the storage food shelf life notification device by the character reading artificial intelligence of the refrigerator according to the present invention, as shown in Figure 3 by taking a picture of the inside of the storage box on the front, rear, left, right, upper surface of each storage compartment of the refrigerator image
  • a plurality of cameras 20a to 20n for photographing are installed.
  • the front surface of the storage compartment may be a support disposed on the inner surface of the door or separately in the storage compartment.
  • the camera photographing the inside of the storage compartment may be installed on the left side, the right side, and the upper side of the refrigerator.
  • the expiration date extraction unit 30 is provided on the output side of the cameras 20a to 20n, and analyzes the stored food image acquired by the cameras 20a to 20n under the control of the controller 10 to determine the expiration date of the food. It can be extracted.
  • the expiration date extraction unit 30 may be provided separately, or may be provided as a software module in the control unit 10.
  • a new ID is issued to issue a food with an expiration date for the food. It is designed to store information.
  • the controller 10 controls the cameras 20a to 20n to obtain a stored food image, detects an expiration date through the expiration date extraction unit 30 provided therein, and stores a series of information together with the corresponding food information. It serves to control to store in the unit (40).
  • the touch screen unit 50 may be installed on the outer surface of the refrigerator door so that the user can easily check by displaying the food information and the expiration date stored in the database unit 40 as shown in Figure 2, the user Is made to be able to modify or add food information stored in the database unit 40 through a touch input.
  • control unit 10 is provided with a door sensor 60 for detecting the opening of the refrigerator door and a speaker 70 for outputting an alarm sound.
  • the door sensor 60 detects the opening of the refrigerator door and transmits it to the control unit 10 so that the control unit 10 informs the user of the state information of the stored food or operates the cameras 20a to 20n to load the food into each storage compartment.
  • the control unit 10 searches the database unit 40 so that the expiration date is imminent or has passed. This function informs the user by outputting information about the voice signal.
  • the information on the food due date or when the expiration date is imminent may be displayed through the touch screen unit 50.
  • control unit 10 may be connected to the wired communication unit 80 and the wireless communication unit 90 is connected to the Internet network.
  • the wired communication unit 80 and the wireless communication unit 90 serves to transmit information on the food stored in the refrigerator and the expiration date according to a request of a user connected using a wired or wireless terminal at a remote location.
  • control unit 10 determines whether the refrigerator door is opened by judging the signal received from the door sensor 60, and if it is determined that the refrigerator door is opened, the controller 10 is loaded / loaded to detect whether food is newly loaded or unloaded. Initiate the unload detection mode.
  • the cameras 20a to 20n installed in the respective compartments of the refrigerator are operated to acquire an image before each movement of the compartments, and then the new frame is continuously photographed to move the motion.
  • a step of determining whether or not there is a motion is performed.
  • the frame delay time is delayed, and the new frame is obtained by comparing the change in pixel value between the acquired frames.
  • the process of capturing a new frame is repeated and the image is stored after the last new frame, ie, the end of the motion, which does not change the pixel value.
  • the pixel difference value comparison for determining the motion is as follows.
  • Avg Difference exceeds the threshold, it is regarded as a motion detection state.
  • Sum (Difference) represents the total pixel difference value
  • R represents the red pixel value
  • G represents the green pixel value
  • B represents the blue pixel value
  • After.R represents the end of the motion.
  • the red pixel value of the image, Before.R is the red pixel value of the image just before the movement
  • After.G is the green pixel value of the image after the movement is finished
  • Before.G is the green pixel value of the image immediately before the movement
  • After.B is the end of the movement.
  • the blue pixel value of the image, Before.B represents the blue pixel value of the image immediately before the movement.
  • FIG. 6A illustrates an example of an image immediately before a movement in which a first food 101 is stored in a storage compartment in an initial state when a refrigerator door is opened and there is no movement
  • FIG. 6C shows the frame between the frames after the second food 102 is stored in the storage compartment.
  • the following shows an example of the image after the end of the movement when it is determined that there is no movement during the shooting delay time.
  • the food compartment was loaded only on the left side just before the movement, but as shown in (c) of FIG. 6, another food was additionally added to the right side of the storage compartment. It shows the stored state.
  • Such movement determination step is performed for all the frames obtained by the camera of each storage box.
  • FIG. 7 is a view for explaining an operation of extracting a changed image region, in which FIG. 7A shows an example of an image immediately before a movement, and FIG. 7B shows an example of an image after an end of a movement. (c) shows the difference value image.
  • the control unit 10 controls each of the images immediately before the movement B (see FIG. 7A) and the images C after the movement (see FIG. 7B) obtained from all the cameras 20a to 20n.
  • the difference value image D (see FIG. 7C) is extracted by applying the following pixel difference value formula to the pixel. If the image B just before the movement does not exist, the empty storage box image A taken while the refrigerator is empty is replaced.
  • After.R is the red pixel value of the image after the movement
  • Before.R is the red pixel value of the image immediately before the movement
  • After.G is the green pixel value of the image after the movement
  • Before.G is the green pixel value of the image immediately before the movement.
  • After.B represents a blue pixel value of the image after the end of the movement
  • Before.B represents a blue pixel value of the image immediately before the movement.
  • an opacity image F which is a black and white dualization image having no intermediate brightness, from the difference value image D (see FIG. 7C) between the image immediately before the movement and the image after the movement ends (FIG. 8B). Step).
  • the controller 10 complements the difference value image D extracted in the step of extracting the changed image region to generate an opacity image F including complete black and white without intermediate brightness. This process is performed for the images taken by each camera.
  • the controller 10 selects the opacity image F as an alpha channel and selects a 'color + alpha image' G (see FIG. 9C) to select an image extracted as the color image channel as the color channel. Create and store in temporary memory.
  • FIG. 9A shows an image after the end of the movement
  • FIG. 9B shows an opacity image
  • FIG. 9C shows a 'color + alpha image' (G).
  • a two-dimensional bounding box extraction step is performed to extract a three-dimensional new bounding box as shown in FIG. 10 from the opacity image generated in the opacity image generation step according to FIG. 8.
  • FIG. 10 is a diagram illustrating a step of extracting a 2D bounding box and extracts a 2D bounding box from each opacity image obtained from an image photographed by a camera installed on the upper, right, and left sides of each storage box.
  • the two-dimensional bounding box H (see (b) of FIG. 10) is extracted from the opacity image F, and the coordinate values (x, x2, y, y2) is temporarily stored in the memory, and this process is performed for images captured by each camera.
  • the controller 10 converts the two-dimensional bounding box H calculated from the images photographed by two or more cameras belonging to the same storage box into a three-dimensional bounding box.
  • the inconsistency area between each two-dimensional bounding box is a union (U) method. This is to recover as much as possible the areas that may have been obscured by other objects.
  • U union
  • the uppermost coordinate is selected and used as the y-coordinate of the three-dimensional bounding box, and the lowermost coordinate is selected.
  • the y2 coordinate of the 3D bounding box is selected among the two-dimensional bounding boxes extracted from the images taken by the right and left two cameras.
  • the largest volume is selected as the effective three-dimensional bounding box and temporarily stored in the memory.
  • a storage compartment for storing the effective three-dimensional bounding box is temporarily determined as a storage compartment in which a loading or a loading occurs.
  • the control unit 10 determines whether the effective three-dimensional bounding box is generated by loading food in the storage compartment or by loading. If the volume is below the critical point, the load / unload change is regarded as not occurring, and the process returns to the initial load / unload detection mode.
  • the new effective three-dimensional bounding box is compared with the three-dimensional bounding box of the stored items stored in the existing database 40 to determine whether it is loaded or unloaded, and is not included in the loaded item of the database 40. If it is determined that it is considered to be loaded, the expiration date extraction step described below is immediately performed.
  • the dissimilarity with the new effective 3D bounding box is applied to each of the 3D bounding boxes included in the load item of the database 40. To judge.
  • the dissimilarity is numerically determined by comparing the position and volume of two three-dimensional bounding boxes by examining the sum of the distances of two vertices with respect to the two three-dimensional bounding boxes to be compared.
  • the formula representing is as follows.
  • length (H.xyz-OH.xyz) represents the distance between the first vertices and length (H.xyz2-OH.xyz2) represents the distance between the second vertices.
  • the controller 10 calculates dissimilarity values for each of the stored items stored in the database 40 in the same manner as described above, and then extracts the smallest dissimilarity value that is the smallest dissimilarity value.
  • the loading item for the corresponding object among the loading items in the database 40 eg, 'color + alpha image', 3D bounding box, loading date and time. Deleted from the database 40, and output to the voice through the speaker 70 or displayed on the touch screen 50 to notify the user, and returns to the initial loading / unloading detection mode.
  • G is the color + alpha image
  • A is the empty compartment image
  • Red is the red pixel value
  • Green is the green pixel value
  • Blue is the blue pixel value
  • is the color +
  • G.Green-A.Green is the absolute value of the green pixel value difference between the' color + alpha image 'and the empty compartment image
  • the difference is less than the critical point, it is found as a loading while comparing the images taken by each camera, and when loading is not found, it is confirmed as loading and the loading information of the object (for example, 'color + alpha image'). , 3D bounding box, loading date and time) is stored in the database 40 and the expiration date extraction step described below is performed.
  • the controller 10 extracts a plurality of regions estimated as a text printing portion from a 'color + alpha image' of a newly loaded food, in the form of a rectangle.
  • the region extracted in the form of a rectangle is referred to as a 'single box'.
  • the single box is indicated by a green rectangle.
  • the widths below the predetermined lower limit point or exceeding the upper limit point are excluded. This is to exclude the 'noise individual' which is unlikely to correspond to the expiration date label among the detected box (BX1 ⁇ BXn).
  • the heights below the predetermined lower limit point or above the upper limit point are also excluded. This is to exclude 'noise individuals', which are unlikely to correspond to the expiration date label among the detected box (BX1 ⁇ BXn).
  • FIG. 12 illustrates an image in which the width, height, and area of the single box extracted from the single box BX1 to BXn extracted as described above are excluded from the image determined as a “noise object” beyond a predetermined threshold point. .
  • the critical point of the center point distance is set in advance. This is also to exclude the 'noise individual' which is unlikely to correspond to the expiration date label of the detected areas (see Fig. 13).
  • a group box of a union that surrounds objects adjacent to each other below a critical point is generated.
  • a group box contains data structure information about the single box it contains.
  • the size of the single box and the quantity of the single box included in the group box are judged to remove the single box and the group box determined to be 'noise object', and for each remaining group box, each group box within each group box.
  • the order is determined from the smaller x-coordinate value of each single box, and if the vertical of the group box is larger than the horizontal, the y of the single box
  • the sequence number may be determined from the smaller the coordinate value.
  • FIG. 16A shows an example of determining the order number of the single boxes in a group box.
  • the horizontal group box consisting of eight single boxes of 2,0,1,6,1,1,1,0, x-coordinate values
  • the small numbers are assigned in order from 1 to 8 in order from left to right.
  • a directionality inspection step of searching for and removing the word box that violates the direction of the letter is performed.
  • This is a step of calculating the average value of the direction angles of all the through lines in one group box, and then excluding the single box having the difference between the through line having the average value and the direction angle more than a predetermined threshold. Accordingly, when the exclusion of some single box occurs, the step of checking the directivity is repeated after performing the through-line extraction step.
  • FIG. 16 shows the relationship between the through line angle and the rotation angle, and shows an example of the original image before the reverse rotation and the horizontally cut single image after the reverse rotation.
  • the character is read out using the OCR (Optical Character Recognition) algorithm in the forward order with respect to the single image leveled on the basis of the penetrating line angle. It also rotates all horizontally flattened images 180 degrees and reads the characters again using the OCR algorithm in reverse order.
  • OCR Optical Character Recognition
  • the OCR reading result in the forward or 180-degree rotation of the group box selects a group box in which the whole or the middle of the sentence conforms to one of the following year and month patterns, for example, having four digits of a year.
  • YY is a letter representing the second two digits of the year, and is valid only when the number is between 16 and 99, for example, to determine whether the expiration date has passed or the expiration date has not yet arrived.
  • DD is the character representing the two digits of the day, for example valid only for numbers 1 or 01 to 31. .
  • the date and month extracted from the group box are regarded as the expiration date of the loaded food, and when two or more are selected, The latest date is considered expiration date. However, if there is no elected group box, it is re-inspected, for example, to select a group box that matches one of the following year, month, and year patterns.
  • YY is the year
  • MM is the month
  • DD is the letter representing the day
  • the effective processing method is the same as in the case where the year is 4 digits.
  • the date and month extracted from the group box are regarded as the expiration date of the loaded food.
  • the late date is regarded as the expiration date.
  • a group box matching one of the following month day patterns displaying only the month day without a year is selected.
  • MM-DD MM-DD
  • MM.DD MM / DD
  • MM DD MMDD
  • MM is the month and DD is the letter representing day
  • the effective processing method is the same as that of the year, month, and day pattern inspection example.
  • the load is regarded as a short-term food distribution and the following process is performed.
  • the date is regarded as the expiration date of the loaded food.
  • the current time is December and the expiration date is January
  • the year of the expiration date is stored in the database 40 as the year +1, otherwise it is regarded as the year.
  • the group box on the right is regarded as information of time and time (HHMM) and excluded. If the group image inspection of the group box has a 180-degree rotation history, exclude the group box on the left side instead of the right side. Finally, if there is only one group box remaining, the same process as in the case of the pattern matching the above MMDD pattern is performed.
  • the date with the smallest value is considered to be the expiration date. This is because it is often displayed.
  • the year of the expiration date is regarded as the year + 1 and stored in the database 40. For example, if the date extracted from two or more group boxes is 12/29 (December 29) and 01/05 (January 5), 01/05 is considered expiration date, and 01/05 is 12/29 It shall be regarded as the date of the year following the year following this year.
  • the date with the largest value is considered as the shelf life.
  • the year of expiration date shall be regarded as the year concerned and remembered.
  • the expiration date is increased, for example, it can be converted into months for convenience and can be reproduced as "Recommended to eat within 12 months”. If the expiration date is same day, it can be reproduced as "Recommended to eat today”. If a warning is issued, you can play "Out of Shelf Life. We recommend disposal.”
  • the information stored in the database 40 can be transmitted to the outside via wired or wireless Internet.
  • an app on the smartphone that can access the refrigerator to check the storage information of the database 40 from the outside the user can check the food loaded in the user's refrigerator through the smartphone app at any place.
  • the information displayed through the user's smartphone app may be character information, but as shown in FIG. 17, the food image may be displayed using RGBA images, which are 'color + alpha images' extracted in the previous process.
  • controller 10 may periodically guide the voice or the screen through the speaker 70 or the touch screen unit 50 for the foods that are about to expire in the foods loaded through the database 40 search. You can also send it to your smartphone to let you know.
  • control unit 10 may repeatedly notify the user with a voice, a screen, and a smartphone about the food after the expiration date, the user may select or forcibly terminate the corresponding notification function through the touch screen.
  • the expiration date of the food stored in the refrigerator can be notified to the user, thereby providing a refrigerator that can prevent duplicate purchases and prevent deterioration due to long-term storage of the food.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Business, Economics & Management (AREA)
  • General Engineering & Computer Science (AREA)
  • Mechanical Engineering (AREA)
  • Thermal Sciences (AREA)
  • Chemical & Material Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Combustion & Propulsion (AREA)
  • Quality & Reliability (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Economics (AREA)
  • Finance (AREA)
  • Accounting & Taxation (AREA)
  • Development Economics (AREA)
  • Multimedia (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Human Resources & Organizations (AREA)
  • Marketing (AREA)
  • Operations Research (AREA)
  • Strategic Management (AREA)
  • Tourism & Hospitality (AREA)
  • General Business, Economics & Management (AREA)
  • Cold Air Circulating Systems And Constructional Details In Refrigerators (AREA)

Abstract

La présente invention concerne un appareil de notification d'une date d'expiration d'un aliment stocké par une intelligence artificielle de lecture de caractère dans un réfrigérateur et un procédé associé, pouvant notifier à un utilisateur des informations de date d'expiration par détection d'une date d'expiration par reconnaissance de caractères à partir d'informations d'image obtenues par capture, par une caméra, d'une région d'impression placée sur le papier d'emballage d'aliments stockés dans le réfrigérateur, pouvant notifier à un utilisateur un message de notification en émettant le message de notification pour les aliments dont la date d'expiration est imminente ou est dépassée, lorsqu'une porte de réfrigérateur est ouverte et pouvant accéder au réfrigérateur par une connexion filaire ou sans fil par l'intermédiaire d'une application de téléphone intelligent à distance, permettant ainsi à des aliments stockés dans le réfrigérateur d'être vérifiés.
PCT/KR2017/013434 2017-02-03 2017-11-23 Appareil de notification de date d'expiration d'aliments stockés par une intelligence artificielle de lecture de caractères dans un réfrigérateur et procédé associé WO2018143550A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR10-2017-0015827 2017-02-03
KR1020170015827A KR101756620B1 (ko) 2017-02-03 2017-02-03 냉장고의 문자판독 인공지능에 의한 보관식품 유통기한 알림장치와 그 방법

Publications (1)

Publication Number Publication Date
WO2018143550A1 true WO2018143550A1 (fr) 2018-08-09

Family

ID=59352878

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2017/013434 WO2018143550A1 (fr) 2017-02-03 2017-11-23 Appareil de notification de date d'expiration d'aliments stockés par une intelligence artificielle de lecture de caractères dans un réfrigérateur et procédé associé

Country Status (2)

Country Link
KR (1) KR101756620B1 (fr)
WO (1) WO2018143550A1 (fr)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107677031A (zh) * 2017-09-30 2018-02-09 珠海市领创智能物联网研究院有限公司 一种基于物联网家用安全型冰箱
CN108573387A (zh) * 2018-04-20 2018-09-25 开放智能机器（上海）有限公司 一种食品保质期的管理方法及系统
US20190384990A1 (en) * 2018-06-15 2019-12-19 Samsung Electronics Co., Ltd. Refrigerator, server and method of controlling thereof
CN110987088A (zh) * 2019-12-26 2020-04-10 广州中国科学院软件应用技术研究所 一种食材储存环境的监控系统
CN111292828A (zh) * 2018-12-07 2020-06-16 海信（山东）冰箱有限公司 智能冰箱及其食材管理方法、装置及存储介质
EP3677861A1 (fr) * 2019-01-04 2020-07-08 Samsung Electronics Co., Ltd. Appareil domestique et son procédé de commande
EP3852038A4 (fr) * 2018-11-05 2021-07-21 Samsung Electronics Co., Ltd. Système de gestion des aliments, appareil serveur et réfrigérateur
CN113660423A (zh) * 2021-08-18 2021-11-16 深圳市金博奥科技有限公司 一种智能摆台和智能摆台系统

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019177343A1 (fr) * 2018-03-13 2019-09-19 삼성전자주식회사 Réfrigérateur, et système et son procédé de commande
KR102217024B1 (ko) * 2018-03-13 2021-02-19 삼성전자주식회사 냉장고, 그 제어 방법 및 시스템
KR102663888B1 (ko) 2018-09-18 2024-05-08 삼성전자주식회사 냉장고 및 그의 제어방법
US11353259B2 (en) 2018-09-18 2022-06-07 Samsung Electronics Co., Ltd. Augmented-reality refrigerator and method of controlling thereof
KR20200037657A (ko) * 2018-10-01 2020-04-09 삼성전자주식회사 냉장고, 서버 및 냉장고의 객체 인식 방법
JP2020180739A (ja) * 2019-04-25 2020-11-05 シャープ株式会社 物品トレイ、冷蔵庫、及び物品トレイシステム
KR20190104955A (ko) 2019-08-23 2019-09-11 엘지전자 주식회사 냉장고 및 그의 구동 방법과 정보 제공 시스템
KR20210051580A (ko) 2019-10-30 2021-05-10 조현택 냉장고내 보관식품의 유통기한 자동 산출 장치
CN110864493B (zh) * 2019-11-20 2021-07-16 青岛海尔科技有限公司 温度调节方法和装置、冰箱及计算机可读存储介质
CN113483525A (zh) * 2020-05-13 2021-10-08 海信集团有限公司 保鲜设备及食材管理方法
KR20220003725A (ko) 2020-07-02 2022-01-11 한남대학교 산학협력단 소비자의 동선을 고려한 시간대별 지능형 상품 추천 기법
KR102466396B1 (ko) * 2020-09-17 2022-11-11 주식회사 니즈 상품 정보 관리 방법 및 장치
KR102473282B1 (ko) * 2022-08-25 2022-12-02 주식회사 두잉랩 인공지능을 활용한 영상 분석 기반 영양정보 제공 시스템 및 방법

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003004367A (ja) * 2001-06-21 2003-01-08 Nisshin Seifun Group Inc 冷蔵庫用食品賞味期限管理装置
KR20100047707A (ko) * 2008-10-29 2010-05-10 웅진코웨이주식회사 알림 기능용 냉장고
KR20110130901A (ko) * 2010-05-28 2011-12-06 전자부품연구원 냉장고의 보관 식품 관리 장치 및 방법
JP2012193873A (ja) * 2011-03-15 2012-10-11 Nikon Corp 収納装置
KR20160000931A (ko) * 2014-06-25 2016-01-06 (주)한국크레딕라이프 냉장고에 보관되는 식품을 사물인터넷(IoT)을 통해 관리하는 시스템 및 이에 관한 냉장고

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003004367A (ja) * 2001-06-21 2003-01-08 Nisshin Seifun Group Inc 冷蔵庫用食品賞味期限管理装置
KR20100047707A (ko) * 2008-10-29 2010-05-10 웅진코웨이주식회사 알림 기능용 냉장고
KR20110130901A (ko) * 2010-05-28 2011-12-06 전자부품연구원 냉장고의 보관 식품 관리 장치 및 방법
JP2012193873A (ja) * 2011-03-15 2012-10-11 Nikon Corp 収納装置
KR20160000931A (ko) * 2014-06-25 2016-01-06 (주)한국크레딕라이프 냉장고에 보관되는 식품을 사물인터넷(IoT)을 통해 관리하는 시스템 및 이에 관한 냉장고

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107677031A (zh) * 2017-09-30 2018-02-09 珠海市领创智能物联网研究院有限公司 一种基于物联网家用安全型冰箱
CN108573387A (zh) * 2018-04-20 2018-09-25 开放智能机器（上海）有限公司 一种食品保质期的管理方法及系统
US20190384990A1 (en) * 2018-06-15 2019-12-19 Samsung Electronics Co., Ltd. Refrigerator, server and method of controlling thereof
US11521391B2 (en) * 2018-06-15 2022-12-06 Samsung Electronics Co., Ltd. Refrigerator, server and method of controlling thereof
EP3852038A4 (fr) * 2018-11-05 2021-07-21 Samsung Electronics Co., Ltd. Système de gestion des aliments, appareil serveur et réfrigérateur
CN111292828A (zh) * 2018-12-07 2020-06-16 海信（山东）冰箱有限公司 智能冰箱及其食材管理方法、装置及存储介质
EP3677861A1 (fr) * 2019-01-04 2020-07-08 Samsung Electronics Co., Ltd. Appareil domestique et son procédé de commande
US11262962B2 (en) 2019-01-04 2022-03-01 Samsung Electronics Co., Ltd. Home appliance and control method thereof
US11599320B2 (en) 2019-01-04 2023-03-07 Samsung Electronics Co., Ltd. Home appliance and control method thereof
CN110987088A (zh) * 2019-12-26 2020-04-10 广州中国科学院软件应用技术研究所 一种食材储存环境的监控系统
CN113660423A (zh) * 2021-08-18 2021-11-16 深圳市金博奥科技有限公司 一种智能摆台和智能摆台系统
CN113660423B (zh) * 2021-08-18 2024-02-20 深圳市金博奥科技有限公司 一种智能摆台和智能摆台系统

Also Published As

Publication number Publication date
KR101756620B1 (ko) 2017-07-12

Similar Documents

Publication Publication Date Title
WO2018143550A1 (fr) Appareil de notification de date d'expiration d'aliments stockés par une intelligence artificielle de lecture de caractères dans un réfrigérateur et procédé associé
WO2017204596A1 (fr) Procédé et dispositif de correction de contours faciaux
WO2014168265A1 (fr) Procédé de gestion de produit de stockage dans un réfrigérateur utilisant un système de reconnaissance d'image et réfrigérateur associé
JP4811653B2 (ja) 物体検出装置
JPH06325180A (ja) 移動体自動追跡装置
CN108872265A (zh) 检测方法、装置及系统
US20220004748A1 (en) Video display method, device and system, and video camera
JPH0844874A (ja) 画像変化検出装置
JP2011060067A (ja) 自動検針システム
WO2014073939A1 (fr) Procédé et appareil de capture et d'affichage d'image
WO2014148691A1 (fr) Dispositif mobile et son procédé de commande
JPH0779429A (ja) 画像監視装置
JP5865584B2 (ja) 特定人物検知システムおよび検知方法
JPH0793558A (ja) 画像監視装置
KR102429557B1 (ko) 디스플레이 장치, 재생 판단 장치, 컨텐츠 재생 확인 방법
JP2005070985A (ja) 画像処理装置および方法、並びにプログラム
WO2022124673A1 (fr) Dispositif et procédé pour mesurer le volume d'un objet dans un réceptacle sur la base d'une image d'appareil de prise de vues en utilisant un modèle d'apprentissage automatique
CN115456984A (zh) 基于二维码高速识别图像识别缺陷检测系统
JP2002188939A (ja) 画像認識による指針式メーター測定システム及び指針式メーター測定方法
JP2004007246A (ja) 画像処理システム、画像処理方法、画像処理プログラム
WO2024035237A1 (fr) Système, procédé et programme informatique pour fournir des informations de chargement de produit de livraison
WO2014148741A1 (fr) Dispositif mobile et procédé de commande associé
JPH057363A (ja) 画像監視装置
CN112307882A (zh) 图像判断装置及图像判断系统
JPH0652311A (ja) 画像処理方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17894874

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205 DATED 28/10/2019)

122 Ep: pct application non-entry in european phase

Ref document number: 17894874

Country of ref document: EP

Kind code of ref document: A1