JP7292019B2 - Control device, interior photography system, article storage, interior photography method and program - Google Patents

Description

The present invention relates to a control device, an in-fridge photography system, an article storage, an in-fridge photography method, and a program.

A technology has been proposed that allows the user to check the inventory status of a refrigerator using a mobile terminal by capturing an image of the inside of the refrigerator with a CCD camera and outputting the image to the mobile terminal (see, for example, Patent Document 1). . The refrigerator described in Patent Document 1 is a refrigerator having a plurality of chambers that are divided vertically within the chamber, and is equipped with a CCD camera that can move up and down in the chamber, and is operated from a mobile terminal. Lifting and photographing are performed by a CCD camera, and the image of each of a plurality of storage rooms is output to a portable terminal by one CCD camera. With the recent trend toward larger refrigerators, a CCD camera necessary for photographing the inside of the refrigerator is used to photograph the inside of the refrigerator using a single vertically movable CCD camera, such as the refrigerator described in Patent Document 1. There is growing interest in reducing the number of cameras and reducing the cost of refrigerators.

JP-A-2002-303480

By the way, a refrigerator equipped with a camera for photographing the inside of the refrigerator, such as the refrigerator described in Patent Document 1, usually has a light source for illuminating the inside of the refrigerator. shoot inside. In this case, the brightness varies at various locations in the refrigerator depending on the distance from the light source in the refrigerator. As a result, the brightness of the stored items to be photographed varies depending on the position of the camera in the refrigerator, and the user may find it difficult to see the obtained photographed image.

The present invention has been made in view of the above reasons, and provides a control device, an in-fridge imaging system, an article storage, an in-fridge imaging method, and a program capable of improving the visibility of a photographed image inside the storage. for the purpose.

In order to achieve the above object, the control device according to the present invention includes:
A lighting device that turns on or off a light source that illuminates the interior of an article storage based on light source control information, an imaging device that captures the interior of the storage, and a movement of the imaging device within the storage based on movement control information. A control device that controls a moving device that causes
an opening/closing determination unit that determines whether the door of the article storage is in an open state or a closed state;
a movement control unit that generates movement control information for causing the imaging device to move in the refrigerator and outputs the movement control information to the movement device;
When the opening/closing determination unit determines that the door is open, light source control information for lighting the light source at a predetermined light emission intensity is generated and sent to the lighting device. light source control information for turning on the light source at a light emission intensity corresponding to the position of the imaging device when the open/close determination unit determines that the door is closed; a light source control unit that generates and outputs to the lighting device;
an image acquisition unit that acquires image information indicating an image captured by the imaging device;
a brightness calculation unit that calculates average brightness in at least part of an image indicated by the image information acquired by the image acquisition unit;
The light source control unit changes the light emission intensity of the light source so that the average luminance becomes equal to or higher than a preset reference luminance when the open/close determination unit determines that the door is closed. Generate information.

According to the present invention, when the opening/closing determination unit determines that the door of the article storage is open, the light source control unit sets the light emission intensity of the light source to the preset light emission intensity to turn on the light source. Generate and output light source control information. On the other hand, when the opening/closing determination unit determines that the door is closed, the light source control unit generates light source control information for turning on the light source with the light emission intensity corresponding to the position of the imaging device. and output. As a result, variations in the brightness of the articles to be photographed due to the difference in the position of the imaging device in the refrigerator are suppressed, so the visibility of the photographed image of the interior obtained by the imaging device is improved. can be made

1 is a schematic configuration diagram of an in-fridge image browsing system according to an embodiment of the present invention; Schematic front view of an example of a refrigerator according to an embodiment 1 is a block diagram showing the configuration of an in-fridge photography system according to an embodiment; FIG. 4 shows information stored in a light emission intensity storage unit according to the embodiment; (A) is a diagram showing the position of the imaging device according to the embodiment, and (B) is a diagram showing the light irradiation direction of the light emitting unit at each position according to the embodiment. 1A is a block diagram showing the configuration of a server according to an embodiment, and FIG. 1B is a block diagram showing the configuration of a terminal device according to an embodiment; 4 is a sequence diagram showing the operation of the refrigerator interior image viewing system according to the embodiment; FIG. 4 is a sequence diagram showing the operation of the refrigerator interior image viewing system according to the embodiment; FIG. 4 is a flowchart showing an example of the flow of in-fridge photography processing executed by the control device according to the embodiment; 4 is a flowchart showing an example of the flow of image transfer processing executed by the control device according to the embodiment; 4 is a flowchart showing an example of the flow of server control processing executed by the server according to the embodiment; Flowchart showing an example of the flow of image browsing processing executed by the terminal device according to the embodiment Block diagram showing the configuration of an in-fridge photography system according to a modification Flowchart showing an example of the flow of in-fridge photographing processing executed by the control device according to the modification (A) is a schematic front view of an example of a refrigerator according to a modification, and (B) is a schematic front view of an example of a refrigerator according to another modification. Block diagram showing the configuration of an in-fridge photography system according to a modification The figure which shows the information which the sub light source memory|storage part which concerns on a modification stores. Block diagram showing the configuration of an in-fridge photography system according to a modification The figure which shows the information which the sub light source memory|storage part which concerns on a modification stores. Flowchart showing an example of the flow of in-fridge photographing processing executed by the control device according to the modification Flowchart showing an example of the flow of image transfer processing executed by a control device according to a modification

Hereinafter, a control device according to an embodiment of the present invention and an in-fridge photography system including the control device will be described with reference to the accompanying drawings. The control device according to the present embodiment includes a lighting device for lighting a light source for illuminating the interior of a refrigerator, which is an article storage, based on light source control information, an imaging device for capturing an image of the interior, and an imaging device in the interior. a mobile device that moves based on the mobile control information. The control device includes an open/close determination section, a movement control section, and a light source control section. The open/close determination unit determines whether the door of the refrigerator is open or closed. The movement control unit generates movement control information for moving the imaging device to one of a plurality of preset positions in the refrigerator and outputs the movement control information to the movement device. The light source control unit generates light source control information for turning on the light source at a preset maximum light emission intensity when the opening/closing determination unit determines that the refrigerator door is open. Output to the lighting device. On the other hand, when the opening/closing determination unit determines that the refrigerator door is closed, the light source control unit provides light source control information for turning on the light source with the light emission intensity corresponding to the position of the imaging device. is generated and output to the lighting device.

As shown in FIG. 1, a refrigerator interior image viewing system 100 according to the present embodiment includes a refrigerator 1 having an interior photographing system 2 according to the present embodiment, a router 3, a server 5, and a terminal device 4. , provided. The in-fridge photography system 2 can communicate with the server 5 via the router 3 and the network NT. Also, the server 5 can communicate with the terminal device 4 via the network NT. Here, the network NT is, for example, a wide area network using public lines. The router 3 is, for example, a wireless LAN (Local Area Network) router, and communicates with the in-fridge photography system 2 by a communication method conforming to the wireless LAN standard or the Bluetooth (registered trademark) standard.

Refrigerator 1 is, for example, as shown in FIG. 2, in the shape of a rectangular box with an open front, and includes a refrigerating compartment 30 for refrigerating articles including food, a freezer compartment (not shown) for freezing articles, and a vegetable compartment. (not shown). Here, the refrigerating compartment 30 , the freezing compartment (not shown) and the vegetable compartment (not shown) correspond to the inside of the refrigerator 1 . The refrigerator 1 also includes two doors 31 and 32 covering an open portion corresponding to the refrigerator compartment 30 of the main body 1a, a drawer 35 installed in a portion corresponding to the freezer compartment of the main body 1a, and a vegetable compartment of the main body 1a. and a drawer 36 mounted on the corresponding part. Shelves 33 and 34 on which articles are placed are provided inside the refrigerator compartment 30 of the main body 1a, and two doors 31 and 32 are provided on the front side of the refrigerator compartment 30 in the main body 1a. The refrigerator 1 also includes a compressor, an evaporator, a circulation fan, a temperature sensor, a buzzer, and the like (not shown) to cool the articles in the refrigerator. In the example shown in FIG. 2, two shelves 33 and 34 are provided in the refrigerator compartment 30 of the main body 1a, but the number of shelves may be one or three or more. Moreover, the number of doors provided on the front side of the refrigerator compartment 30 in the main body 1a is not limited to two, and may be one.

The refrigerator 1 also includes an open/close sensor 23 , a light source 24 , a lighting device 26 , an imaging device 22 , a moving device 25 and a control device 21 . The open/close sensor 23 is a sensor for detecting the open/close state of each of the doors 31 and 32, and is provided at two positions at the upper end of the open portion corresponding to the refrigerator compartment 30 of the main body 1a. As the open/close sensor 23, for example, a magnetic sensor or an infrared sensor can be employed. The open/close sensor 23 outputs an open detection signal when the doors 31 and 32 are opened. The positions of the open/close sensor 23 are not limited to the two positions at the upper end of the open portion corresponding to the refrigerator compartment 30 of the main body 1a.

The light source 24 is arranged on the ceiling of the refrigerator compartment 30 of the main body 1a. As the light source 24, for example, a light source using an LED (Light Emitting Diode) element or an organic EL element, or a fluorescent lamp or an incandescent lamp can be adopted. Based on a light source control signal input from the control device 21, the lighting device 26 lights the light source 24 with a plurality of preset luminous intensities that allow identification of articles stored in the refrigerator compartment 30. The lighting device 26 has, for example, a step-up/step-down chopper circuit, and dims the light source 24 according to the signal voltage of the light source control signal. Also, the lighting device 26 turns off the light source 24 based on the light source control signal input from the control device 21 .

The imaging device 22 is, for example, a CCD camera, and images the interior of the refrigerator compartment 30 based on a control signal input from the control device 21 . The image capturing device 22 captures an image of the interior of the refrigerator compartment 30 from the central portion in the left-right direction in FIG. 2 with the door 31 closed. Further, when imaging the inside of the refrigerator compartment 30 , the imaging device 22 outputs an image signal corresponding to the captured image to the control device 21 . The moving device 25 is provided at the tip of the door 31 on the left side in FIG. 2, and moves the imaging device 22 in the vertical direction. The moving device 25 includes, for example, a slide body (not shown) holding the imaging device 22, a long ball screw (not shown) arranged along the tip of the door 31, and a ball screw extending in the longitudinal direction. and a motor 2501 that rotates around a central axis along. The moving device 25 raises and lowers the imaging device 22 held by the slide body by rotating the ball screw with the motor 2501 . The positions of the imaging device 22 and the moving device 25 are not limited to the tip of the left door 31 in FIG. 2. For example, the right door 32 in FIG. Other possible positions are also possible.

Control device 21 is arranged above refrigerator compartment 30 . Note that the position of the control device 21 is not limited to above the refrigerator compartment 30, and may be other positions. As shown in FIG. 3, the control device 21 includes a CPU (Central Processing Unit) 201, a main memory section 202, an auxiliary memory section 203, a wireless module 204, and a sensor interface (hereinafter referred to as "I/F") 251. , an imaging device I/F 252, a mobile device I/F 253, a light source I/F 254, and a bus 209 interconnecting them. A main storage unit 202 is a volatile memory and is used as a work area for the CPU 201 . The auxiliary storage unit 203 is a non-volatile memory such as a magnetic disk or semiconductor flash memory, and stores programs for realizing various functions of the control device 21 . Wireless module 204 is connected to network NT via router 3 . The wireless module 204 communicates with the router 3 using a communication method conforming to the wireless LAN standard (for example, IEEE802.11b/g/n).

The sensor I/F 251 is connected to the open/close sensor 23 and repeatedly outputs open detection information to the CPU 201 while an open detection signal is being input from the open/close sensor 23 . The imaging device I/F 252 is connected to the imaging device 22 , and when imaging command information for instructing the imaging device 22 to shoot the inside of the refrigerator compartment 30 is input from the CPU 201 , the imaging device 22 shoots the inside of the refrigerator compartment 30 . is generated and output to the imaging device 22 . Further, when the image signal described above is input from the imaging device 22, the imaging device I/F 252 generates image information corresponding to the input image signal. Further, the imaging device I/F 252 has, for example, a DMA (Direct Memory Access) controller, and DMA-transfers the generated image information to the auxiliary storage unit 203 .

The mobile device I/F 253 is connected to a driver 2502 that drives a motor 2501 of the mobile device 25 . When movement control information for driving the motor 2501 to move the imaging device 22 is input from the CPU 201, the movement device I/F 253 transmits a control signal to the driver 2502 for a period corresponding to the input movement control information. keep outputting. The light source I/F 254 is connected to the lighting device 26 and outputs a light source control signal to the lighting device 26 to turn on the light source 24 . Light source I/F 254 generates a corresponding light source control signal based on the light source control information input from CPU 201 and outputs it to lighting device 26 .

The CPU 201 functions as an open/close determination unit 211, a light source control unit 212, a movement control unit 213, an imaging control unit 214, and a transfer unit 215 by reading programs stored in the auxiliary storage unit 203 into the main storage unit 202 and executing them. . Note that the CPU 201 further includes an open time measurement unit (not shown) that measures an open time corresponding to the time during which the doors 31 and 32 are maintained in an open state, It also functions as an open time determination unit (not shown) that determines whether there is a do. The auxiliary storage unit 203 also has an image storage unit 231 , a light emission intensity storage unit 232 and an imaging position storage unit 233 . The image storage unit 231 stores image information indicating images captured by the imaging device 22 . Furthermore, the main storage unit 202 stores completion flag information indicating whether or not the transfer of the image information stored in the image storage unit 231 to the server 5 has been completed. When this completion flag information is in the "OFF" state, it indicates that the transfer of the image information to the server 5 has not been completed, and in the "ON" state, the transfer of the image information to the server 5 has been completed. indicates that

For example, as shown in FIG. 4, the light emission intensity storage unit 232 stores light emission intensity information indicating the light emission intensity of the light source 24 in association with imaging position information indicating the position of the imaging device 22 . Here, for example, as shown in FIG. 5A, when the light source 24 is provided in the upper part of the refrigerator compartment 30, the position Pos1 is the closest to the light source 24 and the position Pos3 is the farthest from the light source 24. In addition, position Pos1 is located above intermediate position C1 in the vertical direction between shelf 33 and the ceiling of refrigerator compartment 30 . Position Pos2 is located above middle position C2 in the vertical direction between shelf 33 and shelf 34 . Position Pos3 is located above intermediate position C3 in the vertical direction between shelf 34 and the bottom of refrigerator compartment 30 . Then, as shown in FIG. 5B, the imaging device 22 is tilted at each position Pos1, Pos2, and Pos3 so as to look down on the articles placed on the shelves 33 and 34 or the bottom of the refrigerator compartment 30 from above. Take an image while holding it (see areas A1, A2, and A3 in FIG. 5B). The light emission intensity information is set so that the light emission intensity of the light source 24 increases as the distance between the light source 24 and the imaging device 22 increases. For example, as shown in FIG. 4, it is assumed that light emission intensity information LU[1], LU[2], and LU[3] are associated with imaging position information of positions Pos1, Pos2, and Pos3, respectively. In this case, the emission intensity indicated by the emission intensity information LU[1] is the lowest, and the emission intensity indicated by the emission intensity information LU[3] is the highest. The emission intensity indicated by the emission intensity information LU[2] is set higher than the emission intensity indicated by the emission intensity information LU[1] and lower than the emission intensity indicated by the emission intensity information LU[3]. The imaging position storage unit 233 stores, for example, the number of rotations of the motor 2501 when moving the imaging device 22 from the initial position to positions Pos1, Pos2, and Pos3 in association with the imaging position information.

Returning to FIG. 3 , open/close determination unit 211 determines whether doors 31 and 32 of refrigerator 1 are open or closed based on whether or not open detection information is input from sensor I/F 251 . The open/close determination unit 211 determines that the doors 31 and 32 are open while the open detection information is being input from the sensor I/F 251, and when the open detection information is no longer input from the sensor I/F 251, the door 31 is opened. , 32 are open.

The movement control unit 213 generates movement control information for moving the imaging device 22 to any one of three preset positions Pos1, Pos2, and Pos3 in the refrigerator compartment 30, and outputs the movement control information to the movement device 25. do. Here, the movement control unit 213 may first move the imaging device 22 to the position Pos1 and then move the imaging device 22 to the positions Pos2 and Pos3 in this order, or may move the imaging device 22 to the position Pos3 first. , the imaging device 22 may be moved in the order of positions Pos2 and Pos1. Further, when the movement of the imaging device 22 to the positions Pos1, Pos2, and Pos3 is completed, the movement control unit 213 notifies the imaging control unit 214 of movement completion notification to that effect.

When the above-mentioned movement completion notification is notified from the movement control unit 213 , the photographing control unit 214 generates the above-mentioned photographing command information and outputs it to the imaging device I/F 252 .

When the opening/closing determination unit 211 determines that the doors 31 and 32 are open, the light source control unit 212 controls the light source 24 to light the light source 24 by setting the light emission intensity of the light source 24 to a preset maximum light emission intensity. Information is generated and output to the light source I/F 254 . On the other hand, when the opening/closing determination unit 211 determines that the doors 31 and 32 are closed, the light source control unit 212 sets the light emission intensity of the light source 24 according to the positions Pos1, Pos2, and Pos3 of the imaging device 22. light source control information for turning on the light source 24 and outputs it to the light source I/F 254 . Furthermore, when turning off the light source 24 , the light source control unit 212 generates light source control information for turning off the light source 24 and outputs the information to the light source I/F 254 . Further, when the opening/closing determination unit 211 determines that the doors 31 and 32 are closed, the light source control unit 212 increases the light emission intensity of the light source 24 as the distance between the light source 24 and the imaging device 22 increases. The light source control information is generated so that Specifically, the light source control unit 212 is associated with positions Pos1, Pos2, and Pos3, for example, as shown in FIG. light source control information is generated based on the light emission intensity information set to .

Returning to FIG. 3, transfer unit 215 transfers image information stored in image storage unit 231 to server 5 via wireless module 204, router 3 and network NT.

As shown in FIG. 6A, the server 5 has a CPU 501, a main storage unit 502, an auxiliary storage unit 503, a communication unit 504, and a bus 509 interconnecting them. Like the main memory 202, the main memory 502 is a volatile memory. The auxiliary memory 503, like the auxiliary memory 203, is a non-volatile memory. Memorize the program. The communication unit 504 is a wireless module or a wired communication interface, and communicates with the in-fridge photography system 2 via the network NT.

The CPU 501 functions as an image acquisition unit 511 , a state determination unit 512 , an image transmission unit 513 , and an authentication unit 514 by reading programs stored in the auxiliary storage unit 503 into the main storage unit 502 and executing them. The auxiliary storage unit 503 has an image storage unit 531 that stores image information received from the in-fridge photography system 2 and an authentication information storage unit 532 that stores authentication information of the terminal device 4 . Here, the authentication information includes, for example, identification information unique to the terminal device 4, password information indicating a password given to the user of the terminal device 4, and the like.

The image acquisition unit 511 acquires image information indicating an image obtained by photographing the inside of the refrigerating compartment 30 of the refrigerator 1 transmitted from the interior photographing system 2 and stores the acquired image information in the image storage unit 531 . The state determination unit 512 acquires load amount information indicating the load amount of the CPU 501, and determines whether or not the load amount indicated by the acquired load amount information is equal to or less than a preset reference load amount. In addition, the state determination unit 512 acquires free space information indicating the free space of the image storage unit 531, and determines whether or not the free space indicated by the acquired free space information is equal to or greater than a preset reference free space. .

The image transmission unit 513 transmits an image transmission request from the image information stored in the image storage unit 531 based on the identification information of the terminal device 4 included in the image transmission request information requesting the server 5 to transmit the image information. Image information corresponding to the terminal device 4 that is the transmission source of the information is acquired. Then, the image transmission unit 513 transmits the acquired image information to the terminal device 4 that is the transmission source of the image transmission request information. Upon receiving the image transmission request information, the authentication unit 514 refers to the authentication information stored in the authentication information storage unit 532, and based on the identification information, password information, etc. of the terminal device 4 included in the received image transmission request information. , the terminal device 4 is authenticated.

As shown in FIG. 6B, the terminal device 4 is, for example, a mobile terminal, and includes a CPU 401, a main storage unit 402, an auxiliary storage unit 403, a communication unit 404, a display unit 405, and an input unit 406, which are interconnected. It has a bus 409 that The main memory unit 402 is a volatile memory like the main memory unit 202, and the auxiliary memory unit 403 is a non-volatile memory like the auxiliary memory unit 203. program. The communication unit 404 is a wireless module or a wired communication interface, and communicates with the server 5 via the network NT. A display unit 405 has a display device such as a liquid crystal display and an interface, and displays information input from the CPU 401 . The input unit 406 has, for example, a touch panel and an interface, and outputs operation information to the CPU 401 according to the operation of the terminal device 4 by the user.

The CPU 401 functions as an operation reception unit 411 , an image acquisition unit 412 and a display control unit 413 by reading programs stored in the auxiliary storage unit 403 into the main storage unit 402 and executing them. The auxiliary storage unit 403 has an image storage unit 431 that stores image information received from the server 5 .

When the user of the terminal device 4 operates the input unit 406, the operation accepting unit 411 accepts operation information corresponding to the content of the operation. The image acquisition unit 412 acquires image information from the server 5 by generating image transmission request information for requesting transmission of image information from the server 5 and transmitting it to the server 5, and stores the acquired image information in the image storage unit. Store in 431. Here, the image transmission request information includes identification information of the terminal device 4, password information, etc., as described above. The display control unit 413 of the terminal device 4 acquires image information stored in the image storage unit 431 based on the operation information received by the operation reception unit 411 and outputs the image information to the display unit 405 .

Next, the operation of the in-fridge image viewing system 100 according to the present embodiment will be described with reference to FIGS. 7 and 8. FIG. First, as shown in FIG. 7, when the interior photographing system 2 determines that the doors 31 and 32 of the refrigerator 1 are closed (step S1), the light emission intensity of the light source 24 is measured at the position of the imaging device 22 (for example, the position Pos1) is set to the emission intensity corresponding to Pos1) and turned on (step S2). Next, the in-fridge imaging system 2 moves the imaging device 22 to a position (for example, position Pos1) (step S3). Subsequently, after the imaging device 22 has moved to the imaging position, the in-fridge imaging system 2 performs imaging of the interior of the refrigerator compartment 30 (step S4). Thereafter, the in-fridge imaging system 2 causes the image storage unit 231 to store image information indicating the image obtained by imaging (step S5). As long as it is determined that the doors 31 and 32 of the refrigerator 1 are closed, the interior imaging system 2 positions the imaging devices 22 at a plurality of preset positions Pos1, Pos2, and Pos3 in the refrigerator compartment 30, respectively. The processing from steps S2 to S5 is repeatedly executed until all photographing in the moved state is completed.

Then, it is assumed that the in-fridge imaging system 2 determines that imaging of the interior of the refrigerator compartment 30 has been completed (step S6). Here, the in-fridge photographing system 2 moves the imaging device 22 to each of a plurality of preset positions Pos1, Pos2, and Pos3 in the refrigerator compartment 30 when all photographing is completed. is determined to have been completed. In this case, transfer permission request information requesting permission to transfer the image information to the server 5 is transmitted from the in-fridge photographing system 2 to the server 5 (step S7).

On the other hand, when the server 5 receives the transfer permission request information, it acquires free space information indicating the free space of the image storage unit 531 and load amount information indicating the load amount of the CPU 501 (step S8). Here, it is assumed that the server 5 determines that the free space of the image storage unit 531 is greater than or equal to a preset reference capacity and that the load amount of the CPU 501 is less than or equal to a preset reference load amount (step S9). In this case, transfer permission information for notifying the in-fridge photographing system 2 of permission to transfer the image information is transmitted from the server 5 to the in-fridge photographing system 2 (step S10). When the in-fridge photography system 2 receives the transfer permission information, the image information is transferred from the in-fridge photography system 2 to the server 5 (step S11). On the other hand, when receiving the image information, the server 5 stores the received image information in the image storage unit 531 (step S12).

After that, it is assumed that the user performs an operation to display an image inside the refrigerator compartment 30 on the display unit 405 via the input unit 406 of the terminal device 4 . In this case, the terminal device 4 receives operation information corresponding to the operation performed by the user on the input unit 406 (step S13). Then, as shown in FIG. 8, image transmission request information requesting transmission of image information to the server 5 is transmitted from the terminal device 4 to the server 5 (step S14). On the other hand, when the server 5 receives the image transmission request information, the server 5 refers to the authentication information stored in the authentication information storage unit 532, and based on the identification information, password information, etc. of the terminal device 4 included in the received image transmission request information. Then, authentication processing is executed for the terminal device 4 (step S15). Then, when the authentication process is successful, the server 5 determines to permit transmission of the image information to the terminal device 4 (step S16). In this case, the image information selected from the image storage unit 531 in the server 5 based on the identification information of the terminal device 4 is transmitted from the server 5 to the terminal device 4 that transmitted the image transmission request information (step S17). On the other hand, when receiving the image information, the terminal device 4 causes the display unit 405 to display an image of the inside of the refrigerator compartment 30 indicated by the received image information (step S18).

Next, the in-fridge photographing process executed by the control device 21 according to the present embodiment will be described with reference to FIG. This in-fridge photographing process is started, for example, when the refrigerator 1 equipped with the in-fridge photographing system 2 including the control device 21 is powered on. Also, it is assumed that the completion flag information indicating whether or not the photographing in the refrigerator compartment 30 is completed and stored in the main storage unit 202 is set to the "OFF" state. First, the opening/closing determination unit 211 determines whether or not the doors 31 and 32 of the refrigerator 1 are closed (step S101). As long as the opening/closing determination unit 211 determines that the doors 31 and 32 are open (step S101: No), the process of step S101 is repeatedly executed.

On the other hand, when the open/close determination unit 211 determines that the doors 31 and 32 are closed (step S101: Yes), the light source control unit 212 determines the predetermined positions Pos1, Pos2, and Pos3 in the refrigerator compartment 30. It is determined whether or not all photographing has been completed with the imaging devices 22 moved respectively, and photographing of the inside of the refrigerator compartment 30 has been completed (step S102). When the light source control unit 212 determines that the photographing of the inside of the refrigerator compartment 30 has not been completed yet (step S102: No), the light source control unit 212 sets the light emission intensity of the light source 24 to the light emission intensity corresponding to the position of the imaging device 22 which has not yet taken the photograph. Light source control information for setting the intensity is generated and output to the light source I/F 254 (step S103). At this time, the light source I/F 254 outputs a light source control signal corresponding to the light source control information to the lighting device 26, and the lighting device 26 lights the light source 24 with the emission intensity set based on the light source control signal.

Next, the movement control unit 213 generates movement control information for moving the imaging device 22 to a position where photography has not yet been performed, and outputs the movement control information to the movement device I/F 253 (step S104). At this time, the moving device I/F 253 generates a moving control signal corresponding to the moving control information and outputs it to the moving device 25, and the moving device 25 moves the imaging device 22 based on the input moving control signal. Let Here, when the movement of the imaging device 22 is completed, the movement control unit 213 notifies the imaging control unit 214 of a movement completion notification.

Subsequently, when the movement completion notification is notified from the movement control section 213, the photographing control section 214 generates the aforementioned photographing instruction information and outputs it to the imaging device I/F 252 (step S105). At this time, when the imaging device I/F 252 receives the imaging command information, the imaging device I/F 252 generates a control signal for causing the imaging device 22 to take an image of the inside of the refrigerator compartment 30, and outputs the control signal to the imaging device 22. When the control signal is input, the inside of the refrigerator compartment 30 is photographed. The imaging device 22 outputs an image signal representing the captured image to the imaging device I/F 252 , and the imaging device I/F 252 generates image information corresponding to the input image signal and transfers it to the image storage unit 231 . . Then, the image storage unit 231 stores the image information transferred from the imaging device I/F 252 (step S106). After that, the process of step S109, which will be described later, is executed. At the same time when the image storage unit 231 stores the image information in step S106, the opening/closing determination unit 211 may execute the process of step S109, which will be described later. In this case, after the door 31, 32 is determined to be in the closed state by the opening/closing determination unit 211 in step S109, which will be described later (step S109: No), the steps from step S101 to step S103 described above are performed again, and then step S104 described above is performed. The higher the processing power of the CPU 201, the shorter the time until the process of . For example, if the operating frequency of the CPU 201 is high, it is possible to complete the photographing of the inside of the refrigerator compartment 30 without stopping the movement of the imaging device 22 .

Further, when the light source control unit 212 determines in the process of step S102 that the photographing in the refrigerator compartment 30 has been completed (step S102: Yes), it generates light source control information for turning off the light source 24 and turns off the light source. Output to I/F 254 (step S107). At this time, the light source I/F 254 receives the light source control information and stops outputting the light source control signal to the lighting device 26 , and the lighting device 26 turns off the light source 24 . Next, image transfer processing is executed (step S108).

In this image transfer process, for example, as shown in FIG. 10, first, the transfer unit 215 determines whether or not the completion flag information stored in the main storage unit 202 is set to "OFF" (step S201). . When the transfer unit 215 determines that the completion flag information is set to the "ON" state (step S201: No), the process of step S109, which will be described later, is executed as it is.

On the other hand, when the transfer unit 215 determines that the completion flag information is set to the "OFF" state (step S201: Yes), it generates transfer permission request information and transmits it to the server 5 (step S202). Next, the transfer unit 215 determines whether transfer permission information has been received from the server 5 (step S203). When the transfer section 215 determines that the transfer permission information has been received from the server 5 (step S203: Yes), the transfer section 215 transfers the image information stored in the image storage section 231 to the server 5 (step S204). Subsequently, the transfer unit 215 sets the completion flag information to the "ON" state (step S205), and the process of step S109, which will be described later, is executed.

On the other hand, when the transfer unit 215 determines that the transfer permission information has not been received from the server 5 (step S203: No), it determines whether or not the preset transfer standby reference time has passed (step S206). When the transfer unit 215 determines that the transfer standby reference time has not passed (step S206: No), the process of step S109, which will be described later, is executed as it is. On the other hand, when the transfer unit 215 determines that the transfer standby reference time has passed (step S206: Yes), the transfer unit 215 sets the completion flag information to the "ON" state.

Returning to FIG. 9, after that, the opening/closing determination unit 211 determines whether or not the doors 31 and 32 of the refrigerator 1 are open (step S109). When the opening/closing determination unit 211 determines that the doors 31 and 32 are closed (step S109: No), the process of step S101 is executed again. On the other hand, when the opening/closing determination unit 211 determines that the doors 31 and 32 are open (step S109: Yes), the imaging control unit 214 generates imaging stop command information and outputs it to the imaging device I/F 252 ( step S110). At this time, when the imaging stop command information is input, the imaging device I/F 252 generates a control signal for causing the imaging device 22 to stop imaging and outputs it to the imaging device 22, and the imaging device 22 receives the control signal. Shooting is stopped when input is made.

Next, the transfer unit 215 sets the completion flag information stored in the main storage unit 202 to the "OFF" state (step S111). Subsequently, the light source control unit 212 generates light source control information for causing the light source 24 to emit light with the maximum emission intensity, and outputs the light source control information to the light source I/F 254 (step S112). At this time, the light source I/F 254 outputs a light source control signal corresponding to the light source control information to the lighting device 26, and the lighting device 26 lights the light source 24 with the maximum emission intensity based on the light source control signal. After that, the process of step S101 is executed again.

Next, server control processing executed by the server 5 according to the present embodiment will be described with reference to FIG. 11 . This server control process is started, for example, when the server 5 is powered on. First, the state determination unit 512 determines whether transfer permission request information requesting permission to transfer image information has been received from the in-fridge photography system 2 (step S301). When the state determination unit 512 determines that the transfer permission request information has not been received (step S301: No), it executes the process of step S306, which will be described later.

On the other hand, when the state determination unit 512 determines that the transfer permission request information has been received (step S301: Yes), the state determination unit 512 acquires free space information indicating the free space of the image storage unit 531 and load amount information indicating the load amount of the CPU 501. (Step S302). Next, the state determination unit 512 determines whether the load amount indicated by the acquired load amount information is equal to or less than the aforementioned reference load amount and whether or not the free space indicated by the acquired free space information is equal to or greater than the aforementioned reference free space. Determine (step S303). When the state determination unit 512 determines that the load amount of the CPU 501 exceeds the reference load amount or that the free space of the image storage unit 531 is less than the reference free space (step S303: No), the process of step S306 described later is performed. is executed.

On the other hand, when the state determination unit 512 determines that the load amount of the CPU 501 is equal to or less than the reference load amount and the free space of the image storage unit 531 is equal to or more than the reference free space (step S303: Yes), the image acquisition unit 511 Transfer permission information is generated and transmitted to the in-fridge photographing system 2 (step S304). Subsequently, the image acquisition unit 511 receives image information from the in-fridge imaging system 2, and stores the received image information in the image storage unit 531 (step S305).

After that, the authentication unit 514 determines whether image transmission request information requesting transmission of image information has been received from the terminal device 4 (step S306). If the authentication unit 514 determines that the image transmission request information has not been received (step S306: No), the process of step S301 is executed again. On the other hand, it is assumed that the authentication unit 514 determines that the image transmission request information has been received from the terminal device 4 (step S306: Yes). In this case, the authentication unit 514 refers to the authentication information stored in the authentication information storage unit 532, and authenticates the terminal device 4 based on the identification information, password information, etc. of the terminal device 4 included in the received image transmission request information. Authentication processing is executed (step S307). Next, when the authentication unit 514 fails the authentication process and determines that the transmission of the image information to the terminal device 4 is not permitted (step S308: No), the process of step S301 is executed again. On the other hand, it is assumed that the authentication unit 514 succeeds in the authentication process and determines that transmission of image information to the terminal device 4 is permitted (step S308: Yes). In this case, the image transmission unit 513 selects image information from the image storage unit 531 based on the identification information of the terminal device 4 that is the transmission source of the image transmission request information, and transmits the selected image information to the image transmission request information. It is transmitted to the original terminal device 4 (step S309). After that, the process of step S301 is executed again.

Next, image viewing processing executed by the terminal device 4 according to the present embodiment will be described with reference to FIG. This image viewing process is started when, for example, the terminal device 4 is powered on and a program for viewing images in the refrigerating compartment 30 of the refrigerator 1 is activated in the terminal device 4 . First, the operation reception unit 411 determines whether operation information corresponding to an operation for displaying an image inside the refrigerator compartment 30 on the display unit 405 has been received (step S401). For example, when the user of the terminal device 4 performs an operation to display an image inside the refrigerator compartment 30 on the display unit 405 via the input unit 406 of the terminal device 4, the operation reception unit 411 receives the operation information. . The operation reception unit 411 repeatedly executes the process of step S401 unless the operation information described above is received (step S401: No).

On the other hand, when the operation reception unit 411 determines that the above-described operation information has been received (step S401: Yes), the image acquisition unit 412 transmits image transmission request information requesting transmission of image information from the server 5 to the server 5. (step S402). Next, after transmitting the image transmission request information, the image acquisition unit 412 determines whether or not image information has been received (step S403). If the image acquisition unit 412 does not receive the image information within the preset image information reception standby period after transmitting the image transmission request information (step S403: No), the process of step S401 is executed again.

On the other hand, when the image acquisition unit 412 receives the image information (step S403: Yes), the image storage unit 431 stores the received image information. Then, the display control unit 413 causes the display unit 405 to display the image inside the refrigerator compartment 30 indicated by the image information acquired by the image acquisition unit 412 from the image storage unit 431 (step S404). After that, the process of step S401 is executed again.

As described above, according to the interior photography system 2 according to the present embodiment, when the light source control unit 212 determines that the doors 31 and 32 of the refrigerator 1 are open by the open/close determination unit 211, Light source control information for turning on the light source 24 with the light emission intensity of the light source 24 set to the maximum light emission intensity is generated and output to the light source I/F 254 . On the other hand, when the opening/closing determination unit 211 determines that the doors 31 and 32 are closed, the light source control unit 212 sets the light emission intensity of the light source 24 according to the positions Pos1, Pos2, and Pos3 of the imaging device 22. light source control information for turning on the light source 24 and outputs it to the light source I/F 254 . As a result, variation in the brightness of the object to be photographed due to differences in the positions Pos1, Pos2, and Pos3 of the imaging device 22 in the refrigerating compartment 30 of the refrigerator 1 is suppressed. The visibility of the photographed image in the refrigerator compartment 30 can be improved.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. For example, when the light source control unit 212 determines that the doors 31 and 32 of the refrigerator 1 are closed by the opening/closing determination unit 211, the average brightness of the image obtained by the imaging device 22 is set to a preset reference value. The light source control information may be generated so as to be equal to or higher than the luminance. For example, as shown in FIG. 13, in the control device 2021 according to the present modification, the CPU 201 reads out the program stored in the auxiliary storage unit 203 to the main storage unit 202 and executes it, so that the opening/closing determination unit 211 and the light source control unit 212 , a movement control unit 213 , an imaging control unit 214 , a transfer unit 215 , an image acquisition unit 2216 and a luminance calculation unit 2217 . In addition, in FIG. 13, the same reference numerals as in FIG. 3 denote the same configurations as in the embodiment.

The image acquisition unit 2216 acquires image information indicating an image captured by the imaging device 22 and stored in the image storage unit 231 . The brightness calculation unit 2217 calculates the average brightness of the entire image indicated by the image information acquired by the image acquisition unit 2216 .

The light source control unit 212 determines whether or not the average luminance calculated by the luminance calculation unit 2217 is equal to or higher than a preset reference luminance. Then, when the average brightness is less than the reference brightness, the light source control unit 212 generates light source control information so that the average brightness becomes equal to or higher than the preset reference brightness, and outputs the light source control information to the light source I/F 254 .

Next, the in-fridge photographing process executed by the control device 2021 according to this modification will be described with reference to FIG. 14 . In addition, in FIG. 14, the same reference numerals as in FIG. 9 are assigned to the same processing as the in-fridge photographing processing according to the embodiment. After the processing from steps S101 to S104 is executed, the imaging control unit 214 generates the imaging command information described above and outputs it to the imaging device I/F 252 (step S105). At this time, as described in the embodiment, the image capturing device 22 captures an image of the inside of the refrigerator compartment 30, and the image information indicating the image obtained by capturing the image with the image capturing device 22 is transferred to the image storage unit 231. and the image storage unit 231 stores the transferred image information.

Next, the image acquisition unit 2216 acquires the image information stored in the image storage unit 231, and the brightness calculation unit 2217 calculates the average brightness of the entire image indicated by the image information acquired by the image acquisition unit 2216 (step S501).

Subsequently, the light source control unit 212 determines whether or not the average luminance calculated by the luminance calculation unit 2217 is equal to or higher than a preset reference luminance (step S502). Here, when the light source control unit 212 determines that the above-described average luminance is equal to or higher than the reference luminance (step S502: Yes), the processing from step S106 onwards is executed as it is.

On the other hand, when the light source control unit 212 determines that the average brightness is less than the reference brightness (step S502: No), it generates light source control information so that the average brightness is equal to or higher than the preset reference brightness, and transmits the light source control information to the light source I/F 254. Output (step S503). After that, the process of step S105 is executed again.

According to this configuration, the light emission intensity of the light source 24 is adjusted based on the average brightness of the image inside the refrigerator compartment 30 captured by the imaging device 22 . As a result, variations in the average brightness of the image inside the refrigerator compartment 30 obtained by being photographed by the imaging device 22 are suppressed, so that the visibility of the photographed image inside the refrigerator compartment 30 can be improved.

In the in-fridge photography system 2 according to the embodiment, for example, a plurality of light sources other than the light source 24 are provided, the light emission intensity of the light source 24 is constant, and the plurality of light sources other than the light source 24 are turned on according to the position of the imaging device 22. It may be one that switches the state. For example, as in an interior imaging system 3002 of a refrigerator 3001 shown in FIG. The sub-light sources 3024a and 3024b are arranged below the shelves 33 and 34 on the inner wall of the refrigerator compartment 30 on the -X direction side. The sub-light sources 3024c and 3024d are arranged below the shelves 33 and 34 on the inner wall of the refrigerator compartment 30 on the +X direction side.

16, the auxiliary storage section 203 according to this modified example has an image storage section 231, a sub-light source storage section 3232, and an imaging position storage section 233. As shown in FIG. In addition, in FIG. 16, the same reference numerals as in FIG. 3 denote the same configurations as in the embodiment. The lighting device 3026 is connected to the light source 24 and the four sub-light sources 3024a, 3024b, 3024c, and 3024d. 3024d is selectively lit. For example, as shown in FIG. 17, the sub-light source storage unit 3232 stores the identification information of each of the sub-light sources 3024a, 3024b, 3024c, and 3024d to be turned on when the imaging device 22 takes an image, as imaging position information indicating the position of the imaging device 22. is stored in association with Here, the distances between the sub-light sources 3024a, 3024b, 3024c, and 3024d associated with the positions Pos2 and Pos3 are within a preset reference distance from the positions Pos2 and Pos3.

When the open/close determination unit 211 determines that the doors 31 and 32 are closed, the light source control unit 3212 switches the four sub-light sources 3024a, 3024b, 3024c, and 3024d arranged at different positions in the refrigerator compartment 30. The light source control information is generated so as to turn on at least the sub-light sources whose distance from the imaging device 22 is within the reference distance. Specifically, the light source control unit 3212, for example, as shown in FIG. Light source control information is generated based on the sub light source identification information.

For example, assume that the doors 31 and 32 of the refrigerator 3001 are closed, and the imaging device 22 moves to a position Pos2 (see FIGS. 5A and 5B) under the shelf 33 to capture an image. In this case, the light source control unit 3212 refers to the sub light source identification information stored in the sub light source storage unit 3232 to generate light source control information for turning on the light source 24 and the sub light sources 3024a and 3024c. should be output to . Further, it is assumed that the doors 31 and 32 of the refrigerator 3001 are closed, and the imaging device 22 moves to a position Pos3 (see FIGS. 5A and 5B) under the shelf 34 to take an image. In this case, the light source control unit 3212 refers to the sub light source identification information stored in the sub light source storage unit 3232 to generate light source control information for lighting the sub light sources 3024 b and 3024 d together with the light source 24 . should be output to .

Note that in this modification, the light source control unit 3212 controls the sub-light source corresponding to the position of the imaging device 22 among the four sub-light sources 3024a, 3024b, 3024c, and 3024d according to the positions Pos1, Pos2, and Pos3 of the imaging device 22. An example of lighting only the light source has been described. However, the light source control unit 3212 may generate the light source control information so that all of the four sub light sources 3024a, 3024b, 3024c, and 3024d are turned on, for example. In this case, the light source control unit 3212 sets the light emission intensity of the sub light source whose distance from the imaging device 22 is at least within the reference distance to the light emission intensity according to the position of the imaging device 22 so as to turn on the sub light source. Light source control information may be generated.

Further, for example, as in an in-fridge imaging system 4002 of a refrigerator 4001 shown in FIG. may In addition to the light source 24, a light source 4024 positioned above the imaging device 22 and moved together with the imaging device 22 by the moving device 25 may be provided. For example, the doors 31 and 32 of the refrigerator 4001 are closed, and the imaging device 22 moves to a position Pos2 under the shelf 33 or a position Pos3 under the shelf 34 (see FIGS. 5A and 5B). Suppose you take an image. In this case, the control device 4021 may output a light source control signal for lighting the light source 4024 together with the light source 24 to the lighting device 4026 .

According to these configurations, even if there are places in the refrigerating chamber 30 where the light emitted from the light source 24 does not reach sufficiently due to the large number of articles stored in the refrigerating chamber 30, the articles to be photographed by the imaging device 22 can be captured. It can illuminate the area where it is placed. Therefore, variations in the brightness of the image inside the refrigerator compartment 30 obtained by the imaging device 22 are suppressed, so that the appearance of the image can be improved.

The in-fridge photography system 2 according to the embodiment may include an image recognition unit that recognizes the types of articles stored in the refrigerator compartment 30, for example. The refrigerator according to this modification has, for example, the same configuration as that of FIG. For example, as shown in FIG. 18, a lighting device 5026 according to this modified example is connected to a light source 24 and four sub-light sources 3024a, 3024b, 3024c, and 3024d. Based on this, the four sub light sources 3024a, 3024b, 3024c, and 3024d are selectively turned on. In the control device 5021, the CPU 201 reads out the program stored in the auxiliary storage unit 203 into the main storage unit 202 and executes it, whereby the opening/closing determination unit 211, the light source control unit 5212, the movement control unit 213, and the shooting control unit 214 , the transfer unit 215 , the image acquisition unit 5216 and the image recognition unit 5217 . In addition, in FIG. 18, the same reference numerals as in FIG. 3 denote the same configurations as in the embodiment. The auxiliary storage unit 203 also has an image storage unit 231 , a light emission intensity storage unit 232 , an imaging position storage unit 233 , a sub light source storage unit 5234 and a recognition information storage unit 5235 . For example, as shown in FIG. is stored in association with Here, the sub-light source storage unit 5234 stores the four sub-light sources 3024a, 3024b, 3024c, and 3024d in the first group arranged on one side (the −X direction side in FIG. 15A) in the refrigerator compartment 30. The sub-light sources 3024a and 3024b and the second group of sub-light sources 3024c and 3024d arranged on the other side (+X direction side in FIG. 15A) in the refrigerator compartment 30 are classified and stored. The recognition information storage unit 5235 stores template information used when performing image recognition using image information representing an image captured by the imaging device 22 in association with product information representing the corresponding product.

Returning to FIG. 18 , the image acquisition unit 5216 acquires image information indicating an image captured by the imaging device 22 and stored in the image storage unit 231 . The image recognition unit 5217 recognizes the types of articles stored in the refrigerator compartment 30 based on the image information acquired by the image acquisition unit 5216 . The image recognition unit 5217 stores images stored in the refrigerator compartment 30 based on a plurality of types of image information obtained by capturing images with the imaging device 22 while changing the lighting states of the four sub-light sources 3024a, 3024b, 3024c, and 3024d. Recognize the type of goods.

Next, the in-fridge photographing process executed by the control device 5021 according to this modification will be described with reference to FIGS. 20 and 21. FIG. In FIGS. 20 and 21, the same reference numerals as in FIGS. 9 and 10 denote the same processing as the in-fridge photographing processing according to the embodiment. After the processing from steps S101 to S104 is executed, the light source control unit 5212 generates light source control information for turning on at least one of the sub light sources 3024a and 3024b belonging to the first group, and the light source I/ Output to F254 (step S601). Here, the light source control unit 5212 generates the light source control information based on the sub light source identification information belonging to the first group associated with the positions Pos1, Pos2, and Pos3 as shown in FIG. 19, for example. For example, when the imaging device 22 is positioned at the position Pos1, the light source control unit 5212 generates light source control information so as to turn on the sub light sources 3024a belonging to the first group. Also, when the imaging device 22 is positioned at the position Pos2, the light source control unit 5212 generates light source control information so as to turn on both the sub light sources 3024a and 3024b belonging to the first group. At this time, the light source I/F 254 outputs a light source control signal corresponding to the light source control information to the lighting device 5026, and the lighting device 5026 lights at least one of the sub light sources 3024a and 3024b based on the light source control signal. Let

Next, the imaging control unit 214 generates the aforementioned imaging command information and outputs it to the imaging device I/F 252 (step S602). At this time, as described in the embodiment, the image capturing device 22 captures an image of the inside of the refrigerator compartment 30, and the image information indicating the image obtained by capturing the image with the image capturing device 22 is transferred to the image storage unit 231. be done. Then, the image storage unit 231 stores the transferred image information (step S603).

Subsequently, the light source control unit 5212 generates light source control information for turning off the lighted sub light sources 3024a and 3024b, and outputs the light source control information to the light source I/F 254 (step S604). At this time, the light source I/F 254 receives the light source control information and stops outputting the light source control signal to the lighting device 5026, and the lighting device 5026 turns off the sub light sources 3024a and 3024b.

After that, the light source control unit 5212 generates light source control information for turning on at least one of the sub light sources 3024c and 3024d belonging to the second group, and outputs it to the light source I/F 254 (step S605). Here, the light source control section 5212 generates the light source control information based on the sub light source identification information belonging to the second group associated with the positions Pos1, Pos2, and Pos3 as shown in FIG. 19, for example. For example, when the imaging device 22 is positioned at position Pos1, the light source control unit 5212 generates light source control information so as to turn on the sub-light source 3024c belonging to the second group. Further, when the imaging device 22 is positioned at the position Pos2, the light source control section 5212 generates light source control information so as to turn on both the sub light sources 3024c and 3024d belonging to the second group. At this time, the light source I/F 254 outputs a light source control signal corresponding to the light source control information to the lighting device 5026, and the lighting device 5026 lights at least one of the sub light sources 3024c and 3024d based on the light source control signal. Let

Next, the imaging control unit 214 generates the imaging command information described above and outputs it to the imaging device I/F 252 (step S606). At this time, as described in the embodiment, the image capturing device 22 captures an image of the inside of the refrigerator compartment 30, and the image information indicating the image obtained by capturing the image with the image capturing device 22 is transferred to the image storage unit 231. be done. Then, the image storage unit 231 stores the transferred image information (step S607).

Subsequently, the light source control unit 5212 generates light source control information for turning off the lighted sub light sources 3024a and 3024b, and outputs the light source control information to the light source I/F 254 (step S608). At this time, the light source I/F 254 receives the light source control information and stops outputting the light source control signal to the lighting device 5026, and the lighting device 5026 turns off the sub light sources 3024c and 3024d.

After that, the image acquisition unit 5216 acquires image information from the image storage unit 231, and the image recognition unit 5217 determines the type of article stored in the refrigerator compartment 30 based on the image information acquired by the image acquisition unit 5216. is executed (step S609). Here, the image acquisition unit 5216 acquires image information indicating an image obtained by shooting with the sub light sources 3024a and 3024b turned on for each of the positions Pos1, Pos2, and Pos3 of the imaging device 22, the sub light sources 3024c, and image information indicating an image obtained by shooting with the 3024d turned on. Then, the image recognition unit 5217 executes image recognition processing using two types of image information and the template information stored in the recognition information storage unit 5235 for each of the positions Pos1, Pos2, and Pos3 of the imaging device 22. . Also, the image recognition unit 5217 causes the main storage unit 202 to store article information obtained by image recognition processing. Next, the processing after step S109 is executed.

Further, in the image transfer process according to this modification, as shown in FIG. 21, the processes of steps S201 to S203 are executed, and it is assumed that the transfer unit 215 has received the transfer permission information from the server 5 (step S203: Yes). In this case, after transferring the image information to the server 5 (step S204), the transfer unit 215 transfers the product information obtained by the image recognition process stored in the main storage unit 202 to the server 5 (step S701). . After that, the processes after step S205 are executed.

In this modified example, the refrigerator has four sub-light sources 3024a, 3024b, 3024c, and 3024d. However, the number of sub-light sources is not limited to four. , or five or more. Also, in this modification, an example has been described in which the four sub-light sources 3024a, 3024b, 3024c, and 3024d are classified into two groups, the first group and the second group, but the number of classifications is not limited to two. may be classified into three or more groups. In this case, the light source control unit 5212 sequentially turns on the sub light sources belonging to each of the three or more groups, and each time the imaging control unit 214 turns on the sub light sources belonging to each of the three or more groups, the imaging device 22 may be used to image the inside of the refrigerator compartment 30 .

According to this configuration, when the image recognition unit 5217 captures an image of the inside of the refrigerator compartment 30 with the imaging device 22, image information corresponding to two types of images in which the items stored in the refrigerator compartment 30 are illuminated differently. Image recognition processing is executed based on. As a result, the recognition accuracy in the image recognition process can be improved, so the reliability of the product information transferred to the server 5 can be improved.

In the embodiment, an example in which the imaging device 22 is positioned at three locations has been described, but the number of positions of the imaging device 22 is not limited to three, and may be two or four. or more. Moreover, in the embodiment, the image captured by the imaging device 22 is not limited to a still image, and may be a moving image. For example, when the transfer speed of the image captured by the imaging device 22 to the image storage unit 231 is high, the moving image is captured by the imaging device 22 without determining (fixing) the imaging position while moving the imaging device 22 . You may make it In this case, the emission intensity information stored in the emission intensity storage unit 232 and the sub-light source identification information stored in the sub-light source storage unit 3232 are associated with an imaging range including any one of the imaging positions Pos1, Pos2, and Pos3. The light source controllers 212 and 3212 generate light source control information based on the range in which the device 22 moves.

Further, in the embodiment, when the imaging control unit 214 acquires load amount information indicating the load amount of the CPU 201 and determines that the load amount indicated by the acquired load amount information is equal to or less than a preset reference load amount, The imaging command information may be output to the imaging device I/F 252 . As a result, the time from the start of photographing of the interior of the refrigerator compartment 30 to the completion of photographing can be shortened.

Also, in the embodiment, an example in which the moving device 25 moves the imaging device 22 in the vertical direction has been described, but the moving direction of the imaging device 22 is not limited to the vertical direction. For example, the moving device 25 may move the imaging device 22 in a direction intersecting the vertical direction.

In the embodiment, when the open/close determination unit 211 determines that the doors 31 and 32 are open, the light source control unit 212 sets the light emission intensity of the light source 24 to the preset maximum light emission intensity and turns on the light source 24 . An example has been described in which the light source control information for controlling the light source is generated and output to the light source I/F 254 . However, for example, when the light source control unit 212 determines that the doors 31 and 32 are open by the open/close determination unit 211, the light source control unit 212 reduces the light emission intensity of the light source 24 to a value less than the preset maximum light emission intensity. Light source control information for lighting the light source 24 at the set emission intensity may be generated and output to the light source I/F 254 .

In the embodiment, an example in which the in-fridge photography system 2 is provided in the refrigerator 1 has been described, but the article storage in which the in-fridge photography system 2 is provided is not limited to the refrigerator 1, and may be other than the refrigerator 1. It may be provided in an article storage of the type.

Various functions of the control devices 21, 2021, 3021, 4021, and 5021 according to the present invention may be realized by software, firmware, or a combination of software and firmware. In this case, the software or firmware is written as a program, and the program is stored on computer-readable media such as flexible discs, CD-ROMs (Compact Disc Read Only Memory), DVDs (Digital Versatile Discs) and MOs (Magneto-Optical Discs). By storing and distributing the program in a recording medium, and reading and installing the program in the computer, a computer capable of realizing each function described above may be configured. When each function is shared between an OS (Operating System) and an application, or when the OS and an application cooperate with each other, only portions other than the OS may be stored in the recording medium.

Furthermore, it is also possible to superimpose each program on a carrier wave and distribute it via a communication network. For example, the program may be posted on a bulletin board (BBS, Bulletin Board System) on a communication network and distributed via the network. Then, these programs may be activated and executed in the same manner as other application programs under the control of the OS so that the above-described processing can be executed.

INDUSTRIAL APPLICATION This invention is suitable as a control apparatus which controls a refrigerator and a refrigerator which has the function to image|photograph the goods in a refrigerator compartment.

1, 3001, 4001 Refrigerator 2, 2002, 3002, 4002, 5002 In-fridge photography system 3 Router 4 Terminal device 5 Server 21, 2021, 3021, 4021, 5021 Control device 22 Imaging device 23 Open/close sensor , 24 light source, 25 moving device, 26, 3026, 4026, 5026 lighting device, 100 in-fridge image viewing system, 201, 401, 501 CPU, 202, 402, 502 main storage unit, 203, 403, 503 auxiliary storage unit, 204 wireless module, 209, 409, 509 bus, 211 open/close determination unit, 212, 3212, 5212 light source control unit, 213 movement control unit, 214 photographing control unit, 215 transfer unit, 231, 431, 531 image storage unit, 232 light emission intensity storage unit, 233 imaging position storage unit, 251 sensor I/F, 252 imaging device I/F, 253 mobile device I/F, 254 light source I/F, 404, 504 communication unit, 405 display unit, 406 input unit, 411 operation reception unit 412, 511 image acquisition unit 413 display control unit 512 state determination unit 513 image transmission unit 514 authentication unit 532 authentication information storage unit 2216, 5216 image acquisition unit 2217 luminance calculation unit 3232 , 5234 sub-light source storage unit 2501 motor 2502 driver 3024a, 3024b, 3024c, 3024d sub-light source 5217 image recognition unit 5235 recognition information storage unit C1, C2, C3: intermediate position, NT network, Pos1, Pos2, Pos3 position

Claims (9)

A lighting device that turns on or off a light source that illuminates the interior of an article storage based on light source control information, an imaging device that captures the interior of the storage, and a movement of the imaging device within the storage based on movement control information. A control device that controls a moving device that causes
an opening/closing determination unit that determines whether the door of the article storage is in an open state or a closed state;
a movement control unit that generates movement control information for causing the imaging device to move in the refrigerator and outputs the movement control information to the movement device;
When the opening/closing determination unit determines that the door is open, light source control information for lighting the light source at a predetermined light emission intensity is generated and sent to the lighting device. light source control information for turning on the light source at a light emission intensity corresponding to the position of the imaging device when the open/close determination unit determines that the door is closed; a light source control unit that generates and outputs to the lighting device;
an image acquisition unit that acquires image information indicating an image captured by the imaging device;
a brightness calculation unit that calculates average brightness in at least part of an image indicated by the image information acquired by the image acquisition unit;
The light source control unit changes the light emission intensity of the light source so that the average luminance becomes equal to or higher than a preset reference luminance when the open/close determination unit determines that the door is closed. generate information,
Control device. The movement control unit generates movement control information for moving the imaging device to one of a plurality of preset positions within the refrigerator.
A control device according to claim 1 . The light source control unit is configured to increase the light emission intensity of the light source as the distance between the light source and the imaging device increases when the open/close determination unit determines that the door is closed. generate light source control information;
3. A control device according to claim 1 or 2. The lighting device further turns on or off a plurality of sub-light sources arranged at different positions in the refrigerator based on the light source control information,
When the opening/closing determination unit determines that the door is closed, the light source control unit determines whether the distance between at least the imaging device, among the plurality of sub-light sources, is within a preset reference distance. generating the light source control information so as to turn on the light source by setting the light emission intensity of a certain sub-light source to the light emission intensity according to the position of the imaging device;
A control device according to any one of claims 1 to 3. The lighting device further turns on or off a plurality of sub-light sources arranged at different positions in the refrigerator based on the light source control information,
an image acquisition unit that acquires image information indicating an image captured by the imaging device;
an image recognition unit that recognizes the type of article stored in the warehouse based on the image information acquired by the image acquisition unit;
The image recognition unit recognizes types of articles stored in the warehouse based on a plurality of types of image information obtained by photographing with the imaging device while changing lighting states of the plurality of sub-light sources.
A control device according to any one of claims 1 to 4. a lighting device that turns on or off a light source that illuminates the interior of an article storage cabinet based on light source control information;
an image capturing device for capturing an image of the inside of the refrigerator;
a moving device that moves the imaging device in the refrigerator based on movement control information;
An in-fridge photography system comprising a control device that controls the lighting device, the imaging device, and the moving device,
The control device is
an opening/closing determination unit that determines whether the door of the article storage is in an open state or a closed state;
a movement control unit that generates movement control information for causing the imaging device to move in the refrigerator and outputs the movement control information to the movement device;
When the opening/closing determination unit determines that the door is open, light source control information for lighting the light source at a predetermined light emission intensity is generated and sent to the lighting device. light source control information for turning on the light source at a light emission intensity corresponding to the position of the imaging device when the open/close determination unit determines that the door is closed; a light source control unit that generates and outputs to the lighting device;
an image acquisition unit that acquires image information indicating an image captured by the imaging device;
a brightness calculation unit that calculates average brightness in at least part of an image indicated by the image information acquired by the image acquisition unit;
The light source control unit changes the light emission intensity of the light source so that the average luminance becomes equal to or higher than a preset reference luminance when the open/close determination unit determines that the door is closed. generate information,
In-house photography system. a lighting device that turns on or off a light source that illuminates the refrigerator interior based on light source control information; an imaging device that photographs the interior of the refrigerator; and a moving device that moves the imaging device within the refrigerator interior based on movement control information. , a control device for controlling the lighting device, the imaging device, and the moving device;
The control device is
an opening/closing determination unit that determines whether the door of the article storage is in an open state or a closed state;
a movement control unit that generates movement control information for causing the imaging device to move in the refrigerator and outputs the movement control information to the movement device;
generating and outputting light source control information for turning on the light source at a predetermined light emission intensity when the open/close determination unit determines that the door is open; When the opening/closing determination unit determines that the door is closed, light source control information is generated and output for lighting the light source by setting the light emission intensity of the light source to the light emission intensity corresponding to the position of the imaging device. a light source control unit that
an image acquisition unit that acquires image information indicating an image captured by the imaging device;
a brightness calculation unit that calculates average brightness in at least part of an image indicated by the image information acquired by the image acquisition unit;
The light source control unit changes the light emission intensity of the light source so that the average luminance becomes equal to or higher than a preset reference luminance when the open/close determination unit determines that the door is closed. generate information,
Goods storage. determining whether the item storage door is open or closed;
controlling the light emission intensity of a light source in the article storage to a preset light emission intensity when it is determined that the door is open;
when it is determined that the door is in the closed state, moving an imaging device for photographing the inside of the refrigerator within the refrigerator;
setting the light emission intensity of the light source according to the position of the imaging device, and photographing the interior of the article storage with the imaging device;
obtaining image information indicating an image captured by the imaging device;
calculating average luminance in at least part of an image indicated by the obtained image information;
and changing the emission intensity of the light source so that the average luminance is equal to or higher than a preset reference luminance when it is determined that the door is closed.
How to shoot inside the warehouse. the computer,
an open/close determination unit that determines whether the door of the goods storage is open or closed;
Movement control information for moving the inside of the article storage is generated by an imaging device for photographing the interior of the article storage, and is output to a moving device for moving the imaging device within the storage based on the movement control information. movement control unit to
When the open/close determination unit determines that the door is open, light source control information is generated for lighting the light source at a predetermined light emission intensity for illuminating the inside of the refrigerator. and outputs the light source to a lighting device that turns on or off the light source based on the light source control information, and when the opening/closing determination unit determines that the door is closed, the light emission intensity of the light source is adjusted to the position of the imaging device. a light source control unit that generates light source control information for lighting the light source with the corresponding light emission intensity and outputs the light source control information to the lighting device;
an image acquisition unit that acquires image information indicating an image captured by the imaging device;
a brightness calculation unit that calculates average brightness in at least part of an image indicated by the image information acquired by the image acquisition unit;
A program for functioning as
The light source control unit changes the light emission intensity of the light source so that the average luminance becomes equal to or higher than a preset reference luminance when the open/close determination unit determines that the door is closed. generate information,
program.

Images