JP6036294B2 - Information providing system, image display apparatus, information providing method, and program - Google Patents

Description

  The present invention provides an information providing system, an image display device, an information providing method, and an information providing device that provide information from the image display device to the information processing device using an image display device having a display screen and an information processing device that captures the display screen. Regarding the program.

  Opportunities for users to view a wide variety of images and moving images (represented by the word “image” in the following description) through TV and Internet HP are increasing. Such images contain various information, and there is a demand from the image provider side and from the user that the user obtains information related to the image and refers to it in the next action. It is growing. As an example, when an introductory program about a restaurant is broadcast on a TV, the broadcaster also provides the viewer with information about the restaurant, for example, the URL (Uniform Resource Locator) of the restaurant's HP, thereby introducing the introductory program itself. On the other hand, if the user who is a viewer is interested in a restaurant introduced in this introduction program, if the information about the restaurant can be easily obtained, the user can also watch the introduction program next time Increased motivation.

  However, although both the image provider side and the user side have requested for the provision and acquisition of information related to the image as described above, as a means for providing the image information from the image provider to the user, for example, In the case of the above-mentioned example, the URL of the restaurant's HP is described in the image, or the link of the restaurant's HP URL is provided to the HP of the introduction program itself. It was. Therefore, the user takes note of the URL of the restaurant's HP and manually inputs this URL into the PC or the like, or the user activates the PC or the like separately to access the introduction program's HP and enter the URL of the restaurant's HP In any case, the user is forced to perform complicated operations. That is, it is difficult for the user to directly obtain information related to the image.

  As a method in which the user directly obtains information related to the image, it can be considered that the information itself is embedded in the image. As a technique for embedding information in such an image, there is known a digital watermark (watermark) technique developed for the purpose of clarifying the origin of an image without unauthorized copying (see, for example, Patent Documents 1 and 2). ).

JP-T-2006-517068 JP 2004-56414 A

  However, in the above-described conventional digital watermark technology, the embedded information may be perceived by the user as noise for the image, which may affect viewing of the image. In addition, the conventional digital watermark technology requires a large amount of work for analyzing information by restoring an image, and there are certain barriers to application to devices owned by general users.

  The present invention has been made in view of the above-described problems, and provides an information providing system, an image display device, an information providing method, and a program capable of transmitting related information without greatly affecting user image viewing. Is one of its purposes. Another object of the present invention is to provide an information providing system, an image display device, an information providing method, and a program capable of acquiring the transmitted related information by a simple operation.

  The present invention includes an image display device having a display unit that displays an image on a display screen, and an information processing device having an image pickup unit that captures the display screen and outputs a captured image, from the image display device to the information processing device. It is applied to an information provision system that provides information. Then, the image display device divides the display screen into four regions, and changes at least one of the luminance value or the color difference value of the divided image displayed in each of the four divided regions based on the information to be provided. An information embedding unit that embeds unit information constituting information in an image is provided, an image in which unit information is embedded by an information embedding unit is displayed on a display screen, and an image output by an imaging unit is output to an information processing device By providing an information extraction unit that compares four divided captured images corresponding to regions in the images, extracts unit information according to the comparison result, and obtains information, thereby solving at least one of the above-described problems Yes.

  The information embedding unit embeds unit information constituting the information in the image by changing at least one of the luminance value or the color difference value of the divided image displayed in each of the four divided regions. Among the captured images output from the above, four divided captured images corresponding to the regions are compared, and unit information is extracted according to the comparison result to obtain information. As described above, in the present invention, information is provided from the image display apparatus to the information processing apparatus by changing at least one of the luminance value and the color difference value of the divided image.

  Here, the information processing apparatus is provided with a communication unit that communicates with an external network, and the communication unit transmits information extracted by the information extraction unit to a server that exists in the external network, and the server adds information corresponding to this information. It is preferable to transmit information to the information processing apparatus.

  Further, the information embedding unit changes unit information constituting the image by changing at least one of the luminance value or the color difference value of the divided image displayed in one of the regions as a reference region and the other region. It is preferable to embed in.

  Furthermore, it is preferable that the information processing apparatus is provided with an area determination unit that determines a divided captured image corresponding to the reference area by detecting a change in luminance value or color difference value of the divided captured image. Preferably, unit information is extracted based on a change in luminance value or color difference value of the divided captured image corresponding to the reference region, which is determined by the region determination unit, to obtain information.

  Furthermore, the information embedding unit preferably embeds unit information constituting the information in the image by changing the luminance value of the divided image displayed in each of the four divided areas. When a unit image constituting information is embedded in the image by changing the luminance value of the image, the information extraction unit further uses the divided captured image corresponding to the reference region determined by the region determination unit as a reference. It is preferable to detect a divided captured image having an intermediate luminance value from the other three divided captured images and extract information based on the divided captured image having the intermediate luminance value.

  Furthermore, it is preferable that the information embedding unit embeds unit information constituting the information in the image by changing the luminance value at the center of the divided image. Alternatively, the information embedding unit preferably embeds information including a plurality of unit information in the image by periodically changing the luminance value of the divided image. Furthermore, it is preferable that the communication unit converts the information obtained by the information extraction unit and transmits it to the server.

  Then, the information embedding unit divides the display screen into a plurality of regions each having four regions as a unit, and the brightness of the divided image displayed in each of the four divided regions based on the information to be provided for each unit. It is preferable to embed unit information constituting the information in the image by changing at least one of the value and the color difference value.

  In addition, the present invention is applied to an image display device that includes a display unit that displays an image on a display screen and provides information to an information processing device that images the display screen. In this image display device, the display screen is divided into four areas, and at least one of the luminance value or the color difference value of the divided image displayed in each of the four divided areas is changed based on the information to be provided. Thus, at least one of the above-described problems is solved by providing an information embedding unit that embeds unit information constituting information in an image.

  Furthermore, the present invention is applied to an information providing method for providing information from an image display apparatus to an information processing apparatus using an image display apparatus that displays an image on a display screen and an information processing apparatus that captures the display screen. Then, the display screen is divided into four areas, and information is configured by changing at least one of the luminance value or the color difference value of the divided image displayed in each of the four divided areas based on the information to be provided. A step of embedding unit information in an image, a step of displaying an image in which unit information is embedded on a display screen, a step of capturing a display screen to obtain a captured image, and a captured image obtained by capturing the display screen Among these, at least one of the above-mentioned problems is solved by providing a step of comparing four divided captured images corresponding to regions and extracting unit information according to the comparison result to obtain information.

  The present invention is applied to a program executed by a computer that includes a display unit that displays an image on a display screen and provides information to an information processing apparatus that images the display screen. When this program is executed by the computer, at least a luminance value or a color difference value of a divided image displayed in each of the four divided areas based on the step of dividing the display screen into four areas and the information to be provided. By changing one of them, at least one of the above-described problems is solved by causing the computer to execute the step of embedding unit information constituting the information in the image.

  As described above, in the information providing system according to the present invention, information is configured by changing at least one of the luminance value or the color difference value of the divided image displayed in the area obtained by dividing the display screen of the image display device into four. Since the unit information is embedded in the image, it is unlikely that the user can visually recognize that the unit information is embedded even if the user is viewing an image in which the unit information is embedded. Accordingly, it is possible to provide an information providing system, an image display apparatus, an information providing method, and a program that can transmit related information without greatly affecting the user's image appreciation.

  Furthermore, in the information providing system of the present invention, information is acquired by comparing the divided captured image data corresponding to the areas, so even if the imaging capability of the information processing device is low, the information is transmitted from the image display device. Information providing system, image display device, information providing method, and program capable of easily and surely obtaining the received information in the information processing apparatus and thereby obtaining the transmitted related information with a simple operation Can be provided.

It is a block diagram which shows schematic structure of the information provision system which is one Embodiment of this invention. It is a block diagram which shows schematic structure of the image display apparatus applied to the information provision system of one Embodiment. It is a block diagram which shows schematic structure of the information processing apparatus applied to the information provision system of one Embodiment. It is a functional block diagram which shows the function structure of the information provision system of one Embodiment. It is a flowchart for demonstrating an example of operation | movement of the image display apparatus applied to the information provision system of one Embodiment. It is a flowchart for demonstrating an example of operation | movement of the information processing apparatus applied to the information provision system of one Embodiment. It is a figure for demonstrating an example of the information embedding procedure in the information provision system of one Embodiment. It is a figure for demonstrating the other example of the information embedding procedure in the information provision system of one Embodiment. It is a figure for demonstrating the other example of the information embedding procedure in the information provision system of one Embodiment.

(One embodiment)
Hereinafter, embodiments of an information providing system of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a schematic configuration of an information providing system according to an embodiment of the present invention. In this figure, reference numeral 1 denotes an information providing system according to the present embodiment. The information providing system 1 includes an image display device 2 and an information processing device 3.

  The image display device 2 includes a display screen 2a, and displays an image on the display screen 2a. The image display device 2 is a display such as a TV. The information processing device 3 captures the captured image by capturing the display screen 2a of the image display device 2. The information processing apparatus 3 is a digital still camera, for example. The image display device 2 is connected to the router 4 via the network cable 6 so that wired communication is possible between the image display device 2 and the router 4, while the information processing device 3 is connected to the router 4. Wireless communication is possible. As a result, both the image display device 2 and the information processing device 3 can be connected to the Internet 5 as an external network via the router 4. An external server 7 is connected to the Internet 5, so that the image display device 2 and the information processing device 3 can transmit and receive information to and from the external server 7 via the Internet 5.

(Configuration of image display device)
FIG. 2 is a block diagram showing a schematic configuration of the image display device 2 used in the information providing system 1 of the present embodiment. 2, the image display apparatus 2 according to the present embodiment includes a CPU (Central Processing Unit) 10, a ROM (Read Only Memory) 11, a RAM (Random Access Memory) 12, an input interface (I / F) 13, and a display unit 14. USB (Universal Serial Bus) interface (I / F) 15, HDMI (High-Definition Multimedia Interface) interface (I / F) 16, network interface (I / F) 17, and HDD (Hard Disk Drive) unit 18 These are connected to each other by a common bus.

  The CPU 10 controls the entire image display device 2 by executing a program such as firmware stored in the ROM 11 described later. The CPU 10 also operates as each functional unit as shown in FIG. 4 by executing a program stored in the ROM 11. The operation of each functional unit shown in FIG. 4 will be described later. The ROM 11 stores programs such as the firmware described above. The RAM 12 functions as a work memory for the image display device 2 and stores programs, data, and the like that are temporarily used during the operation of the image display device 2 including the CPU 10.

  An input device 30 is connected to the input interface 13 and receives an input signal input by the user operating the input device 30. Examples of the input device 30 include a remote controller, a keyboard, and a mouse.

  The display unit 14 includes a liquid crystal driver 14a and a liquid crystal panel 14b. The liquid crystal panel 14 b is provided with its display surface exposed on the surface of the image display device 2. When the data constituting the display screen 2a is supplied from the CPU 10 to the liquid crystal driver 14a, the liquid crystal driver 14a drives the liquid crystal panel 14b to display the desired display screen 2a on the display surface of the liquid crystal panel 14b.

  The USB interface 15 includes a USB connector (not shown), and reads / writes data from / to the USB device 31 connected to the USB connector directly or via a USB cable. Examples of the USB device 31 include a USB flash memory and a digital still camera.

  The HDMI interface 16 includes an HDMI connector (not shown), and performs an AV stream (video signal and audio signal) input / output operation with the HDMI device 32 connected to the HDMI connector via an HDMI cable (not shown). To do. Examples of the HDMI device 32 include a recording device capable of recording program data such as terrestrial digital television broadcast waves, and a tuner device capable of receiving terrestrial digital television broadcast waves.

  The network interface 17 includes a network connector (not shown), and the router 4 is connected to the network connector via the network cable 6. By connecting the router 4 to the Internet 5, data can be exchanged with an external network. Send and receive. The network interface 17 performs wired communication based on, for example, IEEE (The Institute of Electrical and Electronics Engineers, Inc.) 802.3 standard. Alternatively, the network interface 17 may perform wireless communication based on the IEEE 802.11 standard, or may perform either wired / wireless communication.

  The HDD unit 18 includes an HDD interface (I / F) 18a and an HDD 18b. The HDD 18b includes a disk that is a recording medium, a rotating unit that rotates the disk, and a head unit that reads / writes data from / to the disk (all not shown). When there is a data read / write command to the HDD 18b, the HDD interface 18a controls the entire HDD 18b, performs data read / write control, and outputs the read data. The HDD 18b stores transmission information 33 to be transmitted and provided to the information processing apparatus 3.

(Schematic configuration of information processing device)
FIG. 3 is a block diagram illustrating a schematic configuration of the information processing apparatus 3 used in the information providing system 1 of the present embodiment. 3, the information processing apparatus 3 includes a CPU 20, a ROM 21, a RAM 22, a wireless communication interface (I / F) 23, an imaging unit 24, an input unit 25, and an HDMI interface (I / F) 26, which are common buses. Are connected to each other.

  The CPU 20 controls the information processing apparatus 3 as a whole by executing a program such as firmware stored in the ROM 21 described later. The CPU 20 also operates as each functional unit as shown in FIG. 4 by executing a program stored in the ROM 21. The operation of each functional unit shown in FIG. 4 will be described later. The ROM 21 stores programs such as the above-described firmware and various setting data. The RAM 22 operates as a work memory of the information processing apparatus 3 and temporarily stores various programs and data.

  The wireless communication interface 23 performs wireless communication with the router 4 in conformity with, for example, the IEEE 802.11 standard. The wireless module 23 can operate the information processing apparatus 3 as a client apparatus that performs communication via the router 4.

  The imaging unit 24 includes a camera module 24a and an image processing unit 24b. The camera module 24a includes an imaging optical system and a drive unit that controls and drives the imaging optical system (all not shown). The optical axis of the imaging optical system extends outward from the information processing apparatus 3, and is configured to be able to image the outside of the information processing apparatus 3, particularly the display screen 2a of the image display apparatus 2. The image processing unit 24b performs various image processing operations on the subject image captured by the camera module 24a, and finally outputs captured image data.

  The input unit 25 includes an input interface (I / F) 25a, an input instruction unit 25b, and a USB interface (I / F) 25c. An input instruction unit 25b is connected to the input interface 25a and receives an input signal input by the user operating the input instruction unit 25b. An example of the input instruction unit 25b is an information capture instruction button. The USB interface 25c includes a USB connector (not shown), and reads / writes data from / to the USB device 31 connected to the USB connector directly or via a USB cable. An example of the USB device 31 is a USB flash memory.

  The HDMI interface 26 includes an HDMI connector (not shown), and performs an AV stream (video signal and audio signal) input / output operation with the HDMI device 32 connected to the HDMI connector via an HDMI cable (not shown). To do. Examples of the HDMI device 32 include a TV and a monitor.

(Functional configuration of information provision system)
Next, FIG. 4 is a functional block diagram showing a functional configuration of the information providing system 1 of the present embodiment. In FIG. 4, the image display device 2 includes a control unit 40, a storage unit 41, an input unit 42, and a display unit 14, and the information processing device 3 includes a control unit 50, an imaging unit 24, and a communication unit 51.

  The control unit 40 includes an image monitoring unit 43, a region determination unit 44, and an information embedding unit 45. The image monitoring unit 43 monitors the luminance value and color difference value of the divided image displayed at that time in each of the four areas of the display screen 2a. Based on the monitoring result from the image monitoring unit 43, the region determining unit 44 determines in which of the four regions of the display screen 2a the unit information embedded by the information embedding unit 45 is embedded. More specifically, the region determination unit 44 determines one region as a reference region from among the four divided regions based on the monitoring result by the image monitoring unit 43, and preferably includes three regions other than the reference region. Decides to embed unit information in any one area.

  The information embedding unit 45 changes the luminance value or color difference value, preferably the luminance value, of the divided image of the region determined by the region determining unit 44 based on the transmission information 33 stored in the storage unit 41. Embed unit information in the image. More specifically, the information embedding unit 45 changes the luminance value or the color difference value, preferably the luminance value of the divided image of three regions other than the reference region determined by the region determining unit 44, preferably any one region. Unit information is embedded in the image. More specifically, the information embedding unit 45 changes the luminance value of the divided image of three regions other than the reference region determined by the region determining unit 44, preferably any one of the divided images. Unit information is embedded in the image by changing the luminance value at the center. More specifically, the information embedding unit 45 periodically changes the luminance values of the divided images of three regions other than the reference region determined by the region determining unit 44, preferably any one of the plurality of regions. Information consisting of unit information is embedded in the image. Detailed operations of the image monitoring unit 43, the region determination unit 44, and the information embedding unit 45 will be described later.

  The storage unit 41 stores various information at least temporarily and stores transmission information 33 to be transmitted and provided to the information processing device 3. In this embodiment, when the transmission information 33 is transmitted to the information processing device 3 as described later, the information received by the information processing device 3 is transmitted to the external server 7, and the external server 7 converts the received information into the received information. Corresponding additional information is returned to the information processing apparatus 3. Therefore, the transmission information 33 may be any information that allows the external server 7 to uniquely recognize each transmission information 33, and may be, for example, a fixed-length code string (bit string) or a predetermined size. It may be data constituting an image having a thickness. Of course, the length and configuration of the transmission information 33 are not limited, and information of a known configuration can be suitably used.

  The input unit 42 receives information input to the image display device 2, in particular, input of image data to be displayed on the display screen 2 a of the display unit 14, and sends it to the control unit 40 and the storage unit 41. The display unit 14 displays an image on the display screen 2 a based on the image data input from the control unit 40 and the storage unit 41 to the display unit 14. The image displayed on the display screen 2 a by the display unit 14 is an image in which unit information is embedded by the information embedding unit 45 of the control unit 40.

  The control unit 50 includes an area determination unit 52 and an information extraction unit 53. The area determination unit 52 determines which of the areas obtained by dividing the display screen 2 a of the image display device 2 into four is the reference area based on the captured image captured by the imaging unit 24. More specifically, the area determination unit 52 includes luminance values or color difference values of four divided captured images corresponding to a region obtained by dividing the display screen 2a of the image display device 2 into four of the captured images output by the imaging unit 24. Preferably, a divided captured image corresponding to the reference region is determined by detecting a change in luminance value.

  Based on the determination result of the region determination unit 52, the information extraction unit 53 compares four divided captured images corresponding to the region obtained by dividing the display screen 2a of the image display device 2 into four of the captured images output by the imaging unit 24. Then, unit information is extracted according to the comparison result to obtain information. More specifically, the information extraction unit 53 extracts unit information based on the luminance value or color difference value of the divided captured image corresponding to the reference region, preferably the change in luminance value, determined by the region determination unit 52. get information. More specifically, the information extraction unit 53 uses the divided captured image corresponding to the reference region determined by the region determination unit 52 as a reference, and divided imaging having an intermediate luminance value among the other three divided captured images. An image is detected, and information is extracted based on the divided captured image having the intermediate luminance value. Detailed operations of the area determination unit 52 and the information extraction unit 53 will be described later.

  The imaging unit 24 captures the display screen 2 a of the image display device 2 and outputs the captured image to the control unit 50. The communication unit 51 is configured to be able to wirelessly communicate with the router 4 and transmits the information extracted by the information extraction unit 53 to the external server 7 existing in the Internet 5 via the router 4, and the external server 7 adds information corresponding to this information. When you send information, you receive it. Further, the communication unit 51 may convert the information obtained by the information extraction unit 53 and transmit it to the external server 7.

  In the above configuration, the control unit 40 is mainly configured by the CPU 10, the storage unit 41 is mainly configured by the ROM 11, the RAM 12, and the HDD unit 18, the input unit 42 is mainly configured by the HDMI interface 16, and the control unit 50 is The communication unit 52 is mainly configured by the wireless communication interface 23.

(Operation of information provision system)
Next, the operation of the information providing system of this embodiment will be described with reference to the flowcharts of FIGS.

  FIG. 5 is a flowchart for explaining the operation of the image display device 2 used in the information providing system of the present embodiment. The operation illustrated in the flowchart of FIG. 5 is started when the image display apparatus 2 performs an information transmission operation on the information processing apparatus 3.

  First, in step S <b> 1, the control unit 40 of the image display device 2 selects the transmission information 33 to be transmitted in the current operation from the transmission information 33 stored in the storage unit 41. read out. Next, in step S2, the control unit 40 encodes the transmission information selected in step S1. In this embodiment, as will be described later, a binary code string consisting of “0” and “1” is transmitted to the information processing apparatus 3 as information. Each of the codes “0” and “1” is unit information constituting the information. The method for encoding the transmission information 33 is arbitrary, and a known encoding method can be suitably employed. If the transmission information 33 is composed of a code string (bit string), the operation in step S2 may be omitted.

  Next, in step S3, the image monitoring unit 43 of the control unit 40 monitors the image data to be displayed on the display screen 2a of the display unit 14 input to the input unit 42, and the region determination unit 44 performs the image monitoring unit. Based on the monitoring result of 43, an area in which a code as unit information is to be embedded is determined.

  In the image display device 2 of the present embodiment, as will be described later, the information embedding unit 45 of the control unit 40 embeds unit information in a divided image displayed in one of the four divided areas of the display screen 2a. Details of region division and region determination will be described with reference to FIG. As shown in FIG. 7, the image monitoring unit 43 of the image display device 2 of the present embodiment has at least luminance values and color difference values of the four divided images 61 displayed in the four regions 60 obtained by dividing the display screen 2 a into four. Monitor one side. The region determination unit 44 then displays one region 60 in which the divided image 61 suitable for unit information embedding is displayed based on at least one of the luminance value and the color difference value of the four divided images 61 monitored by the image monitoring unit 43. To decide.

  The method of determining the region 60 by the region determining unit 44 is arbitrary. As an example, when the information embedding unit 45 embeds unit information by changing the luminance value of the divided image 61, the luminance value of the entire divided image 61 is uniform. In addition, a method of selecting the region 60 in which the divided image 61 is displayed so that even if the luminance value is changed, it is difficult for the user to visually recognize the change in the luminance value. On the other hand, when the information embedding unit 45 embeds unit information by changing the color difference value of the divided image 61, conversely, the change of the color difference value in the entire divided image 61 is large, and the user can change the color difference value. A method of selecting the divided image 61 that makes it difficult to visually recognize the color difference value change is preferable.

  Next, in step S <b> 4, the information embedding unit 45 of the control unit 40 embeds unit information in the area 60 determined by the area determining unit 44. Details of unit information embedding by the region determination unit 44 will be described with reference to FIGS. As shown in FIG. 7, it is assumed that one region 60 in which a divided image 61 suitable for unit information embedding determined by the region determining unit 44 is displayed is an upper right region in the drawing. The information embedding unit 45 determines a reference area (upper left area 60 in the example shown in FIG. 7) as a reference when the unit information is read by the information processing apparatus 3 among the remaining three areas 60, and this reference area It is determined that neither the luminance value nor the color difference value of the divided image 61 displayed at 60 is changed. Next, the information embedding unit 45 displays at least one of the luminance value and the color difference value of the divided image 61 displayed in three regions other than the reference region 60 (upper right, lower right, and lower left regions 60 in the example illustrated in FIG. 7). To change. At this time, the degree of change of the luminance value and the color difference value is determined by setting at least one of the luminance value and the color difference value of the divided image 61 in the region in which the unit information is embedded (upper right region 60 in the example shown in FIG. 7) as the middle. At least one of the luminance value and the color difference value of the divided image 61 displayed in the two areas is changed. FIG. 7 shows an example of changing the luminance value. In this case, the luminance value of the divided image 61 displayed on one side (lower left region 60 in the example shown in FIG. 7) is increased, and the other (in the example shown in FIG. 7). It is determined that the luminance value of the divided image 61 displayed in the lower right area 60) is to be lowered. Even when the color difference value is changed, it is determined that the color difference value of the divided image 61 displayed in one of the regions 60 is increased and the color difference value of the divided image 61 displayed in the other area is decreased. The extent to which at least one of the luminance value and the color difference value of the divided image 61 displayed in these two regions 60 is changed is determined in advance in consideration of the accuracy of unit information reading in the information processing apparatus 3 and the like.

In the example illustrated in FIG. 7, the luminance value of the divided image 61 is changed. When the luminance values of the divided images 61 displayed in the three regions 60 other than the reference region 60 are arranged in order,
Lower right region 60> Upper right region 60 (unit information embedding region)> Lower left region 60
It becomes. It should be noted that the magnitude relationship between the luminance value and the color difference value of the divided image 61 displayed in the reference area 60 and the other three areas 60 is determined by the rules described later.

Furthermore, the information embedding unit 45 embeds unit information to be embedded according to the following rules. That is,
M> n: code “0”
M <n: code “1”
M = n: indefinite where m: at least one of the luminance value and the color difference value of the divided image 61 displayed in the reference area 60
n: Displayed in the region 60 in which the unit information is to be embedded, based on the sign of the unit information to be embedded, so as to be at least one of the luminance value and the color difference value of the divided image 61 displayed in the region 60 in which the unit information is to be embedded. It is determined that at least one of the luminance value and the color difference value of the divided image 61 to be changed is changed. In addition, the value of | n | − | m | (where | x | represents the absolute value of x) is determined in advance in consideration of the accuracy of reading unit information in the information processing apparatus 3 and the like. As a result, the absolute value of the luminance value and the color difference value of the divided image 61 displayed in the area 60 in which the unit information is to be embedded is determined. Accordingly, the absolute value of the luminance value and the color difference value of the remaining two areas 60 is also determined.

  Furthermore, the information embedding unit 45 changes at least one of the luminance value and the color difference value to be changed for the divided image 61 displayed in the region 60 other than the reference region, which is determined according to the above-described procedure. Unit information is embedded in the divided image 61 of the region 60. Here, when the image is a moving image, the luminance value and the color difference value change every moment when viewed in the pixel unit of the divided image 61. Therefore, the change of the luminance value and the color difference value is the luminance value of the pixel unit. It is preferable not to change the color difference value but to change the luminance value or the like in a certain area of the divided image 61. In particular, when unit information is embedded in the divided image 61 by changing the brightness value, a change area 62 that is changed to approximately the center of the area 60 is set, and the brightness value gradually decreases from the periphery of the change area 62 toward the center. It is preferable to change the luminance value of the divided image 61 so as to change. When the luminance value is changed in this way, there is an effect that it is difficult for the user to visually recognize that the luminance value of the divided image 61 has changed. The method of gradually changing the luminance value from the periphery of the change region 52 toward the center is arbitrary, but as an example, the change of the luminance value of the change region 62 and its surroundings is brought about the region 60 in which unit information is to be embedded. A method of preparing such a two-dimensional digital filter and performing a correlation operation with the divided image 61 displayed in the region 60 is preferable.

  The information embedding unit 45 then divides all unit information constituting the transmission information 33 to be transmitted by the above-described procedure by periodically changing at least one of the luminance value and the color difference value of the divided image 61. Embed in the image 61. The relationship between the code string constituting the transmission information 33 and changes in luminance values and color difference values will be described with reference to FIG. In FIG. 9, it is assumed that the code string 63 constituting the transmission information 33 is as shown in the figure. The right side of FIG. 9 shows a column of time units 64 for changing the luminance value or the color difference value. The time unit 64 is determined from the viewpoint of information reading performance by the information processing apparatus 3, information transmission speed, and the viewpoint that the user cannot see the luminance value and the color difference value as much as possible, and is set to 0.1 to 1 second as an example. Is done. Preferably, the time unit 64 includes a plurality of frames for image display. The information embedding unit 45 changes the code as unit information by changing at least one of the luminance value and the color difference value of the divided image 61 for each time unit 64 based on the code string 63 constituting the transmission information 33. Embed in. A solid line 65 indicates the time during which at least one of the luminance value and the color difference value changes. Here, when the same code continues, a partitioning time 66 during which the luminance value and the color difference value are not changed is provided in order to identify individual codes. At this time, if the above formula is used, m = n. On the other hand, when shifting to a different code, the luminance value and the color difference value continuously change by embedding the next code without providing the partition time 66. In the example of FIG. 9, the next code is embedded without providing the partition time 66 in the scene where the code string “101” shifts from “1” to “0”. It is also possible to embed the next code without providing the partition time 66 in the scene where the code string “0” shifts to “1”. By embedding the next code without providing the partition time 66, the transmission information 33 can be transmitted in a shorter time.

  In step S5, the image data in which the unit information is embedded by the information embedding unit 45 is output from the display unit 14, and the image in which the unit information is embedded is displayed on the display screen 2a. Thereafter, the operation shown in the flowchart of FIG. 5 ends.

  Next, FIG. 6 is a flowchart for explaining the operation of the information processing apparatus 3 used in the information providing system of the present embodiment. The operation shown in the flowchart of FIG. 6 is started when the user operates the input instruction unit 25b or the like of the information processing device 3 in order to obtain information from the image displayed on the display screen 2a of the image display device 2. Is done.

  First, in step S <b> 10, an operation of imaging the display screen 2 a of the image display device 2 by the imaging unit 24 of the information processing device 3 is started. Captured image data of the display screen 2 a of the image display device 2 captured by the imaging unit 24 is input to the control unit 50.

  Next, in step S11, the area determination unit 52 of the control unit 50 of the information processing device 3 is based on the captured image data that is output from the imaging unit 24, and the luminance value and color difference at any location of the captured image data. Whether or not at least one of the values has changed is monitored, and if it is determined that at least one of the luminance value and the color difference value has changed (YES in step S11), the program proceeds to step S12.

  Here, when the image is a moving image, the luminance value and the color difference value change every moment when viewed in the pixel unit of the captured image data. Therefore, the detection of the luminance value and the color difference value change in step S11 is performed on a pixel basis. It is preferable to detect a change in luminance value or the like in the entire captured image data, instead of a change in luminance value or color difference value. For example, a method of calculating an addition value or an average value of luminance values and color difference values of captured image data and detecting a change in the addition value or the average value can be used. In particular, since the change cycle of the luminance value or the like in step S4 described above is relatively gradual (0.1 to 1 second in the above example), it is preferable to detect the change according to this change cycle.

  Next, in step S12, the region determination unit 52 detects a reference region determined by the image display device 2 from the captured image data. That is, the area determination unit 52 divides the captured image data output by the imaging unit 24 by imaging the display screen 2a into four to obtain four divided captured image data. Among the four divided captured image data, the luminance is determined. The divided captured image data in which neither the value nor the color difference value is changed is detected as the reference area. When the image display device 2 embeds the unit information, as described in step S4 above, since at least one of the luminance value and the color difference value of the divided image 61 in the region 60 other than the reference region has changed, the luminance value, The reference region can be detected by detecting the divided captured image data in which none of the color difference values have changed.

  Next, in step S13, the information extraction unit 53 determines the luminance values and color difference values of the three divided captured image data other than the divided captured image data corresponding to the reference region 60 based on the determination result of the region determination unit 52 in step S12. By calculating at least one of them and comparing at least one of the luminance value and the color difference value of the three divided captured image data, the divided captured image data in which the unit information is embedded is specified. The divided captured image data in which the unit information is embedded is determined according to the unit information embedding procedure described in step S4. That is, since it is possible to determine whether or not either the luminance value or the color difference value is changed by the detection operation in step S11, there is a change among the luminance value and the color difference value of the three divided captured image data. Compare the values and specify the divided captured image data that takes the intermediate value of either or both of the luminance value and color difference value that have changed, as the divided captured image data in which the unit information is embedded To do.

  Next, in step S14, the information extraction unit 53 compares the divided captured image data in which unit information is embedded identified in step S13 with the divided captured image data corresponding to the reference area, thereby obtaining unit information. Extract the code. That is, the information extraction unit 53 divides the value corresponding to the reference area and the value of the divided captured image data in which the unit information is embedded with respect to the value of which the change is detected among the luminance value and the color difference value detected in step S11. The value of the captured image data is compared, and a code that is an embedded unit image is determined according to the magnitude relationship.

  In step S15, the area determination unit 52 monitors the captured image data that is output from the imaging unit 24, and determines whether or not the change in the luminance value and the color difference value detected in step S11 has ended. If it is determined that the luminance value and color difference value changes have been completed in the captured image data (YES in step S15), the program proceeds to step S16, where the luminance value and color difference value changes continue and have not yet been completed. (NO in step S15), the program returns to step S12, and continues the unit information extraction operation shown in steps S12 to S14.

  In step S16, the information extraction unit 53 constructs information that is a code string from the codes that are unit information extracted by the unit information extraction operations shown in steps S12 to S14 described above. Next, in step S <b> 17, the communication unit 52 of the information processing device 3 sends the information constructed in step S <b> 16 to the external server 7. In step S18, additional information generated by the external server 7 based on the information sent in step S17 is received from the external server 7, and the operation shown in the flowchart of FIG. 6 is terminated. Additional information received from the external server 7 is, for example, a store URL captured as a subject in the image, store map data, and character image data captured as a subject.

(Effect of one embodiment)
As described above, according to the information providing system 1 of the present embodiment, information can be provided from the image display device 2 to the information processing device 3. Here, in the information providing system 1 of the present embodiment, information is changed by changing at least one of the luminance value or the color difference value of the divided image 61 displayed in the area 60 obtained by dividing the display screen 2a of the image display device 2 into four. Is embedded in the image. Since the change of the luminance value or the color difference value is difficult for the user to visually recognize, the unit information is embedded by changing at least one of the luminance value or the color difference value of the divided image 61 as in the information providing system 1 of the present embodiment. Even if the user views the image, the possibility that the user can visually recognize that the unit information is embedded is low. As a result, information can be transmitted from the image display device 2 to the information processing device 3 without greatly affecting the user's appreciation of the image.

  In addition, in the information providing system 1 of the present embodiment, the region determination unit 44 determines the region 60 suitable for unit information embedding from among the divided images 61 displayed in the region 60 divided into the display screen 2a. Since the unit information is embedded in the divided image 61 by changing at least one of the luminance value or the color difference value of the divided image 61 displayed in the region 60, the information can be displayed in the image without greatly affecting the image viewing by the user. The effect that it can transmit to the information processing apparatus 3 from the display apparatus 2 can be show | played further.

  In particular, since it is more difficult for the user to visually recognize the change when the luminance value is changed than when the color difference value of the divided image 61 is changed, information can be obtained from the image display device 2 without greatly affecting the user's image appreciation. From the viewpoint of transmission to the information processing device 3, it is more preferable to embed unit information by changing the luminance value of the divided image 61.

  Further, in the information providing system 1 according to the present embodiment, the luminance value of the divided image 61 displayed individually in each of the other areas 60 is set as any one of the areas 60 obtained by dividing the display screen 2a into four areas. Alternatively, by changing at least one of the color difference values to a different value, the region 60 in which the divided image 61 in which the unit information is embedded is specified, so that it is displayed in one region 60 out of the four divided regions 60. Compared with the case where at least one of the luminance value or the color difference value of only the divided image 61 to be changed is changed, the region 60 where the divided image 61 in which the unit information is embedded is easily specified. As a result, the information processing apparatus 3 can easily and reliably acquire information transmitted from the image display apparatus 2. In particular, according to the present embodiment, since the transmitted information is acquired by comparing divided captured image data obtained by dividing the captured image data into four, the imaging capability is low, for example, the resolution is low, and the number of pixels is small. Even if the imaging unit 24 is used, the information transmitted from the image display device 2 can be obtained in the information processing device 3 simply and reliably.

(Modification)
The information providing system of the present invention is not limited to the above-described embodiment, and various modifications are possible. As an example, the reference area 60 described above may be fixed to any one of the four areas 60 during one information providing operation, or one unit information is sent from the image display apparatus 2 to the information processing apparatus 3. The region 60 that becomes the reference region 60 may be changed each time. As described above, when the region 60 serving as the reference region 60 is changed for each unit information, the region 60 in which the divided image 61 in which the unit information is embedded is also changed to the region 60 for each unit information. Unit information can be sent continuously without providing the partition time 66 described in the embodiment. On the other hand, when the reference area 60 is fixed to any one area in advance, the information processing apparatus 3 is provided with posture detection means, for example, an acceleration sensor, and the relative relationship between the image display apparatus 2 and the information processing apparatus 3 is set. By detecting the positional relationship, the reference region 60 can be detected immediately. Alternatively, a specific marker image may be superimposed and displayed on the divided image 61 corresponding to the reference region 60 in the divided image 61. In this case, the region determination unit 52 can detect the reference region 60 immediately.

  Further, even when the reference area 60 described above is fixed to any one of the four areas 60 during one information providing operation, the unit information embedding area among the remaining three areas 60 for changing the luminance value and the color difference value. Is changed in units of codes to be embedded, unit information can be continuously transmitted without providing the partition time 66 described in the above-described embodiment.

  In the above-described embodiment, the information transmitted to the information processing apparatus 3 is transmitted from the communication unit 52 to the external server 7. However, the information amount of information transmitted to and provided to the external server 7 is intended to increase. The information detection unit 54 of the information processing device 3 may convert the information transmitted from the image display device 2 into converted information having a bit length longer than the bit length of this information and transmit the converted information. As an example, a hash value may be calculated for the information transmitted from the image display device 2, and this hash value may be transmitted to the external server 7 as conversion information.

  Furthermore, in the above-described embodiment, at least one of the luminance value or the color difference value of the divided image 61 displayed in the area 60 obtained by dividing the display screen 2a into four is changed discontinuously. At least one of the value and the color difference value may be continuously changed. As an example, at least one of the luminance value and the color difference value of the divided image 61 displayed in the area 60 obtained by dividing the display screen 2a into four is periodically and continuously changed. At this time, the frequency of fluctuation of the luminance value and color difference value of the divided image 61 displayed in the reference area 60 and the frequency of fluctuation of the luminance value and color difference value of the divided image 61 displayed in the other three areas are displayed. If the difference is replaced with the difference between the absolute value of the luminance value and the color difference value described in the above-described embodiment, unit information can be provided to the information processing device 3 as in the above-described embodiment. Alternatively, even if the frequency is the same, the phase difference of the fluctuation of the luminance value and the color difference value may be given. With such a configuration, in the information processing apparatus 3, an operation for detecting at which position the captured image data is divided into the divided captured image data, and further, the optical axis of the imaging unit 24 of the information processing apparatus 3 is the image display apparatus 2. If the captured image data is distorted because it does not match the normal line of the display screen 2a, the frequency and phase difference of the luminance value and the color difference value are changed in the entire captured image data without performing the trapezoidal correction operation. If it is detected in the representative area, it is possible to reliably detect the embedded unit information. In addition, even if the absolute value of the amplitude of the fluctuation of the luminance value and the color difference value is reduced, the information processing apparatus 3 can reliably detect the unit information, which greatly affects the user's image viewing. The effect that information can be transmitted from the image display device 2 to the information processing device 3 can be further exhibited.

  Furthermore, in the above-described embodiment, at least one of the luminance value or the color difference value of the divided image 61 displayed in the area 60 obtained by dividing the display screen 2a into four parts is changed. However, the display screen 2a is divided. The number of regions 60 thus obtained is not limited as long as it is a multiple of four. Then, by embedding unit information in units of four areas as a unit, it becomes possible to provide a plurality of information to the information processing apparatus 3 in parallel using one image.

  In the above-described embodiment, the programs for operating the image display device 2, the information processing device 3 and the like are provided by being stored in the ROMs 11 and 21, but the optical disk drive (not shown) and the USB interfaces 15 and 25c. Etc. are used to connect a DVD (Digital Versatile Disc), a USB flash memory device or the like in which the program is stored, read the program from the DVD into the image display device 2 or the like, and operate the image display device 2 or the like. Also good. Alternatively, a program may be stored in a device on the Internet 5 and the image display device 2 or the like may be operated by reading the program into the image display device 2 or the like via the network interface 17 or the wireless communication interface 23 or the like. Good. Furthermore, in the above-described embodiment, the image display device 2 and the like are configured by a plurality of hardware elements. However, the CPUs 10 and 20 and the like may realize some operations of these hardware elements by the operation of the program. Is possible.

DESCRIPTION OF SYMBOLS 1 Information provision system 2 Image display apparatus 2a Display screen 3 Information processing apparatus 4 Router 5 Internet 7 External server 10, 20 CPU
11, 21 ROM
12, 22 RAM
13, 25a Input interface 14 Display unit 15, 25c USB interface 16, 26 HDMI interface 17 Network interface 18 HDD unit 23 Wireless communication interface 24 Imaging unit 25, 42 Input unit 25b Input instruction unit 30 Input device 31 USB device 32 HDMI device 33 Information for transmission 40, 50 Control unit 41 Storage unit 43 Image monitoring unit 44 Region determination unit 45 Information embedding unit 51 Communication unit 52 Region determination unit 53 Information extraction unit 60 Region 61 Divided image 62 Change region 63 Code sequence 64 Time unit 66 Partition time

Claims (14)

An image display device having a display unit for displaying an image on a display screen, and an information processing device having an image pickup unit for capturing the display screen and outputting a captured image, information from the image display device to the information processing device In the information providing system that provides
The image display device divides the display screen into four regions, and displays one of the four divided regions as a reference region in the other regions based on the information to be provided. An information embedding unit that embeds unit information constituting the information in the image by changing at least one of a luminance value or a color difference value of the divided image;
The display unit displays the image in which unit information is embedded by the information embedding unit on the display screen,
The information processing apparatus compares the four divided captured images corresponding to the region among the captured images output from the imaging unit, extracts the unit information according to the comparison result, and obtains the information An information providing system comprising a unit. The information processing apparatus includes a communication unit that performs communication with an external network,
The communication unit transmits the information extracted by the information extraction unit to a server existing in the external network, and the server transmits additional information corresponding to the information to the information processing apparatus. Item 4. The information providing system according to Item 1. The information processing apparatus includes an area determination unit that determines the divided captured image corresponding to the reference area by detecting a change in luminance value or color difference value of the divided captured image.
The information extraction unit extracts the unit information based on a change in luminance value or color difference value of the divided captured image corresponding to the reference region, which is determined by the region determination unit, and obtains the information. The information providing system according to claim 1. The information embedding unit embeds unit information constituting the information in the image by changing luminance values of the divided images displayed in the regions other than the reference region,
The information extraction unit detects a divided captured image having an intermediate luminance value among the other three divided captured images based on the divided captured image corresponding to the reference region determined by the region determining unit. 4. The information providing system according to claim 3, wherein the information is extracted based on a divided captured image having an intermediate luminance value. 2. The information embedding unit embeds unit information constituting the information in the image by changing a luminance value of each of the divided images displayed in the region other than the reference region. The information provision system in any one of -3. The information providing system according to claim 5, wherein the information embedding unit embeds unit information constituting the information in the image by changing a luminance value of a central portion of the divided image. The information providing system according to claim 5, wherein the information embedding unit embeds information including a plurality of unit information in the image by periodically changing a luminance value of the divided image. The information providing system according to claim 2, wherein the communication unit converts the information obtained by the information extraction unit and transmits the converted information to the server. The image display device includes:
An image monitoring unit that monitors at least one of a luminance value or a color difference value of the divided image displayed in the area;
Based on the monitoring result by the image monitoring unit, one of the regions is determined as the reference region, and the unit information embedded by the information embedding unit is selected from any of the three regions other than the reference region. An information providing system according to any one of claims 1 to 8, further comprising: an area determining unit that determines whether to embed in the area. The information providing system according to claim 9, wherein the area determination unit changes the area serving as the reference area every time one piece of unit information is sent from the image display apparatus to the information processing apparatus. The information embedding unit divides the display screen into an area composed of a plurality of units each having four areas as a unit, and for each unit, the information is embedded in the area other than the reference area based on the information to be provided. The information according to any one of claims 1 to 10, wherein unit information constituting the information is embedded in the image by changing at least one of a luminance value and a color difference value of each of the divided images to be displayed. Offer system. An image display device that includes a display unit that displays an image on a display screen and provides information to an information processing device that images the display screen,
The display screen is divided into four regions, and based on the information to be provided, one of the four regions divided into four regions is used as a reference region, and the luminance values of the divided images respectively displayed in the other regions or An image display apparatus comprising: an information embedding unit that embeds unit information constituting the information in the image by changing at least one of color difference values. An information providing method for providing information from the image display device to the information processing device using an image display device that displays an image on a display screen and an information processing device that captures the display screen,
Each of the image display device and the information processing device includes a control unit,
The control unit of the image display device,
The display screen is divided into four regions, and based on the information to be provided, one of the four regions divided into four regions is used as a reference region, and the luminance values of the divided images respectively displayed in the other regions or Embedding unit information constituting the information in the image by changing at least one of the color difference values;
Executing the step of displaying the image in which the unit information is embedded on the display screen;
The control unit of the information processing apparatus
Imaging the display screen to obtain a captured image;
A step of comparing four divided captured images corresponding to the region among the captured images obtained by capturing the display screen and extracting the unit information according to the comparison result to obtain the information. An information providing method characterized by: A program that is executed by a computer that includes a display unit that displays an image on a display screen and provides information to an information processing apparatus that captures the display screen,
When this program is executed by a computer,
Dividing the display screen into four regions;
Based on the information to be provided, by changing at least one of the luminance value or the color difference value of each of the divided images displayed in each of the other regions using one of the four divided regions as a reference region, A program causing the computer to execute a step of embedding unit information constituting information in the image.

Images