JP2022108280A - Image operation system, image operation server, user terminal, image operation method, and program - Google Patents

Abstract

To provide an image operation system that provides a display image of a place desired to be observed without a user performing a screen operation and gives the user a feeling of actually moving the place being captured by a plurality of imaging devices.SOLUTION: The present invention comprises: an image distribution unit for receiving captured images from a plurality of imaging devices and distributing an image based on at least one of the captured images as a distribution image; an operation information acquisition unit for acquiring information on a moving direction of a display target area in the distribution image corresponding to physical information indicating a user's physical condition; a display control unit for updating a position of a display frame in the distribution image according to the moving direction; an image processing unit for extracting a display image from the distribution image according to the display frame; and an image output unit for outputting the display image.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image manipulation system, an image manipulation server, a user terminal, an image manipulation method, and a program.

In recent years, it has become difficult for disaster victims and elderly people to go to stores due to shrinkage of transportation networks and distribution networks due to depopulation, deterioration in driving ability of automobiles due to aging of users, and disaster victims and elderly people.
For this reason, disaster victims and elderly people are unable to see actual products such as produced foodstuffs by themselves at actual stores, select products, and do shopping.
In addition, when movement is restricted due to an emergency due to infectious disease countermeasures, the user cannot go on a trip or appreciate exhibits at a distant museum or art gallery.

When purchasing products, it is possible to use the Internet on the EC site of the store or order the products by mail order using the telephone (see, for example, Patent Documents 1 and 2).
However, the EC sites and telephone orders are not purchased by observing the actual products in the store, and there are cases where the delivered products (especially produce foods) are unsatisfactory. For this reason, there is also a system that delivers images of products in a store to users, transmits images of products on display to users in real time, and allows users to shop while viewing products that are actually being sold at the store. be.

As for travel, there is also a service in which a virtual tour using the Internet is planned, and the presenter explains famous sightseeing spots at the place the user wants to go, so that the user can enjoy the feeling of traveling.
In the above service, an imaging device is used to provide captured images of the scenery of tourist spots, museums, and art galleries. Inquiries from users are made via telephone or the Internet.

JP-A-2000-099612 JP 2014-178909 A

The configuration of Patent Document 1 can change the imaging position of the imaging device in order to capture the image of the product that the user wants to purchase when providing the image of the product in the physical store to the Internet in real time.
However, in Patent Literature 1, one user exclusively uses one imaging device, and the number of users who can shop at the same time is limited by the number of imaging devices in the store.

Also, when a user goes shopping at a store, generally, the product to be purchased is not decided from the beginning. For example, it is rare that daily menus are planned, and it is often the case that fresh food products are purchased after actually observing the condition and price of the product.
Therefore, in the case of shopping at a physical store, the customer visits sales floors for each type in the store, observes each product, and decides whether or not to purchase the product.
On the other hand, when the user uses the Internet to roam around the store and purchase products, if he/she wants to see images of products of a different type than the product currently being viewed, the image pickup device currently in use can be sequentially selected. Therefore, it is necessary to switch to another imaging device that is imaging another product.

At this time, when switching the imaging device, it is necessary to frequently operate the menu of the user terminal, but not all users are accustomed to the operation of the user terminal.
In addition, Patent Document 2 has a configuration in which product selection is performed using gestures according to the state of the user's hand in order to bring the feeling of purchasing products closer to that of a physical store. are improving.

However, as already described, the operation on the user terminal includes switching the imaging device in the store, so if this switching operation is added to the gestures, the types of gestures used by the user increase.
In particular, there is little relevance between the movement of the sales floor and the movement of the hand when purchasing a product, giving the user the impression that control is difficult to understand, and conversely, the user's operability is reduced.
When visiting sightseeing spots on a trip or viewing artworks at museums and art galleries, the user has the same problem as when shopping at the store on the Internet as described above, because the user wants to appreciate places and artworks of interest while wandering around. There is

The present invention has been made in view of such a situation, and is a process that gives the user a feeling as if he or she were actually moving between locations where images are being captured by a plurality of imaging devices without the user having to operate the screen. To provide an image manipulation system, an image manipulation server, a user terminal, an image manipulation method, and a program, which can easily perform

In order to solve the above-described problems, an image manipulation system of the present invention receives captured images captured by each of a plurality of imaging devices, and distributes an image based on at least one of the captured images as a distribution image. A distribution unit moves a display target area, which is an area to be displayed in the distribution image, corresponding to physical information indicating a user's physical condition and indicating at least a predetermined part of the user's body orientation. an operation information acquisition unit that acquires direction information including a direction; a display control unit that issues an instruction to update a position of a display frame for extracting the display target region in the distribution image based on the display region information including the direction information; an image processing unit that extracts an image surrounded by the display frame from the distribution image and uses the extracted image as a display image; and an image output unit that outputs the display image. The position of the display frame in the distribution image is moved at a speed corresponding to the direction indicated by the direction information.

In the image manipulation system of the present invention, the display control unit moves the display frame in the distribution image to the left or right at a speed corresponding to the leftward or rightward direction indicated by the direction information. The display frame in the direction information is moved up or down according to the indicated up or down direction.

The image manipulation system of the present invention is characterized in that the predetermined part of the body is the head.

The image manipulation system of the present invention includes an imaging device adjacency information acquisition unit that acquires imaging device adjacency information indicating an adjacency relationship at each of the arrangement positions of the imaging devices, When some of the devices are selected as target imaging devices and the target imaging device is one imaging device, the captured image captured by the one imaging device is distributed as a distribution image, and the target imaging device is a plurality of target imaging devices. image pickup device, distributes an image obtained by synthesizing the images picked up by the plurality of image pickup devices as a distribution image, and when the image pickup device change request information instructing the partial change is input, the image pickup device The target imaging device is switched in response to the change request information, and the display control unit instructs to change the target imaging device to another imaging device when a predetermined condition in the direction information is satisfied. Characterized by

The image manipulation system of the present invention comprises an imaging device adjacency information acquisition unit that acquires imaging device adjacency information indicating the adjacency relationship at each of the arrangement positions of the imaging devices, and the image distribution unit includes: When any one of them is selected as a target imaging device, a captured image captured by the target imaging device is distributed as a distribution image, and imaging device change request information instructing to change the imaging device is input, the imaging device change request The target imaging device is switched according to the information, and the display control unit instructs to change the target imaging device to another imaging device when a predetermined condition in the direction information is satisfied. do.

In the image manipulation system of the present invention, each of the imaging devices is arranged at a different position in the store where the product is sold, images different imaging ranges in the store, and the product is displayed in the imaging range. It is characterized by being

The image manipulation system of the present invention further comprises an order processing unit that outputs order form information including an image of the product in response to order information indicating the product purchased by the user, wherein the physical information is the It includes hand position information indicating the position of the user's hand, which is a predetermined part of the user's body, and the operation information acquisition unit determines the position of the product on the display image in accordance with the hand position information. The display control unit generates selected position information by converting the position indicated by the position information into a position on the distribution image in correspondence with the position of the display frame, and the display control unit is, when the order condition for ordering the product indicated by the selected position information is satisfied, the order information including at least the selected position information is transmitted to the order processing unit, and the order processing unit processes the image from the distribution image, a thumbnail image generation unit that extracts an area including the position indicated by the selected position information included in the order information and generates a thumbnail image of the area; and order form information for outputting order form information, which is order form information for the product in the thumbnail image, based on the thumbnail image and the delivery address information. and an output unit.

In the image manipulation system of the present invention, the order condition is that the predetermined part of the body is in a state of being stationary at the position indicated by the position information for a preset time. Characterized by

In the image manipulation system of the present invention, the physical information includes handprint information indicating gestures including poses and states of the user's hand, which is a part of the user's body, and the operation information acquisition unit includes the Gesture information indicating the type of the gesture is acquired based on the bill information, and the necessary condition for the order is that the gesture indicated by the gesture information is a type of gesture preset as a type for placing an order. , characterized in that

The image manipulation system of the present invention further includes a body information recognition section and a user interface section, wherein the user interface section includes a display section for displaying the display image output from the image output section; a motion sensor unit arranged on the head, which is a part of the body, and transmitting head state information regarding the posture of the head to the body information recognition unit, wherein the body information recognition unit detects the head state. The physical information including information on the direction of the head is obtained from the posture of the head in the information.

The image manipulation system of the present invention comprises: a display unit that displays the display image output from the image output unit; a sensor unit that acquires the physical state of the user as sensing information; It is characterized by further comprising a physical information recognition unit for extracting the physical information of the user.

The image manipulation system of the present invention comprises a server, and a user terminal connected to the server and owned by the user, wherein the server comprises the image distribution unit and the imaging device adjacency information acquisition unit. The user terminal is characterized by comprising the operation information acquisition section, the display control section, the image processing section, and the image output section.

The image operation server of the present invention acquires direction information from body direction information indicating the direction of a predetermined portion of the user's body, and moves the position of the display frame in the delivered image at a speed corresponding to the direction indicated by the direction information. an image manipulation server connected to a user terminal that moves the image, and that transmits the distribution image to the user terminal; It is characterized by comprising an image distribution unit that distributes at least a base image as the distribution image.

A user terminal of the present invention is connected to a server that receives captured images captured by each of a plurality of imaging devices, and distributes an image based on at least one of the captured images as a distributed image. A receiving user terminal, which indicates the physical condition of the user and indicates the direction in which the display target area in the distribution image is moved, corresponding to the physical information indicating at least the orientation of a predetermined part of the user's body. a display control unit for instructing to update a position of a display frame for selecting a display target area in the distribution image based on the display area information including the direction information; and the distribution image. and an image processing unit that extracts an image surrounded by the display frame from the display area and generates a display image of the display target area, and an image output unit that outputs the display image.

An image manipulation method according to the present invention includes an image distribution step in which an image distribution unit receives each captured image captured by each of a plurality of imaging devices, and distributes an image based on at least one of the captured images as a distribution image; A direction in which the operation information acquisition unit indicates the physical condition of the user and includes the direction in which the display target area in the distribution image is moved, corresponding to the physical information indicating at least the orientation of a predetermined portion of the user's body. an operation information acquisition process of acquiring information; a display control process of instructing a display control unit to update a position of a display frame for selecting a display target area in the distribution image based on the display area information including the direction information; an image processing process in which an image processing unit extracts an image surrounded by the display frame from the distribution image and generates a display image of the display target area; and an image output unit in which an image output unit outputs the display image. characterized by comprising:

The program of the present invention acquires direction information from body direction information indicating a predetermined partial direction of the user's body, and moves the position of the display frame in the distribution image at a speed corresponding to the direction indicated by the direction information. a program for causing a computer to operate as an image manipulation server that is connected to a user terminal to transmit the distribution image to the user terminal, and the computer receives the captured images captured by each of a plurality of imaging devices. and image distribution means for distributing an image based on at least one of the captured images as the distribution image.

The program of the present invention receives each captured image captured by each of a plurality of imaging devices, and receives the delivered image connected to a server that delivers an image based on at least one of the captured images as a delivered image. A program for operating a computer as a user terminal, which indicates the user's physical condition and indicates the display target area in the distribution image corresponding to physical information indicating at least the orientation of a predetermined part of the user's body. operation information acquisition means for acquiring direction information including the direction of movement; display control means for instructing update of a position of a display frame for selecting a display target area in the distribution image based on the display area information including the direction information; The program operates as image processing means for extracting an image surrounded by the display frame from a distributed image and generating a display image of the display target area and image output means for outputting the display image.

As described above, according to the present invention, it is possible to easily perform processing that gives the user a feeling as if he or she is actually moving between places being imaged by a plurality of imaging devices without the user having to operate the screen. An image manipulation system, an image manipulation server, a user terminal, an image manipulation method, and a program can be provided.

1 is a block diagram showing a configuration example of an image manipulation system according to an embodiment of the present invention; FIG. 2 is a diagram showing a configuration example of an image operation server 10 in the image operation system 1 according to this embodiment; FIG. FIG. 10 is a conceptual diagram explaining each of imaging device identification information and direction information in imaging device change request information; FIG. 3 shows a configuration example of an imaging device adjacency table preliminarily written and stored in an imaging device adjacency information storage unit 3. FIG. 2 is a diagram showing a configuration example of a user terminal 20 and a user interface section 30 in the image manipulation system 1 according to this embodiment; FIG. FIG. 10 is a diagram showing the correspondence between the direction of the user's head (line-of-sight direction) and the direction of movement of the display frame in the distribution image; FIG. 4 is a conceptual diagram for explaining rotation angles in vertical and horizontal rotational movement associated with movement of the head (that is, line of sight) of a user 601; 3 is a conceptual diagram showing correspondence between a display image displayed on a display section 31 of a user interface section 30 and a user's hand captured in a moving image. FIG. FIG. 4 is a conceptual diagram for explaining the operation of processing for changing imaging devices between a plurality of sales floors in a store by the image operation server according to the present embodiment; 10 is a flow chart showing the cooperation of the image manipulation server according to the present embodiment for changing the imaging device in the sales floor of the store and for the purchase of the product by the user. FIG. 10 is a diagram showing a configuration example of an image operation server 10 in the image operation system 1 according to Modification 1 of the present embodiment; FIG. 10 is a diagram showing a configuration example of an image operation server 10 in the image operation system 1 according to Modification 2 of the present embodiment;

An embodiment of the present invention will be described below with reference to the drawings. In the description of this embodiment, an example of a configuration in which an image manipulation system according to one embodiment of the present invention manipulates an image picked up by an imaging device installed in a store that sells products will be described.
FIG. 1 is a block diagram showing an example configuration of an image manipulation system according to an embodiment of the present invention. In FIG. 1, the image manipulation system 1 includes, for example, an image manipulation server 10, a user terminal 20, and a user interface section 30. As shown in FIG. The image operation server 10 and the user terminal 20 are connected by an information communication network 500 including the Internet, for example. Further, the information communication network 500 includes, for example, each of the imaging devices D1 to Dn (n is an integer equal to or greater than 2), the user management database 2, the imaging device adjacency information storage unit 3, and the printer 4. It is connected.

The image operation server 10 supplies each captured image supplied from each of the imaging devices D1 to Dn to the user terminal 20 as a distribution image in response to the imaging device change request information from each user terminal 20. do. The imaging device change request information is, for example, an imaging device ( This is information indicating whether to select as a target imaging device).
In addition, the image operation server 10 uses the user identification information of the user who placed the order with the user terminal 20, which is included in the order information from the user terminal 20, and the display image in which the ordered product is captured, to process the order form information. to generate The image operation server 10 prints the generated order form information (image data) by the printer 4 and notifies the store clerk 602 that the product has been ordered.

The user terminal 20 assigns the distribution image supplied from the image operation server 10 to the position information of the distribution image obtained from the physical information of the user (extracted from the sensing information described later), and defines the image area of the distribution image. The display frame to be extracted is moved, and the image area inside the display frame is used as the display image.
The user terminal 20 then outputs the display image to the user interface section 30 of the user 601 .

When a display image is supplied from the user terminal 20, the user interface unit 30 allows the user 601 to observe the display image and select a product to purchase from each product on the sales floor.
The user interface unit 30 also acquires sensing information indicating the state of each part of the body of the user 601 that is displaced by observing the displayed image, and outputs the sensing information to the user terminal 20 .

FIG. 2 is a diagram showing a configuration example of the image operation server 10 in the image operation system 1 according to this embodiment.
In FIG. 2, the image operation server 10 includes a receiving unit 101, an image distribution unit 102, an imaging device adjacency information acquiring unit 103, a transmitting unit 104, and an order processing unit 110, respectively. The order processing unit 110 also includes a thumbnail image generation unit 111 , a shipping destination information acquisition unit 112 and an order sheet information output unit 113 .
The receiving unit 101 inputs data such as imaging device change request information, order information, and image request information from the user terminal 20 .

The image distribution unit 102 receives captured images from each of the imaging devices D1 to Dn at predetermined intervals, and writes and stores them in a captured image storage unit (not shown).
Then, the image distribution unit 102 selects one of the imaging devices D1 to Dn based on the imaging device change request information supplied from the user terminal 20 .
The image distribution unit 102 reads the captured image of the selected imaging device Dn, uses the read captured image as a distribution image, compresses the data by, for example, encoding it, and outputs it to the user terminal 20 .
The imaging device change request information includes imaging device identification information for identifying the imaging device that captured the captured image currently being sent to the user 601, and direction information (the rotation angle of the rotation of the user's head, that is, the usage direction of the person's line of sight (described later).
Here, in this embodiment, the rotation of the head refers to a direction different from the reference line-of-sight direction in which the front is oriented as the reference direction, that is, the front direction of the human body is set as the reference line-of-sight direction. It indicates that the orientation of the head is changed in the direction indicated by the horizontal angle and the vertical direction (rotation angle of direction information to be described later).

FIG. 3 is a conceptual diagram explaining each of imaging device identification information and direction information in imaging device change request information.
FIG. 3A is a diagram showing the adjacent relationship between each imaging device and each of the other imaging devices and the transition relationship between imaging devices based on direction information.
In FIG. 3A, each of "up", "down", "left", and "right" added to the arrows between imaging devices is direction information. It indicates to which imaging device among other adjacent imaging devices the transition is made.

When the user terminal 20 is activated, if the delivery image (initial image as default) first supplied to the user terminal 20 is the captured image captured by the imaging device D1, the imaging in the imaging device change request information The device identification information is information for identifying the imaging device D1. As an example of the transition of the imaging device, when the imaging device D1 is currently selected and the direction information is "up", the imaging device to be selected is changed from the imaging device D1 to the imaging device D2, and the direction information is changed to " In the case of "downward", the imaging device to be selected is changed from the imaging device D1 to the imaging device D4, and in the case of the direction information being "left", the imaging device to be selected is changed from the imaging device D1 to the imaging device D3. is "right", the imaging device to be selected is changed from the imaging device D1 to the imaging device D5.

FIG. 3B illustrates, as an example, an area corresponding to each direction in direction information from the current imaging device.
In FIG. 3B, the direction of the direction information is determined as "up" when the angle of the direction vector 705 is included in the angle range 701, and is determined as "down" when it is included in the angle range 702. The direction of the direction information is determined, and if it is included in the angle range 703, the direction of the direction information is determined as "left", and if it is included in the angle range 704, the direction of the direction information is determined as "right". be done.

FIG. 4 shows a configuration example of an imaging device adjacency table pre-written and stored in the imaging device adjacency information storage unit 3 . This imaging device adjacency table is provided, for example, for each sales floor. For this reason, each imaging device adjacency table is added with sales floor identification information for identifying each corresponding sales floor.
In FIG. 4, the imaging device adjacency table has columns of imaging device identification information and direction information/transition imaging device for each record.
In the record at the top, for example, the identification information of the imaging device that is imaging the central area of the sales floor is described, and each of the adjacent imaging devices D2 to D5 is an imaging device that can be transitioned. is shown.

When the imaging device identification information indicates the imaging device D1, the imaging device currently selected is the imaging device D1, and when the direction information is "up", the selected imaging device transitions to the imaging device D2. , the imaging device selected as the imaging device D3 transitions when the direction information is "left", the imaging device selected as the imaging device D4 transitions when the direction information is "down", and the imaging device selected as the imaging device D4 transitions when the direction information is "right , the imaging device selected as the imaging device D5 transitions.
Even if each of the other imaging devices D2 to D5 is the currently selected imaging device, as described above, the direction of the direction information shown in the column of direction information/transition imaging device will cause the transition to occur. An imaging device to be used is determined.

Returning to FIG. 2, the imaging device adjacency information acquisition unit 103 reads the data of the imaging device adjacency table from the imaging device adjacency information storage unit 3, and writes and stores the data in the internal storage unit. Then, the imaging device adjacency information acquisition unit 103 causes the image distribution unit 102 to refer to the imaging device adjacency table.

When a distribution image is input from the image distribution unit 102 , the transmission unit 104 outputs the distribution image to the user terminal 20 .

When the order information of the user terminal 20 is supplied via the reception unit 101, the thumbnail image generation unit 111 reads the distribution image that the image distribution unit 102 has transmitted to the user terminal 20 that has output the order information. .
Then, the thumbnail image generation unit 111 reduces the resolution by, for example, thinning out the pixels of the read distribution image to generate a thumbnail image.
Thumbnail image generation unit 111 outputs the generated thumbnail image to order form information output unit 113 .

The destination information acquisition unit 112 refers to the user management table in which the information of the registered user in the user management database 2 is described, and obtains the name, address, and address corresponding to the user identification information added to the distribution image. Read phone number etc. as shipping address information.
Then, the shipping destination information acquiring unit 112 outputs the read shipping destination information to the order form information outputting unit 113 .

The order form information output unit 113 generates, for example, an image showing the ordered product and the relevant Generate purchase order information in a file format that describes the delivery address of the product. Then, the order form information output unit 113 causes the printer 4 to print the generated order form information to create an order form. Instead of printing information to be output to a printer, screen data to be displayed on the display screen of a mobile terminal (tablet computer, smartphone, etc.) carried by the store clerk is generated, sent to each mobile terminal, and sent to the store clerk. It may be configured to notify the order information.

In addition, in the image of the order form described above, rules for identifying the ordered product are set in advance, for example, the ordered product is positioned at the center of the image, or the ordered product is marked with a marker. ing.
As a result, the store clerk 602 goes to the sales floor where the products ordered in the order form are sold, collects the products in the image of the order form at the sales floor, and delivers the collected products to the name and address specified as the delivery destination. Perform shipping processing for delivery via a predetermined delivery company.

FIG. 5 is a diagram showing a configuration example of the user terminal 20 and the user interface section 30 in the image manipulation system 1 according to this embodiment.
In FIG. 5, the user terminal 20 includes a receiving unit 21, an image processing unit 22, an image output unit 23, a physical information recognition unit 24, an operation information acquisition unit 25, a display control unit 26, and a transmission unit 27. . The user interface section 30 also includes a display section 31 and a sensor section 32 . The sensor section 32 includes an imaging section 321 and a motion sensor section 322, respectively. In the present embodiment, for example, an AR (Augmented Reality) glass, which is an example of an HMD (Head Mounted Display), is used as the user interface unit 30 .

When the delivery image of the image pickup device requested by the user 601 is supplied from the image pickup device D1 to the image pickup device Dn, the reception unit 21 outputs the delivery image to the image processing unit 22 .

The image processing unit 22 writes the distributed image supplied from the receiving unit 21 to the internal storage unit for storage.
The image processing unit 22 also extracts a display frame indicating which image area in the distribution image is to be displayed, included in display area information (described later) supplied from the display control unit 26 .
Then, the image processing unit 22 cuts (processes) the image of the display area as the display image, with the area surrounded by the display frame in the distribution image as the display area.
The image processing unit 22 outputs the display image generated by processing the distribution image to the image output unit 23 .

When the display image is supplied from the image processing unit 22 , the image output unit 23 outputs the display image to the user interface unit 30 .

FIG. 6 is a diagram showing the correspondence between the orientation of the user's head (the orientation of the head and indicating the line-of-sight direction of the user) and the orientation of movement of the display frame in the distribution image.
In FIG. 6, a user 601 wears VR glasses, which are the user interface unit 30, on his or her head 601H. Here, the direction information indicates the line-of-sight direction of the user based on the rotation angle of the head 601H.
The operation information acquisition unit 25 corresponds to the orientation of the head indicated by a vertical angle (UP and DOWN direction angles) and a horizontal angle (LEFT and LEFT direction angles). Then, head direction information (an example of direction information indicating the direction of a predetermined part of the user's body) is extracted.
As a result, the display control unit 26 obtains the destination position of the display frame W within the distribution image (updates the position of the display frame W within the distribution image) in accordance with the direction information, and uses the image as the display area information. Output to the processing unit 22 .
Then, the image processing section 22 moves the display frame W within the distribution image according to the display area information supplied from the display control section 26, and extracts the image area within the display frame W as the display image.

FIG. 7 is a conceptual diagram for explaining rotation angles in vertical and horizontal rotational movement associated with movement of the user's 601 head (that is, line of sight).
FIG. 7A shows the angle range of the rotation angle in the horizontal rotation of the head of the user 601 (that is, the rotation in the left-right direction with respect to the reference line-of-sight direction), and the angle range of the head set in the angle range. It shows the correspondence relationship with the moving speed of the display frame W within the distribution image corresponding to the rotation in the horizontal direction. In FIG. 7(a), the direction of rotation of the hands of the clock (leftward) is a + (positive) angle θ, and the direction of rotation opposite to the direction of rotation of the hands of the clock (rightward) is a − (negative) angle −θ. described as.

When the user 601 faces the front, the head rotation angle θ (−θ1≦θ≦θ1), the display frame W in FIG. 6 does not move from the center. That is, regardless of whether the user 601 rotates the neck left or right within the angular range of the section A1, the display image is generated as the image of the center area of the distribution image.

On the other hand, between the angle range from angle θ1 to angle θ2 (section A2L, θ1<θ≦θ2) and the angle range from angle −θ1 to angle −θ2 (section A2R, −θ2≦θ<−θ1) When the angle θ is included, the display frame W moves in the distribution image at the speed V2 corresponding to the rotation direction of the head. That is, the display frame W is moved according to direction information corresponding to the movement of the line of sight (movement of the head) of the user 601 . As a result, the movement of the display frame W within the distribution image is linked to the movement of the line-of-sight direction so that the movement of the visual field of the user 601 is simulated, and the display image extracted from the distribution image is moved in the line-of-sight direction. change accordingly.

Between the angle range from angle θ2 to angle θ3 (section A3L, θ2<θ≦θ3) and the angle range from angle −θ2 to angle −θ3 (section A3R, −θ3≦θ<−θ2) When the angle θ is included, the display frame W moves according to the rotation angle of the head moving within the distribution image at a speed V3 (>V2). In other words, the display frame W is moved so that the user's 601 line of sight moves and the display image is shifted in the line-of-sight direction so as to simulate the movement of the user's 601 field of view, similar to the sections A2L and A2R. Change according to movement.

In the present embodiment, θ1=15°, θ2=30°, θ3=45°, etc. are set from the empirical viewing angle range when a person selects a product.
However, each of the angles θ1, θ2, and θ3 may be adjusted as appropriate depending on the area of the sales floor, the imaging range of the delivery image, the size of the product to be imaged, and the like.
Also, the reason why the speed V3 is set faster than the speed V2 is that the farther the angle of vision is from the front position (reference line-of-sight direction), the farther the line-of-sight direction becomes, and the faster the field of view becomes compared to the front. This is because the user 601 can be visually recognized as if moving.

FIG. 7B shows the angle range of the rotation angle of the head of the user 601 in the vertical direction (that is, the rotation in the vertical direction with respect to the reference line-of-sight direction), and the angle of the head set in the angle range. It shows the correspondence relationship with the moving speed of the display frame W within the distribution image corresponding to the vertical rotation. In FIG. 7B, the direction of rotation of the hands of the clock (upward) is a + (positive) angle α, and the direction of rotation opposite to the direction of rotation of the hands of the clock (downward) is a - (negative) angle -α. described as.

When the user 601 faces the front, the direction of the line of sight is designated as the reference line of sight direction BO, and the head rotation angle α is within the angle range of α1 upward and −α1 to the right (section B1) from the reference line of sight direction BO. (-α1≦α≦α1), the display frame W in FIG. 6 does not move from the center. That is, regardless of whether the user 601 rotates his or her neck up or down within the angular range in section B1, the display image is generated as the image of the center area of the distribution image.

On the other hand, between the angle range from angle α1 to angle α2 (section B2L, α1<α≦α2) and the angle range from angle −α1 to angle −α2 (section B2R, −α2≦α<−α1) When the angle α is included, the display frame W moves to a position corresponding to the field of view at the angle α2 according to the rotation angle of the head at which the display frame W moves within the distribution image at a predetermined speed V0. That is, the display frame W is moved and the display image is changed in accordance with the movement of the line of sight of the user 601 so as to simulate the movement of the field of view of the user 601 as the line of sight of the user 601 moves.

Between the angle range from angle α2 to angle α3 (section B3L, α2<α≦α3) and the angle range from angle −α2 to angle −α3 (section B3R, −α3≦α<−α2) When the angle α is included, the display frame W moves to a position corresponding to the field of view at the angle α3 according to the rotation angle of the head at which the display frame W moves within the distribution image at a predetermined speed V0. In other words, the display frame W is moved so that the user's 601 line of sight moves and the display image is shifted in the direction of the line of sight so as to simulate the movement of the user's 601 field of view in the same manner as in the sections B2L and B2R. Change according to movement.

In this embodiment, α1=5°, α2=10°, α3=15°, etc. are set from the empirical viewing angle range when a person selects a product.
The algorithm for moving the display frame W in the vertical direction is different from the algorithm for moving the display frame W in the horizontal direction because, in the case of perishables, etc., the display width of the product in the vertical direction is larger than the display width in the horizontal direction. This is due to the narrow display width of That is, if the display width of the product is narrow, if the display image is moved frequently, it becomes difficult for the user to see the product. The configuration is such that the display frame W is stopped.
However, in the case of stores such as bookstores, outlet malls, etc., where the number of shelves is large and products are displayed at high positions, the same algorithm as for the horizontal direction is used to determine the head position of the user 601. A configuration may be adopted in which movement control of the display frame W in the distribution image is performed in accordance with rotation in the vertical direction.

Returning to FIG. 5 , when sensing information is supplied from the user interface unit 30 , the physical information recognizing unit 24 responds to the sensing information from the sensor unit 32 to the physical information of the user 601 (rotational movement of the head). The angle and sensor information indicating the state of the body such as the hand of the user 601 (described later) are extracted.
Then, the physical information recognition section 24 outputs the extracted physical information to the operation information acquisition section 25 .

The operation information acquisition unit 25 moves within the distribution image corresponding to the user's line of sight based on the physical information (rotational angle of rotation of the head of the user 601 in the horizontal direction) supplied from the physical information recognition unit 24. Direction information indicating the moving direction of the display frame W is obtained.
Then, the operation information acquiring unit 25 obtains, from the physical information supplied from the physical information recognizing unit 24, direction information based on the rotation angle of the vertical movement of the head of the user 601, and the center position of the display frame W on the display image. Gesture information (an example of physical information) corresponding to the control of the user 601 is extracted from the position information and the image information indicating the hand movement, pose, and state of the user 601 .
Then, the operation information acquisition section 25 outputs the direction information and the gesture information to the display control section 26 .

Based on the direction information, the display control unit 26 determines whether or not the horizontal rotation of the head of the user 601 has exceeded the angles θ3 and -θ3, and whether or not the angles α3 and -α have been exceeded in the vertical rotation of the head. It is determined whether or not it has exceeded.
Then, the display control unit 26 determines when the horizontal rotation of the head exceeds the angle θ3 and the angle −θ3, and when the vertical rotation of the head exceeds the angle α3 and the angle −α3 (in the direction information, (a predetermined condition is satisfied), direction information indicating the moving direction of the user's line of sight is generated, and imaging device change request information is generated for requesting a change from the current imaging device to another imaging device.

Here, as a gesture to change the imaging device, when the horizontal rotation of the head exceeds the angle θ3 and the angle −θ3, and when the vertical rotation of the head exceeds the angle α3 and the angle −α It was stopped for a predetermined time at an angle θ3, angle −θ3 in the horizontal rotation of the head, or at an angle α3, angle −α3 in the vertical rotation of the head ( If the specified condition in the direction information is satisfied), which direction of the current imaging device is selected to change from "left", "right", "up", or "down" It may be configured to generate imaging device change request information.

For example, when the imaging device D1 is currently selected, the display control unit 26 determines that the rotation angle of the head in the horizontal direction in the direction information exceeds the angle θ3, and the standby time set in advance at that rotation angle is If it has passed, the imaging device adjacency table is referenced, and imaging device change request information indicating a change to imaging device D3 is generated. The display control unit 26 then outputs the generated imaging device change request information to the image operation server 10 .

The display unit 31 includes a liquid crystal panel or an organic EL (electroluminescent) panel, and when a display image is supplied from the user terminal 20, displays the display image on the display screen.

The imaging unit 321 is, for example, an imaging device such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) image sensor.
Also, the imaging unit 321 may be a general RGB (Red, Green, Blue) color image sensor, or may be a depth image sensor such as a ToF (Time of Flight) sensor. The imaging unit 321 also captures a moving image of the hand of the user 601 and outputs the moving image to the user terminal 20 as one piece of sensing information. Here, for example, the user 601 moves his or her hand to select a product within the range of the display image displayed on the display unit 31, or stops the hand at the position of the product to be selected.
Also, the coordinate position in the captured image corresponds to the coordinate position of the display image observed by the user.

The motion sensor unit 322 is a sensor that measures the motion of a predetermined part of the user's body. In this embodiment, the motion sensor unit 322 is a sensor mounted on the AR glass that is the user interface unit 30, such as an acceleration sensor, an angular velocity sensor, and a geomagnetic sensor.
In addition, when the user interface unit 30 is AR glasses and the motion sensor unit 322 is a sensor mounted on the AR glasses, the motion sensor unit 322 detects motion on the head of the user on which the AR glasses are mounted. Movement information (head state information) such as acceleration (including gravitational acceleration) and angular velocity is acquired.
In addition, the motion sensor unit 322 is attached to the user's 601 neck muscle such as the platy muscle in place of the sensor mounted on the AR glasses as means for measuring the orientation of the user's 601 head. Alternatively, an extension sensor (capacitance sensor, etc.) may be used. In this case, the motion sensor unit 322 acquires stretching information for estimating the degree of stretching of the platy muscle of the user 601, that is, the orientation of the head linked to the movement of the neck.
Then, the motion sensor unit 322 outputs the acquired movement information or expansion/contraction information corresponding to the rotation of the user's head to the user terminal 20 as one piece of sensing information.

FIG. 8 is a conceptual diagram showing the correspondence between the display image displayed on the display section 31 of the user interface section 30 and the user's hand captured in the moving image.
A display image 801 is an image generated by cutting out an image area within the display frame W in the distribution image supplied from the image output unit 23 .
The hand outline display image 802 is an image showing the outline of the hand (handprint) of the user 601 who observes the display image 801 and touches the product in the display image 801 to select the product. This hand contour display image 802 is superimposed on the display image by the image processing unit 22 and displayed on the display screen of the display unit 31 .

The enlarged display area 803 is a predetermined range centered on a predetermined position of the hand contour display image 802 (specified fingertip, the center of the hand contour display image 802, the center of gravity, etc.), which is enlarged by a predetermined magnification. This is the display area where the image is displayed.
The basket icon 804 is an image superimposed on the display image 801, and is arranged at a predetermined coordinate position of the display image when the image processing unit 22 generates the display image.

The physical information recognition unit 24 extracts the image area of the user's hand from each frame image in the moving image captured by the imaging unit 321 .
Then, the physical information recognition unit 24 performs outline extraction in the extracted image area of the hand, and generates a hand outline display image 802 that is the outline of the hand of the user 601 .
The physical information recognition unit 24 outputs to the operation information acquisition unit 25 a hand outline display image 802, which is one piece of physical information, generated from the moving image in the sensing information.
Further, when the body information recognition unit 24 acquires movement information corresponding to the rotation of the head of the user 601, which is one piece of information in the sensing information supplied from the motion sensor unit 322, the body information recognition unit 24 detects the movement of the head from the movement information. Obtain the rotation angle and rotation direction of the rotation of .
Further, when the body information recognition unit 24 acquires stretch information (an example of the body information) indicating the degree of stretch of the neck muscles of the user 601, which is one piece of information in sensing information from the motion sensor unit 322, The rotation angle and rotation direction of the rotation of the head are obtained from the expansion/contraction information. The physical information recognition unit 24 outputs the rotation angle and the rotation direction of the rotation of the head to the operation information acquisition unit 25 as one piece of physical information.

The operation information acquisition unit 25 extracts gesture information by comparing the motion of the hand outline display image 802 with a preset motion.
For example, when selecting a product imaged in the display image, the hand outline display image 802 in the display image 801 is moved to the position of the product to be ordered, and the product is stopped at the position of the product for a predetermined time. The operation information acquisition unit 25 extracts the image of the product to be ordered by encircling it with a circle (specifying the position of the product based on the hand position information, that is, the position information).

Then, the operation information acquisition unit 25 detects the coordinate position of the selected product in the display image as position information.
The operation information acquisition unit 25 performs processing such as, for example, selecting an image in the detected position information as a selected product, adding a marking to the coordinate position indicated by the position information, and generating a display image centered on the position information. I do.

For example, the operation information acquisition unit 25 detects from the movement of the hand outline display image 802 that the user 601 has moved his or her hand to the position of the basket icon 804 after selecting a product. Then, the operation information acquiring unit 25 determines from the hand movement to the basket icon 804 after selecting the product that it is a gesture indicating that the selected product is to be purchased (order condition for ordering the product), Gesture information indicating that the selected product is to be purchased is output to the display control unit 26 . Here, which gesture is the gesture information indicating the movement of the display frame, the transition of the imaging device, the selection of the product, and the order of the product is set in advance.

In addition, instead of detecting the head movement direction by the motion sensor unit 322, the operation information acquisition unit 25 detects that the user 601 moves the hand in an arbitrary direction at a predetermined speed. A configuration may be adopted in which motion is detected and gesture information is used to move the display frame W in any direction including up, down, left, and right.
In addition, the operation information acquisition unit 25 selects by putting up the desired number of fingers instead of moving the hand (a predetermined part of the body of the user) to the position of the basket icon 804 . The gesture information may be gesture information indicating that the purchased product is to be purchased.
In addition, instead of moving the hand to the position of the basket icon 804, the operation information acquisition unit 25 may stop (pause) the movement of the hand or the movement of the head for a predetermined time at the position of the item to be ordered. , gesture information indicating that the product purchase process is to be performed.

The display control unit 26 generates order information from the position information of the selected product and the gesture information indicating the purchase process of the selected product, and outputs the order information to the image operation server 10 via the transmission unit 27 . The display control unit 26 converts the coordinate values of the selected product in the display image indicating the position information into the coordinate values (selected position information) in the distribution image, and converts the selected position information into the specified information for specifying the ordered product. is included in the order information and output to the image operation server 10 .
The order form information output unit 113 generates order form information corresponding to the order information.

FIG. 9 is a conceptual diagram for explaining the operation of processing for changing imaging devices between a plurality of sales floors in a store by the image operation server according to this embodiment.
In FIG. 9, each of the sales floors in a store is shown in which each of a plurality of, for example four, different product types are displayed. As for the stores, the sales floor zone Z1 is for vegetables and fruits, the sales floor zone Z2 is for fish, the sales floor zone Z3 is for meat, and the sales floor zone Z4 is for bread and sweets.

Each of the sales floor zone Z1 to the sales floor zone Z4 is provided with five imaging devices for imaging the products displayed in the sales floor. The adjacency relationship between the imaging devices described with reference to FIG. 3 is set for each sales floor zone. For each sales floor zone, an imaging device adjacency information table shown in FIG. 4 showing transition information between imaging devices corresponding to the adjacency relationship shown in FIG. remembered.
The processing of the transition of the imaging device in each sales floor is, as described with reference to FIG. performed by

In the process of transitioning imaging devices between different sales floors, for example, a case where the user currently sets imaging device DZ2_1 in sales floor zone Z2 will be described as an example.
For example, in the imaging device adjacency information table, if the imaging device D2_1 is the imaging device DZ2_1, the imaging device DZ2_1 is not set to move according to the "right" direction information.

That is, in the distribution image of the imaging device DZ1_1, when the head is rotated horizontally in the “right” direction and the display frame W cannot be moved at the right end of the distribution image, and a predetermined time elapses, it is selected. The imaging device DZ1_1 in the current state transitions to the imaging device in the adjacent sales floor zone Z1 on the right. At this time, the imaging device of the transition destination is a preset default imaging device and can be set arbitrarily. For example, like the imaging device D1 in FIG. It is an imaging device that is imaging.
The relationship information indicating the transition destination imaging device is written in advance in the imaging device adjacency relation information table. In this case, the display control unit 26 sends imaging device change request information including imaging device identification information of the currently selected imaging device, direction information, and information indicating that a predetermined time has passed since the imaging device was stopped. , to the image operation server 10 .

As a result, when the image distribution unit 102 receives imaging device change request information indicating transition of imaging devices between different sales floors from the user terminal 20, the image distribution unit 102 refers to the imaging device adjacency information table, It is possible to change the selected imaging device from the selected imaging device to the imaging device of another sales floor.
Further, in the above-described configuration, transition to another sales floor imaging device is performed according to the direction of rotation of the user's head. A configuration may also be adopted in which a person selects a sales floor using a sales floor selection button. At this time, the imaging device change request information from the display control unit 26 includes information that can identify the sales floor, such as the sales floor identification information and the coordinate values where the sales floor selection button is displayed. Even in the case of this configuration, the imaging device selected in the new sales floor is the preset default imaging device.

FIG. 10 is a flow chart showing how the image operation server according to the present embodiment cooperates with the operation of changing the imaging device in the sales floor of the store and the purchase of the product by the user.
In the operation on the image operation server 10 side, when the image operation server 10 is activated and starts an image operation service for distributed images (step S101), the order processing unit 110 starts a product purchase management service. (step S102).
Similarly, the image operation server 10 outputs a control signal for starting the operation of the imaging devices D1 to Dn (step S103).
The image operation server 10 starts controlling the image distribution unit 102 that distributes the captured images captured by the imaging devices D1 to Dn (step S104).

Then, each of the imaging devices D1 to Dn starts to take an image of the scenery (for example, the displayed product) in its own imaging direction (step S105). The image distribution unit 102 acquires captured images captured by each of the imaging devices D1 to Dn, and writes and stores them in the storage unit at a predetermined cycle (step S106).

When image request information requesting an initial image of a sales floor is supplied from the user terminal 20, the image distribution unit 102 reads out the image captured by the default imaging device Dn of the requested sales floor from the storage unit.
Then, the image distribution unit 102 compresses the read captured image and supplies it to the user terminal 20 as a distribution image (step S107).

Further, when the user terminal 20 supplies imaging device change request information (information for transitioning to another imaging device in the same sales floor), the image distribution unit 102 changes the imaging device in the imaging device adjacency information storage unit 3. Refer to the adjacency table.
Here, the image distribution unit 102 extracts the transition destination imaging device from the imaging device adjacency table based on the imaging device information and direction information in the imaging device change request information.
As a result, the image delivery unit 102 reads the captured image captured by the imaging device of the transition destination from the storage unit, and outputs it to the user terminal 20 that supplied the imaging device change request information (step S107).

Further, when the image distribution unit 102 is supplied with imaging device change request information (information for transitioning to an imaging device in another sales floor) from the user terminal 20, the image distribution unit 102 stores the imaging device adjacency information storage unit. Refer to each imaging device adjacency table in each sales floor in 3.
Then, the image distribution unit 102 selects the imaging device adjacency table of the destination sales floor based on the direction information in the imaging device change request information (or the sales floor identification information indicating the sales floor when selected with the sales floor selection button), and defaults to the imaging device adjacency table. Extract the imaging device set in . As a result, the image delivery unit 102 reads out the captured image captured by the extracted imaging device from the storage unit and outputs it to the user terminal 20 that supplied the imaging device change request information (step S107).

Further, when image request information requesting the latest captured image is supplied from the user terminal 20, the image distribution unit 102 acquires the captured image captured by the currently selected imaging device Dn, is written and stored in the storage unit as a picked-up image.
Then, the image delivery unit 102 compresses the read latest captured image and supplies it to the user terminal 20 as a delivery image (step S107).

When the order information is supplied from the user terminal 20, the order processing unit 110 corresponds to the imaging device identification information and the selected position information included in the order information, and generates a captured image captured by the imaging device corresponding to the imaging device identification information. is read, and a thumbnail image of the ordered product is generated corresponding to the selected position information.
Also, when the user terminal 20 is activated, the order processing unit 110 reads the user identification information of the user transmitted from the user terminal 20, and reads the address corresponding to the user identification information as the destination.

Then, the order processing unit 110 outputs to the printer 4 the thumbnail image of the ordered product and the shipping address of the product as order form information.
The printer 4 prints and creates the order form information supplied from the order processing unit 110 (step S108), and issues an order form on which the thumbnail image of the ordered product and the shipping address are printed (step S109). .
Further, the order processing unit 110 may be configured to transmit the order form information to a mobile terminal carried by a store clerk. In this configuration, order form information is displayed on the display screen of the mobile terminal carried by the store clerk.

The clerk sees the printed order form or the order form information on the display screen of the mobile terminal, goes to the sales floor where the thumbnail image of the product is sold, and takes out the thumbnail image of the product from the display shelf. picking, step S110).
Then, the store clerk ships (ships) the product to the destination (user's address) in the order information (step S111).

Next, in operations on the user terminal 20 side, the user 601 wears AR glasses as the user interface unit 30 on the head (step S201), and activates an image manipulation application on the user terminal 20 (step S202). ).

For example, the user 601 moves his/her hand to the position of the selection button of the desired sales floor from each of the sales floor selection buttons displayed on the display unit 31 of the user interface unit 30, and touches to select the sales floor. (step S203). As a result, the operation information acquiring unit 25 includes, from the moving image captured by the imaging unit 321, the coordinate value indicating which coordinate value of the sales floor selection button has been touched, and the information indicating that the coordinate value has been touched. Extract gesture information.

Then, the operation information acquisition unit 25 outputs to the display control unit 26 gesture information extracted from the user's physical information indicating that the hand has been moved to the position of the sales floor selection button.
The display control unit 26 transmits imaging device change request information including the coordinate values of the sales floor selection button in the gesture information of the distribution image display button and the information indicating transition to the imaging device of this sales floor to the image operation server 10 . output.

The image processing unit 22 displays an image corresponding to the rotation information of the head of the user 601 input from the user interface unit 30 when the captured image of the imaging device in the sales floor requested by the user is supplied as the distribution image. Generate an image.
Then, the image output section 23 causes the display section 31 of the user interface section 30 to display the display image.

The user 601 observes the products in the display image displayed on the display unit 31 and moves the head to change the displayed product to be observed, that is, to bring into view the display shelf where other products are displayed. The part is moved (step S204).
As a result, the physical information recognition unit 24 acquires the measurement value of the motion sensor unit 322 as sensing information from the user interface unit 30, and based on this sensing information, determines that the user 601 has turned the head and the measurement value. body information indicating head movement information including;
Then, physical information recognition section 24 outputs the generated physical information to operation information acquisition section 25 .

The operation information acquisition unit 25 extracts direction information for moving the display frame W or gesture information for transitioning the imaging device from the human body information, and outputs the extracted information to the display control unit 26 .
Based on the direction information, the display control unit 26 outputs to the image processing unit 22 display area information indicating the position of the display frame W for extracting the display image in the distribution image.
Accordingly, the image processing unit 22 moves the head of the user 601, that is, moves the display frame W corresponding to the line-of-sight direction of the user 601, and performs processing for generating a display image from the distributed image.
Then, the image output section 23 displays the display image generated by the image processing section 22 on the display screen of the display section 31 .

In addition, the display control unit 26 moves the position of the display image extracted from the same distribution image, transitions between the imaging devices of the same sales floor, and the like, in response to direction information, gesture information for transition between imaging devices, and the like. Furthermore, imaging device change request information indicating transition to an imaging device of another sales floor is generated.
The display control unit 26 outputs the generated imaging device change request information to the image operation server 10 via the transmission unit 27 .
Then, the user 601 observes the products in the display image that moves in accordance with the direction of his or her viewpoint, and selects the product to be ordered (step S205).

The user 601 selects a product to be ordered on the display image from among the products being observed. For example, a product to be ordered is selected by holding one's hand at the position of the product for a predetermined period of time.
Then, the user 601 puts the selected product into the basket icon displayed in the display image (step S206).
At this time, the physical information recognition unit 24 acquires the moving image captured by the imaging unit 321 as sensing information.

The physical information recognizing unit 24 extracts the motion of the hand from the moving image and outputs it to the operation information acquiring unit 25 as the physical information of the user 601 .
The operation information acquisition unit 25 determines that the user 601 has selected a product from the hand motion of the user 601, and determines that the position information indicating the position of the image of the selected product and the gesture indicating that the product has been ordered are displayed. information to the display control unit 26 .
The display control unit 26 then outputs order information for the selected product to the image operation server 10 .
The user 601 terminates the application on the user terminal 20 if there are no other products to order (step S207).

As described above, according to the present embodiment, by changing the line-of-sight direction of the user in the same way as when going to a store, the sensor unit 32 in the user interface unit 30 detects the movement of the line-of-sight direction of the user. Obtained from the information on the rotational movement of the head, and generates a display image accompanying the movement of the line of sight within the distribution image according to the movement of the line of sight (movement of the line of sight in the same place, for example, movement of the line of sight between products in the same sales floor (image display processing corresponding to ), and also change the imaging device. The display image moves in the line-of-sight direction that the user wants to observe, and in conjunction with the movement of the line-of-sight, it is possible to switch the image capturing device that captures the distribution image to the image capturing device of the desired location, so it is as if there are multiple image capturing devices. It is possible to easily perform processing that gives the user a feeling that he or she is actually moving in the place where the camera is imaging.
That is, according to the present embodiment, it is possible to seamlessly (smoothly) move the observation location to a different location or observe a specific location within the observation location based on the user's physical information.

In the above-described embodiment, an imaging device that captures images of products on each sales floor in a store has been described as an example. A similar configuration can be used for the device.
Further, in the present embodiment, a configuration using an HMD such as AR glasses as the user interface unit 30 has been described. may be used as the display unit 31 .
Further, in this embodiment, the sensor unit 32 does not include the motion sensor unit 322, the imaging unit 321 captures moving images of the user's hand and head, and the physical information recognition unit 24 receives movement information or stretch information. Alternatively, the rotation angle and the rotation direction of the head may be obtained from each frame image of the moving image by a well-known method. As an example of this method, facial parts such as the eyes, mouth, nose, ears, and eyebrows are extracted as feature regions from an image, and the direction of the head is estimated based on the displacement of the feature regions from their arrangement when viewed from the front. There are known methods such as Further, in the present embodiment, the image processing unit 22, the physical information recognition unit 24, the operation information acquisition unit 25, and the display control unit 26 are provided in the user terminal 20. The configuration may be provided in the image operation server 10 .

<Modification 1>
In the above-described embodiment, each captured image captured by the imaging device D1 to the imaging device Dn is used as a distribution image. Alternatively, an image obtained by synthesizing the captured images captured by each of the imaging devices of the set may be used as the distribution image. In Modified Example 1 of the present embodiment, an example will be described in which the image operation server 10 in the image operation system 1 according to the above-described embodiment synthesizes captured images captured by each of the imaging devices of the set and generates a distribution image. In this modified example, only parts different from the above-described embodiment will be described, and description of other parts will be omitted.

FIG. 11 is a diagram showing a configuration example of the image operation server 10 in the image operation system 1 according to Modification 1 of the present embodiment. The image operation server 10 of this modified example includes an image distribution unit 102a instead of the image distribution unit 102, but otherwise is the same as the image operation server 10 of the above-described embodiment. The image distribution unit 102a synthesizes captured images captured by a predetermined set of some of the imaging devices D1 to Dn (for example, the imaging device D1 and the imaging device D2) into one image. It differs from the image distribution unit 102 in that an image is generated and used as a distribution image, but the rest is the same as the image distribution unit 102 .

Note that the image distribution unit 102a is configured to use a plurality of predetermined image pickup devices among the image pickup devices D1 to Dn, such as a set of the image pickup devices D1 and D2 and a set of the image pickup devices D3 to D5. A distribution image may be generated for each set of . Also, the set of these imaging devices may be determined based on the adjacency of the imaging devices D1 to Dn stored by the imaging device adjacency information acquisition unit 103 . Further, the image distribution unit 102a may use the image to be stored in the captured image storage unit as the distribution image.

Assuming that the object to be photographed is a plane, the image distribution unit 102a may synthesize the picked-up images by connecting images near the center of the picked-up images, that is, images facing the object to be photographed. Furthermore, a method of Steven Seitz et al. (Steven M. Seitz and Charles R. Dyer. 1996. View morphing. 1996. View morphing. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques (SIGGRAPH '96). Association for Computing Machinery, New York, NY, USA, 21-30. DOI: https://doi.org/10.1145/237170.237196) may be used.

In addition, the synthesis of the captured images in the image distribution unit 102a is performed by, for example, using Structure from Motion technology (https://en.wikipedia.org/wiki/Structure_from_motion) to generate a three-dimensional shape from a plurality of captured images, Orthographic projection rendering may be performed from the three-dimensional shape to generate a facing image, which may be used as a distribution image.
In this way, since all areas of the delivery image can be an image facing the shooting target such as a product shelf, the user 601 does not feel uncomfortable even if any area of the delivery image is used as the display image. can be made

It should be noted that the synthesis of the captured images in this modified example is performed, for example, in step S106 in the flowchart of FIG. Also, the thumbnail image generated by the order processing unit 110 in step S108 is generated from the distribution image. In addition, the imaging device change request information may indicate a set of imaging devices instead of the imaging device, or when the imaging device change request information indicates imaging devices constituting a set of imaging devices. , may be regarded as denoting a set of imagers that includes the imager.

<Modification 2>
In the first modified example described above, a distribution image is generated by using a part of the imaging devices D1 to Dn as a set. may In this case, since there is only one distribution image, the processing related to the imaging device change request information is unnecessary. In Modification 2, in the image manipulation server 10 in the image manipulation system 1 in Modification 1, all of the imaging devices D1 to Dn will be described as one set.

FIG. 12 is a diagram showing a configuration example of the image operation server 10 in the image operation system 1 according to Modification 2. As shown in FIG. The image operation server 10 of this modified example differs from the image operation server 10 of the modified example 1 in that it includes an image distribution unit 102b instead of the image distribution unit 102a and does not include the imaging device adjacency information acquisition unit 103. different. The image delivery unit 102b generates a delivery image with all of the imaging devices D1 to Dn as one set. Others are the same as those of the image distribution unit 102a. Also, the user terminal 20 in this modified example has the same configuration as in FIG. 5, but does not require the function related to the imaging device change request information.

As described above, the image manipulation system 1 in the above-described embodiment and each modified example receives each captured image captured by each of the plurality of imaging devices D1 to Dn, and distributes an image based on at least one of the captured images. image distribution units 102, 102a, and 102b that distribute as image distribution units 102, 102a, and 102b, and a region to be displayed in the distribution image corresponding to physical information that indicates the state of the user's body and indicates at least the orientation of a predetermined portion of the user's body and an instruction to update the position of the display frame for extracting the display target area in the distribution image based on the operation information acquisition unit 25 for acquiring direction information including the direction in which the display target area is moved, and the display area information including the direction information. an image processing unit 22 for extracting an image surrounded by a display frame from a distributed image and using the extracted image as a display image; and an image output unit 23 for outputting the display image. The control unit 26 moves the position of the display frame in the distribution image at a speed corresponding to the direction indicated by the direction information. As a result, it is possible to easily perform processing that gives the user a feeling as if he or she were actually moving between locations where the plurality of image capturing devices D1 to Dn are capturing images.

It should be noted that each program for realizing each function of the image operation server 10 and the user terminal 20 of FIG. By loading and executing it in a computer system, processing of transition of imaging device and movement of sales floor is performed, processing of requesting change of imaging device that captures the provided distribution image, use from the supplied distribution image A process of generating a display image corresponding to the line of sight of the person may be performed. It should be noted that the "computer system" referred to here includes hardware such as an OS and peripheral devices.

Also, the "computer system" includes a WWW system provided with a home page providing environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible discs, magneto-optical discs, ROMs and CD-ROMs, and storage devices such as hard discs incorporated in computer systems. In addition, "computer-readable recording medium" means a volatile memory (RAM) inside a computer system that acts as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. , includes those that hold the program for a certain period of time.

Further, the above program may be transmitted from a computer system storing this program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in a transmission medium. Here, the "transmission medium" for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Further, the program may be for realizing part of the functions described above. Further, it may be a so-called difference file (difference program) that can realize the above-described functions in combination with a program already recorded in the computer system.

REFERENCE SIGNS LIST 1 image operation system 2 user management database 3 imaging device adjacency information storage unit 4 printer 10 image operation server 20 user terminal 21, 101 reception unit 22 image processing unit 23 image output unit 24 Body information recognition unit 25 Operation information acquisition unit 26 Display control unit 27, 104 Transmission unit 30 User interface unit 31 Display unit 32 Sensor unit 102, 102a, 102b Image distribution unit 103 Imaging device adjacency information Acquisition unit 110 Order processing unit 111 Thumbnail image generation unit 112 Destination information acquisition unit 113 Order form information output unit 321 Imaging unit 322 Motion sensor unit 500 Information communication network 601 User 602 Clerk D1, D2 , Dn... Imaging device

Claims (18)

an image distribution unit that receives each captured image captured by each of a plurality of imaging devices and distributes an image based on at least one of the captured images as a distribution image;
A direction including a direction to move a display target area, which is a display target area in the distribution image, corresponding to physical information indicating the physical condition of the user and indicating at least the orientation of a predetermined portion of the user's body. an operation information acquisition unit that acquires information;
a display control unit that gives an instruction to update a position of a display frame for extracting the display target area in the distribution image, based on the display area information including the direction information;
an image processing unit that extracts an image surrounded by the display frame from the distribution image and uses the extracted image as a display image;
an image output unit that outputs the display image,
The image manipulation system, wherein the display control unit moves the position of the display frame in the distribution image at a speed corresponding to the direction indicated by the direction information. The display control unit moves the display frame in the distribution image to the left or right at a speed corresponding to the leftward or rightward direction indicated by the direction information, and moves the display frame in the upward or downward direction indicated by the direction information. 2. The image manipulation system according to claim 1, wherein the display frame in the direction information is moved up or down accordingly. 3. The image manipulation system according to claim 1, wherein the predetermined part of the body is the head. The image manipulation system according to any one of claims 1 to 3, wherein the distribution image is an image obtained by synthesizing a plurality of the captured images. an imaging device adjacency information acquisition unit that acquires imaging device adjacency information indicating the adjacency relationship at each arrangement position of each of the imaging devices;
The image distribution unit selects a part of the plurality of imaging devices as a target imaging device, and if the target imaging device is one imaging device, the image distribution unit distributes the captured image captured by the one imaging device as a distribution image. When the target imaging device is a plurality of imaging devices, an image obtained by synthesizing the captured images captured by the plurality of imaging devices is distributed as a distribution image, and an imaging device change request instructing to change the part when information is input, switching the target imaging device in response to the imaging device change request information;
4. The display control unit according to any one of claims 1 to 3, wherein when a predetermined condition in the direction information is satisfied, the display control unit issues an instruction to change the target imaging device to another imaging device. 10. An image manipulation system according to claim 1. an imaging device adjacency information acquisition unit that acquires imaging device adjacency information indicating the adjacency relationship at each arrangement position of each of the imaging devices;
The image distribution unit selects one of the imaging devices as a target imaging device, distributes a captured image captured by the target imaging device as a distribution image, and includes imaging device change request information for instructing change of the imaging device. when input, switching the target imaging device in response to the imaging device change request information;
4. The display control unit according to any one of claims 1 to 3, wherein when a predetermined condition in the direction information is satisfied, the display control unit issues an instruction to change the target imaging device to another imaging device. 10. An image manipulation system according to claim 1. Each of the imaging devices is arranged at a different position in a store that sells products, and images different imaging ranges in the store, and the products are displayed in the imaging ranges. An image manipulation system according to any one of claims 1 to 6. further comprising an order processing unit that outputs order form information including an image of the product corresponding to the order information indicating the product purchased by the user;
The physical information includes hand position information indicating the position of the user's hand, which is a predetermined part of the user's body,
the operation information acquisition unit acquires position information indicating the position of the product in the display image corresponding to the hand position information;
The display control unit generates selected position information by converting the position indicated by the position information into a position in the distribution image in correspondence with the position of the display frame,
the display control unit, when an order condition for ordering the product indicated by the selected position information is satisfied, transmits the order information including at least the selected position information to the order processing unit;
The order processing unit
a thumbnail image generation unit that extracts an area including the position indicated by the selected position information included in the order information from the distribution image and generates a thumbnail image of the area;
a delivery destination information acquisition unit that acquires delivery destination information including the address of the user who ordered the product, which is the delivery destination of the product;
an order form information output unit that outputs order form information, which is order form information for the product in the thumbnail image, based on the thumbnail image and the delivery destination information;
8. The image manipulation system of claim 7, wherein: 9. The order condition according to claim 8, wherein the necessary condition is that a predetermined part of the body remains at the position indicated by the position information for a preset period of time. image manipulation system. The physical information is
including handprint information indicating gestures including poses and states of the user's hands, which are part of the user's body;
The operation information acquisition unit
obtaining gesture information indicating the type of the gesture based on the bill information;
if said order conditions are
A necessary condition is that the gesture indicated by the gesture information is a gesture of a type preset as a type of ordering,
10. The image manipulation system according to claim 8 or 9, characterized in that: a physical information recognition unit;
further comprising a user interface part and
The user interface unit
a display unit for displaying the display image output from the image output unit;
a motion sensor unit arranged on the head, which is a part of the body of the user, for transmitting head state information regarding the posture of the head to the body information recognition unit;
with
11. The physical information recognizing unit acquires the physical information including the orientation information of the head from the posture of the head in the head state information. The image manipulation system according to claim 1. a display unit for displaying the display image output from the image output unit;
a sensor unit that acquires the body state of the user as sensing information indicating the movement of the body;
11. The image manipulation system according to any one of claims 1 to 10, further comprising a body information recognition unit that extracts the body information of the user based on the sensing information. A server, and a user terminal connected to the server and possessed by the user,
the server
comprising the image distribution unit,
The user terminal is
the operation information acquisition unit;
the display control unit;
the image processing unit;
13. The image manipulation system according to claim 1, further comprising the image output unit. Connect to a user terminal that acquires direction information from body direction information indicating a predetermined direction of a part of the user's body, and moves the position of the display frame in the distribution image at a speed corresponding to the direction indicated by the direction information. and an image operation server that transmits the distribution image to the user terminal,
An image operation server, comprising: an image distribution unit that receives captured images captured by each of a plurality of imaging devices and distributes an image based on at least one of the captured images as the distribution image. A user terminal that receives an image captured by each of a plurality of imaging devices, and is connected to a server that distributes an image based on at least one of the captured images as a distribution image, the user terminal receiving the distribution image;
Operation information for acquiring direction information including a direction in which the display target area in the distribution image is moved, corresponding to physical information indicating the user's physical condition and indicating at least the orientation of a predetermined portion of the user's body. an acquisition unit;
a display control unit that instructs to update a position of a display frame for selecting a display target area in the distribution image, based on the display area information including the direction information;
an image processing unit that extracts an image surrounded by the display frame from the distribution image and generates a display image of the display target area;
and an image output unit that outputs the display image. an image distribution process in which an image distribution unit receives each captured image captured by each of a plurality of imaging devices and distributes an image based on at least one of the captured images as a distribution image;
A direction in which the operation information acquisition unit indicates the physical condition of the user and includes the direction in which the display target area in the distribution image is moved, corresponding to the physical information indicating at least the orientation of a predetermined portion of the user's body. an operation information acquisition process for acquiring information;
a display control process in which a display control unit issues an instruction to update a position of a display frame for selecting a display target area in the distribution image based on the display area information including the direction information;
an image processing step in which an image processing unit extracts an image surrounded by the display frame from the distribution image and generates a display image of the display target area;
and an image output process in which an image output unit outputs the display image. Connect to a user terminal that acquires direction information from body direction information indicating a predetermined direction of a part of the user's body, and moves the position of the display frame in the distribution image at a speed corresponding to the direction indicated by the direction information. a program for operating a computer as an image operation server for transmitting the distribution image to the user terminal,
said computer,
Image distribution means for receiving each captured image captured by each of a plurality of imaging devices and distributing an image based on at least one of the captured images as the distribution image;
A program that operates as A computer serving as a user terminal for receiving a distributed image, connected to a server that receives each captured image captured by each of a plurality of imaging devices and distributes an image based on at least one of the captured images as a distributed image. is a program that runs
Operation information for acquiring direction information including a direction in which the display target area in the distribution image is moved, corresponding to physical information indicating the user's physical condition and indicating at least the orientation of a predetermined portion of the user's body. acquisition means,
display control means for instructing to update the position of the display frame for selecting the display target area in the distribution image, based on the display area information including the direction information;
image processing means for extracting an image surrounded by the display frame from the distribution image and generating a display image of the display target area;
A program operated as image output means for outputting the display image.

Images