JP2023038203A - Display device, display method, program and image projection system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a display device and the like that can preferably present a content image associated with a living creature to an observer.

SOLUTION: A display device 1 comprises: an acquisition unit that acquires picked-up images obtained by picking up images of a plurality of living creatures; a recognition unit that recognizes a living creature group composed of the same kind of living creatures from the picked-up images; a content acquisition unit that acquires a content image related to the living creature group; a determination unit that determines a display position of the content image on a display surface which is provided between the plurality of living creatures and an observer on the basis of a recognition position of the living creature group; and a display unit that displays the content image on the display surface on the basis of the determined display position. The picked-up images are moving images, and the recognition unit extracts feature points corresponding to living creatures from a first image and a second image continuing in time series in the moving images, detects movement of the living creatures on the basis of the difference between coordinate values of the feature points extracted from the first image and the second image, and recognizes the living creature group on the basis of the movement of the plurality of living creatures.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2023,JPO&INPIT

Description

The present invention relates to a display device, display method, program, and image projection system.

It has been proposed to use transmissive projector screens, or transmissive displays, with a see-through background to present images related to objects behind the screen or display. For example, in Patent Document 1, a projector is installed inside a water tank of an aquarium, and a transparent screen is provided on a transparent wall of the water tank such as a glass window for an observer (viewer) to observe the inside of the tank. A display device is disclosed. According to Patent Document 1, a camera installed outside the aquarium captures an image of the inside of the aquarium, the fish swimming in the aquarium is recognized from the captured image, and an image such as an explanation of the fish is displayed at an appropriate position on the screen by a projector. indicate.

JP-A-2005-338115

However, when considering a tank of an aquarium, for example, a plurality of fishes are swimming in one tank. Therefore, when a plurality of fish are swimming in a school, and an image such as an explanation is displayed for each individual fish, the images corresponding to each fish are superimposed so that the important fish is not visible to the observer. There is a possibility that it cannot be seen.

An object of one aspect of the present invention is to provide a display device or the like capable of suitably presenting a content image related to living things to an observer.

In one aspect, a display device includes an acquisition unit that acquires captured images obtained by imaging a plurality of organisms, and from the captured images, an organism group composed of the organisms of the same type, and an image feature amount of the organism group are learned. a recognition unit that recognizes based on a trained model; a content acquisition unit that acquires a content image related to the group of organisms; and a display surface provided between the plurality of organisms and an observer based on the recognized position of the group of organisms. and a display unit for displaying the content image on the display surface based on the determined display position.

In one aspect, it is possible to preferably present a content image related to living things to an observer.

1 is a schematic diagram showing a configuration example of an image display system; FIG. It is a block diagram which shows the structural example of PC. FIG. 4 is an explanatory diagram showing an example of a record layout of a label DB; FIG. FIG. 4 is an explanatory diagram showing a display example of a label image; FIG. 10 is an explanatory diagram relating to processing for changing the display position of a label image; FIG. 10 is an explanatory diagram relating to processing for changing the display mode of a label image; 4 is a flow chart showing an example of a processing procedure executed by a PC; FIG. 11 is an explanatory diagram of fish school recognition processing according to Embodiment 2; FIG. 11 is an explanatory diagram regarding label tracking processing according to Embodiment 2; 10 is a flow chart showing an example of a processing procedure executed by a PC according to Embodiment 2; FIG. 11 is an explanatory diagram relating to fish school recognition processing according to Embodiment 3; 10 is a flow chart showing an example of a processing procedure executed by a PC according to Embodiment 3; FIG. 13 is an explanatory diagram showing a configuration example of a screen according to Embodiment 4; FIG. 12 is a flow chart showing an example of a processing procedure executed by a PC according to Embodiment 4; FIG. FIG. 13 is an explanatory diagram showing a display example of a label image according to Embodiment 5; FIG. 13 is a flow chart showing an example of a processing procedure executed by a PC according to Embodiment 5; FIG. FIG. 12 is a schematic diagram showing a configuration example of an image display system according to Embodiment 6; FIG. 12 is an explanatory diagram showing an overview of Embodiment 6; 13 is a flow chart showing an example of a processing procedure executed by an image output device according to Embodiment 6; FIG. 3 is a functional block diagram showing the operation of the image output device of the form described above;

Hereinafter, the present invention will be described in detail based on the drawings showing its embodiments. Embodiment 1 FIG. 1 is a schematic diagram showing a configuration example of an image display system. In the present embodiment, a transmissive screen 4 (display surface) is provided on a transmissive wall such as a glass window forming one surface of the aquarium tank, and a label image (contents) for describing the fish in the aquarium is provided on the screen 4. An image display system (image projection system) for displaying an image) will be described. The image display system has an image output device (display device) 1 , a camera 2 and a projector 3 . Each device is connected for communication by a network such as a LAN (Local Area Network).

The image output device 1 is an information processing device capable of various types of information processing and transmission/reception of information, such as a personal computer and a server device. In the present embodiment, the information processing apparatus 1 is assumed to be a personal computer, which will be read as a PC 1 for the sake of simplicity. The PC 1 stores, as contents, label images for describing various fish swimming in the tank, outputs the label images to the projector 3 , and projects (displays) them on the screen 4 . In the present embodiment, for convenience of explanation, organisms in the aquarium are referred to as "fish" even if they are organisms other than fish, such as mammals, mollusks, and arthropods.

In this embodiment, the PC 1 acquires a captured image of a water tank from the camera 2, recognizes various fish from the captured image, and performs display control so as to project a label image of the recognized fish on a location near the screen 4. In this case, the PC1 identifies whether the fish are swimming alone or in a group in the tank. When recognizing a school, that is, a school of fish, the PC 1 performs display control so as to project one label for the entire school of fish instead of projecting a label image for each fish constituting the school of fish.

The camera 2 is an imaging device that images the inside of an aquarium (space) containing a plurality of fish through a transparent wall such as a glass window for an observer (viewer) to observe the inside of the aquarium. For example, it is installed on the ceiling of an aquarium. The camera 2 takes images of the inside of the water tank through the transparent wall, and sequentially transmits the captured images to the PC 1 as moving images.

The projector 3 is a projection device that projects a label image. The projector 3 is installed on the opposite side of the water tank through the screen 4, that is, outside the water tank, and projects the label image onto the screen 4 provided on the transparent wall. As in the invention described in Patent Document 1, when a projector is installed inside a water tank and an image is projected from the inside of the water tank onto a transmissive screen, there is a risk that the label image may be blocked by fish swimming in the water tank. On the other hand, by installing the projector 3 outside the water tank, not inside the water tank, as in the present embodiment, it is possible to prevent a situation in which the projected light from the projector 3 is blocked by the fish in the water tank.

The screen 4 is a transmissive screen provided on the transmissive wall of the aquarium located between the fish in the aquarium and the observer, and is a transparent screen film that diffuses the irradiated light. For example, the screen 4 is affixed to the entire transparent wall of the water tank, and forms an image by reflecting forward image light projected (projected) from the front.

In this embodiment, the projector 3 and the screen 4 are used to project (display) an image. However, instead of the projector 3 and the screen 4, a transmissive display may be installed on the transmissive wall of the water tank. good. A transmissive display is installed on the transmissive wall of the water tank, and P
Even if C1 is configured to display an image on a transmissive display, the same effect can be obtained.

FIG. 2 is a block diagram showing a configuration example of the PC1. The PC 1 includes a control section 11 , a main storage section 12 , a communication section 13 , a display section 14 , an input section 15 and an auxiliary storage section 16 .
The control unit 11 has an arithmetic processing unit such as one or more CPU (Central Processing Unit), MPU (Micro-Processing Unit), etc. By reading and executing the program P stored in the auxiliary storage unit 16, Various information processing, control processing, and the like relating to the PC 1 are performed. The main storage unit 12 is a temporary storage area such as a RAM (Random Access Memory), and temporarily stores data necessary for the control unit 11 to execute arithmetic processing. The communication unit 13 includes a processing circuit and the like for performing processing related to communication, and transmits and receives information to and from the outside. A display unit 14 is a display device such as a liquid crystal display, and displays an image given from the control unit 11 . The input unit 15 is an operation interface such as a keyboard and a mouse, and receives operation input.

The auxiliary storage unit 16 is a non-volatile storage area such as a large-capacity memory or hard disk, and stores programs P and other data necessary for the control unit 11 to execute processing. The auxiliary storage unit 16 may be an external storage device connected to the PC1. Auxiliary storage unit 16 stores label DB 161 and identification model 162 . The label DB 161 is a database storing data of label images, which are contents to be projected on the screen 4 . The identification model 162 is identification data for identifying the fish in the aquarium from the image captured by the camera 2, and is a learned model constructed by machine learning, for example, as described later.

FIG. 3 is an explanatory diagram showing an example of the record layout of the label DB 161. As shown in FIG. The label DB 161 includes an ID column, a type column, and a label image column. The ID column stores an ID for identifying each fish swimming in the tank. The type column stores the type of each fish in association with the ID. The label image column stores label images for guiding each fish to the observer in association with the ID.

FIG. 4 is an explanatory diagram showing a display example of the label image 41. As shown in FIG. FIG. 4 conceptually illustrates how a fish swimming in a tank is recognized from an image captured by the camera 2 and a label image 41 related to the recognized fish is displayed in the vicinity of the recognized position of the fish. Based on FIG. 4, the outline of this embodiment will be described.

First, the PC 1 successively acquires, as a moving image, images taken from the camera 2 through the transparent wall of the water tank provided with the screen 4 . FIG. 4 schematically shows one frame of a moving image acquired by the PC 1. As shown in FIG. Here, as shown in FIG. 4, it is assumed that there are fish such as "Japanese horse mackerel" swimming in a school and fish swimming alone such as "manta ray".

The PC 1 recognizes each fish swimming in the tank from the captured image. Here, PC1 not only recognizes individual fish, but also distinguishes between fish swimming alone and fish swimming in groups.

For example, the PC 1 uses an identification model 162 set in advance in the auxiliary storage unit 16 to identify a fish swimming alone and a fish swimming in a school of fish. The discrimination model 162 is a learned model constructed by machine learning, and is a discriminator that has learned the image feature amount of a single swimming fish and the image feature amount of an entire school of fish consisting of a plurality of fish of the same type. For example, the discrimination model 162 is a CNN (Convolution Neural Network) generated from teacher data, but the machine learning method and the configuration of the discriminator are not particularly limited. The discriminative model 162 may be generated according to other algorithms such as other neural networks, SVMs (Support Vector Machines), Bayesian networks, and decision trees. For example, the identification model 162 is generated based on teacher data in which an image of an individual fish or an entire school of fish is associated with a label value indicating the name of the fish (individually or of the school of fish).

The PC 1 inputs the captured image acquired from the camera 2 to the identification model 162, and obtains the identification result of identifying the object included in the captured image as an output value. As a result, the PC 1 recognizes individual fish swimming in the water tank after distinguishing between individual fish and schools of fish, as indicated by the dotted rectangular frame in FIG. 4 .

The PC 1 determines the display position (projection position) of the label image 41 to be displayed (projected) on the screen 4 corresponding to each fish, based on the recognized position of each fish (single fish or school of fish) in the captured image. For example, as shown in FIG. 4, the PC 1 determines the display position immediately below the position where the fish is recognized so that the label image 41 is displayed near the fish.

The PC 1 reads the label image 41 corresponding to the recognized fish from the label DB 161, outputs it to the projector 3, and projects (displays) it on the screen 4 provided between the fish in the tank and the observer. The label image 41 is the name of the fish, but the label image 41 may include information other than the name. Also, the label image 41 is not limited to text, and may be a picture image or the like.

As described above, when the fish in the aquarium form a school of fish and swim, the PC 1 does not display the label image 41 for each individual fish in the school of fish, but displays one label image 41 for the entire school of fish. display. As a result, a large number of label images 41 are not displayed on the screen 4, and it is possible to avoid a situation where the label images 41 are overlapped and the fish cannot be seen.

5A and 5B are explanatory diagrams related to the process of changing the display position of the label image 41. FIG. FIG. 5 illustrates how the display position of the label image 41 is changed so as not to overlap other fish. It has been described above that the PC 1 displays the label image 41 near the fish in the tank. Here, when the label image 41 is superimposed on another fish, the PC 1 changes the display position of the label image 41 so that it is not superimposed.

In the example of FIG. 5, in addition to the various fish shown in FIG. 4, a "whale shark" is included in the imaging range. In this state, when the label image 41 is displayed in the same manner as in FIG. 4, the upper left label image 41 of "maji" is displayed in the position of the thick dotted rectangular frame in FIG. becomes difficult to see.

After recognizing each fish from the captured image and determining the display position of the label image 41, the PC 1 compares the position of each fish with the display position of the label image 41 and determines that the display position of the label image 41 is superimposed on the other fish. determine whether or not to When determining that the label image 41 is superimposed, the PC 1 changes the display position so that the label image 41 is not superimposed. In the example of FIG. 5, the PC 1 changes the label image 41 of "Japanese horse mackerel" from the lower side of the school of fish to the upper side to avoid superimposition.

By changing the display position of the label image 41 in this manner, it is possible to avoid a situation in which the label images 41 overlap and the fish cannot be seen.

Although the display position of the label image 41 is changed depending on whether or not the label image 41 is superimposed on the fish, the present embodiment is not limited to this. For example, when the position of the fish recognized from the captured image overlaps with the edge (outer edge) of the screen 4 (display surface), the PC 1 may change the display position so as to bring the label image 41 closer to the center of the screen 4 .

For example, in FIG. 5 , a school of “Japanese horse mackerel” is located at the right end, but part of the school of fish is out of the installation range of the screen 4 , and the school of fish overlaps the edge of the screen 4 . In such a case, if the entire school of fish is recognized, including the portion outside the range of the screen 4, and the label image 41 is projected onto the screen 4, there is a risk that the label image 41 will be projected outside the range of the screen 4. be.

Therefore, for example, the PC 1 determines whether or not the recognition position of the fish (school of fish) recognized from the captured image overlaps with the end (outer edge) of the screen 4. , the display position of the label image 41 is changed toward the center. As a result, the label image 41 can be displayed at an appropriate position even for the fish positioned at the edge of the screen 4 .

6A and 6B are explanatory diagrams related to the processing for changing the display mode of the label image 41. FIG. FIG. 6 illustrates how the display mode of the label image 41 is changed according to the size of the school of fish recognized from the captured image. 4 and 5 show how the label image 41 is similarly displayed for any fish. However, for example, if the tank is large and the school of fish swimming in the tank is also large, even if the label image 41 is displayed for the large school of fish in the same way as for other fish, It may be difficult for an observer to notice or visually recognize.

Therefore, when the PC 1 recognizes the school of fish from the captured image, it determines the scale of the school of fish. For example, the PC 1 determines the scale according to the size of the recognition range (dotted rectangular frame in FIG. 6) of the school of fish occupying the captured image. Alternatively, the PC 1 may determine the scale according to the number of fish forming the school of fish.

When the school of fish is determined to be of a predetermined size or larger, the PC 1 changes the display mode of the label image 41 corresponding to the school of fish. For example, the PC 1 displays the label image 41 of the fish school in a larger size than the label images 41 of other fish (single fish or fish school). This allows the observer to easily visually recognize the label image 41 even for a large school of fish.

In the above description, the label image 41 of the large school of fish is enlarged, but the label image 41 of the other fish may be reduced to relatively emphasize the label image 41 of the school of fish. Also, the change in display mode is not limited to enlargement or reduction. For example, the PC 1 may display the same label image 41 at multiple locations for a large school of fish so that the observer can easily find any one of the label images 41 . In this way, the PC 1 only needs to be able to change the display mode of the label image 41 according to the scale of the school of fish, and the display mode is not particularly limited.

FIG. 7 is a flowchart showing an example of a processing procedure executed by the PC1. Based on FIG. 7, the contents of processing executed by the PC 1 will be described. The control unit 11 of the PC 1 acquires, from the camera 2, captured images of a plurality of fish (creatures) swimming in the water tank (moving space) (step S11). Based on the identification model 162, the control unit 11 performs processing for recognizing fish from the captured image (step S12). For example, as described above, the discrimination model 162 is a learned model constructed by machine learning, and is a discriminator that has learned the image feature amount of a single fish and an entire school of fish. The control unit 11 inputs the captured image acquired from the camera 2 to the classifier, and acquires the identification result of identifying a single fish swimming alone in the tank and a group of fish swimming in the tank.

As a result of the process of step S12, the control unit 11 determines whether or not a school of fish has been recognized from the captured image (step S13). If it is determined that a school of fish has been recognized (S13: YES), the control unit 11 displays (projects) a label image related to the school of fish on the screen 4 based on the recognition position in the captured image where the school of fish is recognized ( projection position) is determined (step S14).

Based on the recognition result in step S12, the control unit 11 determines whether or not the school of fish is a school of fish of a predetermined size or larger (step S15). For example, the control unit 11 determines the scale according to the recognition range of the school of fish in the captured image, the number of fish forming the school of fish, or the like. If it is determined that the school of fish is larger than the predetermined size (S15: YES), the control unit 11 changes the display mode of the label image of the school of fish (step S16). For example, the control unit 11 performs enlarged display, multiple display, and the like of the label image.

After executing the process of step S16 or in the case of NO in steps S13 and S15, the control unit 11 detects another fish that has not been recognized as a school of fish, that is, a fish swimming alone. , the display position for displaying the label image of the fish on the screen 4 is determined (step S17).

The control unit 11 determines whether or not the display position of the label image determined in step S14 or S17 overlaps with another fish (single fish or school of fish) (step S18). If it is determined to be superimposed (S18: YES), the control unit 11 changes the display position so that the label image is not superimposed (step S19).

After executing the process of step S19, or in the case of NO in step S18, the control unit 11 outputs the label image of the fish (single or school of fish) recognized in step S12 to the projector 3, and Project (display) at the display position (step S20). The control unit 11 returns the process to step S11.

As described above, according to the first embodiment, the school of fish is recognized from the captured image, and the label image 41 is displayed for each school of fish. This reduces the risk that the label image 41 will be displayed for each individual fish forming a school of fish and will overlap the important fish, and the label image 41 relating to the organism can be preferably presented to the observer.

Further, according to the first embodiment, the identification model 162 constructed by machine learning is used to recognize a school of fish and display the label image 41 . By using a machine learning algorithm, the fish in the tank can be recognized with high accuracy, and the label image 41 can be presented more appropriately.

Further, according to the first embodiment, the transmissive screen 4 is provided on the transmissive wall (observation surface) of the water tank, and the label image 41 is projected (displayed) from the front of the water tank, that is, from the outside of the water tank. Therefore, unlike the video display device described in Patent Document 1, there is less risk that an image will be projected from the inside of the water tank and will overlap the fish, and the label image 41 can be presented more favorably.

Further, according to the first embodiment, instead of the projector 3 and the screen 4, a transmissive display may be provided on the transmissive wall of the water tank. Even in this case, the label image 41 can be preferably presented while avoiding image overlap.

Further, according to the first embodiment, by changing the display mode of the label image 41 according to the scale of the school of fish, it is possible to serve the convenience of the observer.

Further, according to the first embodiment, by changing the display position of the label image 41 so that it does not overlap with other fish, it is possible to more effectively prevent the label image 41 from overlapping the fish and becoming invisible. can do

(Embodiment 2)
In the first embodiment, the deep learning technique is used to learn the characteristics of a school of fish, and the learned model is used to recognize the school of fish. In the present embodiment, a method such as optical flow is used to identify the movement of each individual fish, and a form of recognizing a school of fish from the movement of each identified individual fish will be described. In addition, the same code|symbol is attached|subjected about the content which overlaps with Embodiment 1, and description is abbreviate|omitted.
FIG. 8 is an explanatory diagram of fish school recognition processing according to the second embodiment. FIG. 8A illustrates how feature points are extracted from a captured image, optical flow is calculated, and movement of an object (fish) moving in a water tank is detected. FIG. 8B illustrates how a school of fish is recognized from the movements of a plurality of objects. Based on FIG. 8, the outline of this embodiment will be described.

As in the first embodiment, the PC 1 sequentially acquires captured images of the water tank from the camera 2 as moving images. Here, the PC 1 uses, for example, an optical flow technique to identify the movement of the object moving in the water tank by comparing each image (frame) included in the moving image.

Specifically, the PC 1 extracts feature points from each of two images (first and second images) that are continuous in time series in the moving image. A feature point is calculated from the brightness (luminance) of each pixel in the image. The PC1 calculates a vector value representing the movement of the feature point, that is, the optical flow, based on the difference in the coordinate values of the feature points between the two images. FIG. 8A illustrates how the optical flow is calculated. In FIG. 8A, white circles represent feature points, and arrows represent vectors corresponding to optical flows.

PC1 detects individual objects, that is, fish, moving in the water tank from the optical flow corresponding to one or more feature points. For example, as shown in FIG. 8A, PC1 detects the plurality of feature points as one moving object (fish) when the optical flows associated with the plurality of adjacent feature points indicate vector values in the same direction and at the same speed. . Note that the PC 1 may detect a moving object from the motion (optical flow) of one feature point.

After detecting the movements of individual objects moving in the aquarium, the PC 1 compares the movements of a plurality of objects to detect (recognize) the objects forming a group and moving in the aquarium, that is, the school of fish. For example, as shown in FIG. 8B, PC1 detects a plurality of objects as a school of fish when the optical flows of a plurality of objects positioned within a predetermined range indicate vector values in substantially the same direction and at substantially the same speed.

The PC 1 performs image recognition on one object (fish) that constitutes the detected school of fish, and identifies what kind of school of fish it is. The PC 1 similarly performs image recognition on other fish (single fish or school of fish) to specify what kind of fish it is. The PC 1 outputs the label image 41 of each identified fish to the projector 3 and projects it at a position near each fish.

FIG. 9 is an explanatory diagram of label tracking processing according to the second embodiment. FIG. 9 conceptually illustrates a tracking display in which the label image 41 is moved in accordance with the detected movement of the school of fish. As described above, PC1 detects the movement of fish (single or school of fish) in the tank. The PC 1 estimates where the fish will move next from the detected movement of the fish, and performs tracking display by moving the label image 41 to the estimated position. The PC 1 estimates the moving direction and moving speed from the optical flow representing the movement of the fish, and controls the projector 3 to move the display position of the label image 41 according to the estimated direction and speed. In this way, by detecting the movement of fish, it is possible not only to recognize a school of fish, but also to perform display control so as to move the label image 41 to a suitable position.

FIG. 10 is a flow chart showing an example of a processing procedure executed by the PC 1 according to the second embodiment.
After acquiring the captured image from the camera 2 (step S11), the control section 11 of the PC 1 executes the following processes. The control unit 11 extracts feature points included in the captured image (step S201). For example, the control unit 11 extracts feature points according to the brightness of each pixel included in the imaging pixels.

The control unit 11 controls the inside of the water tank based on the difference in the coordinate values of the feature points extracted from each of the two images (first and second images) that are consecutive in time series among the images that are sequentially acquired as moving images. A movement of a moving object (fish) is detected (step S202). For example, the control unit 11 calculates the optical flow for each feature point, and detects the motion of each moving object based on the optical flow of one or more adjacent feature points.

Based on the detected motion of each object, the control unit 11 determines whether or not a plurality of objects are moving as a group (step S203). If it is determined that the fish are moving in a group (S203: YES), the control unit 11 performs image recognition on any object that constitutes the group, and identifies the type of fish (step S204). The control unit 11 shifts the process to step S14.

After executing the process of step S16, or in the case of NO in steps S203 and S15, the control unit 11 performs image recognition on an object that is moving alone and identifies the type of fish (step S205). The control unit 11 shifts the process to step S17.

After executing the processing of step S19, or in the case of NO in step S18, the control unit 11 outputs the label image 41 to the projector 3 to project (display) it (step S206). In this case, the control unit 11 estimates the movement direction and movement speed of the fish (single fish or school of fish) corresponding to the label image 41 based on the movement of the fish detected in step S202. The control unit 11 performs tracking display in which the display position of the label image 41 is moved according to the estimated moving direction and moving speed. The control unit 11 returns the process to step S11.

As described above, according to the second embodiment, the movement of fish is detected by a technique such as optical flow, and a school of fish is detected from the detected movement of each individual fish. As a result, it is possible to recognize the school of fish and present the label image 41 even if it does not have a trained model.

Further, according to the second embodiment, it is possible to estimate where the fish will move from the detected movement of the fish, and to track and display the label image 41 . As a result, positional deviation between the fish and the label image 41 can be avoided, and the label image 41 can be preferably presented.

(Embodiment 3)
In the present embodiment, an embodiment will be described in which all fish are recognized from a captured image and a school of fish is identified from the positional relationship of each recognized fish.
FIG. 11 is an explanatory diagram relating to fish school recognition processing according to the third embodiment. In the second embodiment, a form of identifying individual fish after detecting a school of fish was described, but in the present embodiment, individual fish are identified first, and the school of fish is recognized from the positional relationship of the identified fish.

Specifically, the PC 1 performs image recognition on the captured image and recognizes all fish swimming in the tank. In the example of FIG. 11, the PC1 recognizes not only the "spotted dolphin" and the "manta ray" but also the four "bluefin tuna" forming a school of fish as "bluefin tuna".

After that, the PC 1 recognizes a school of fish formed by a plurality of fish of the same species based on the species and positional relationship of the plurality of fish. For example, the PC 1 recognizes a school of fish depending on whether fish (individuals) of the same type and a predetermined number or more are located within a predetermined range. In the example of FIG. 11 , a plurality of fish having a common type of “bluefin tuna” are concentrated in the upper image area of the captured image. When the plurality of fish are located within a predetermined range, the PC 1 recognizes the plurality of fish as a school of "bluefin tuna".

FIG. 12 is a flow chart showing an example of a processing procedure executed by the PC 1 according to the third embodiment.
After acquiring the captured image from the camera 2 (step S11), the control section 11 of the PC 1 executes the following processes. The control unit 11 performs image recognition on the captured image, and recognizes all of the plurality of fish in the tank (step S301). Based on the recognized positional relationship of each fish, the control unit 11 determines whether or not a plurality of fish form a school of fish (step S302). For example, as described above, the control unit 11 determines whether or not a school of fish is formed by determining whether or not a predetermined number or more of fish of the same type are located within a predetermined range. If it is determined that a school of fish is formed (S302: YES), the control section 11 shifts the processing to step S14. If it is determined that a school of fish is not formed (S302: NO), the control section 11 shifts the processing to step S17.

As described above, according to the third embodiment, all fish are recognized from the captured image, and the label image 41 is displayed by identifying the school of fish based on the type, number, and positional relationship of the recognized fish. Thereby, the label image 41 can be preferably presented to the observer.

(Embodiment 4)
In this embodiment, the screen 4 is provided only in a partial region of the permeable wall of the water tank, and the screen 4
A form will be described in which the label image 41 is projected only when the fish passes through the area provided with .
FIG. 13 is an explanatory diagram showing a configuration example of the screen 4 according to the fourth embodiment. In this embodiment, the screen 4 is provided not on the entire surface of the transmission wall of the aquarium, but only on a partial area. For example, as shown in FIG. 13, the screen 4 is provided on the left half surface of the transmissive wall when viewed from the front.

Note that the range in which the screen 4 is provided is not limited to the example shown in FIG. Moreover, the ratio of the area of the screen 4 to the entire surface of the transmission wall is not particularly limited.

The PC 1 projects (displays) the label image 41 only when the fish passes through the area where the screen 4 is provided when viewed from the front. For example, the PC 11 recognizes all fish (single fish or a school of fish) in the tank from the captured image as shown in FIG. After that, the PC 11 identifies fish passing through the installation area of the screen 4 . For example, when projecting (displaying) the label image 41 corresponding to each fish, the PC 11 determines whether or not the display position of the label image 41 is within the installation area of the screen 4 . When the PC 1 determines that it is within the installation area of the screen 4 , the PC 1 controls the projector 3 to project the label image 41 onto the screen 4 .

FIG. 14 is a flow chart showing an example of a processing procedure executed by the PC 1 according to the fourth embodiment. After executing the processing of step S19, or in the case of NO in step S18, the control unit 11 of the PC 1 executes the following processing. The control unit 11 determines whether the display position of the label image 41 determined based on the recognition result of the captured image is within the area where the screen 4 is provided on the transparent wall of the water tank (step S401). When it is determined that it is within the area where the screen 4 is provided (S401: YES), the control unit 11 projects (displays) the label image 41 on the screen 4 (step S402). After executing the process of step S402, or in the case of NO in step S401, the control unit 11 returns the process to step S11.

In the above description, all fish in the tank are recognized and then fish passing through the area of the screen 4 are identified, but the present embodiment is not limited to this. For example, the camera 2 may capture only the area where the screen 4 is provided, and the PC 1 may recognize the fish from the narrowed image, thereby identifying the fish passing within the range of the screen 4 . That is, the PC 1 only needs to be able to recognize the fish from the image area of the captured image corresponding to the partial area of the transparent wall provided with the screen 4, and project the label image 41 onto the screen 4. is not essential to recognize all fish by imaging.

As described above, according to the fourth embodiment, the display area of the label image 41 can be limited to a partial area of the transparent wall of the water tank, and a flexible configuration can be achieved according to the actual operation.

(Embodiment 5)
In this embodiment, a form will be described in which the display position of the label image 41 is determined according to the position of the observer.
FIG. 15 is an explanatory diagram showing a display example of the label image 41 according to the fifth embodiment. FIG. 15 conceptually illustrates how the display position of the label image 41 is controlled so as to approach the position of the observer.

In this embodiment, the PC 1 recognizes not only the fish in the tank but also the observer observing the tank from the captured image. For example, the camera 2 is set with the inside of the tank and a predetermined range in front of the tank where the observer is positioned as an imaging range, and the PC 1 detects the observer around the tank from the image captured by the camera 2 in the imaging range. recognize. Alternatively, the camera 2 may be configured as an omnidirectional camera to capture an image of the entire aquarium room in which the aquarium is installed, and to recognize the observer in front of the aquarium. Alternatively, a camera 2 for imaging the inside of the aquarium and a camera 2 for imaging the front of the aquarium may be provided separately, and the observer may be recognized from the image captured by the latter camera 2 .

The PC 1 controls the display position of the label image 41 according to the observer's position recognized from the captured image. In the example of FIG. 15, the observer is positioned substantially in the center of the water tank in the horizontal direction. In this case, the PC 1 controls the display position so that the label image 41 is positioned closer to the center than when there is no observer. In the example of FIG. 15, the PC 1 shifts the label images 41 of "bluefin tuna", "manta ray", and "great horse mackerel" located on the right side to the left, and shifts the label images 41 of "big horse mackerel" and "spotted dolphin" located on the left side to the left. shifts to the right. Thus, the PC 1 controls the display position so that the label image 41 approaches the observer's position.

By the above-described processing, the label image 41 can be shifted to a position that is easy for the observer to see, and the label image 41 can be presented more preferably.

FIG. 16 is a flow chart showing an example of a processing procedure executed by the PC 1 according to the fifth embodiment. After acquiring the captured image from the camera 2 (step S11), the control section 11 of the PC 1 executes the following processes. The control unit 11 recognizes a fish (single fish or a school of fish) from the captured image, and also recognizes an observer observing the inside of the tank (step S501). The control unit 11 shifts the process to step S13.

If it is determined that a school of fish has been recognized from the captured image (S13: YES), the control unit 11 adjusts the display position of the label image 41 corresponding to the school of fish based on the position of the school of fish recognized from the captured image and the position of the observer. is determined (step S502). For example, as described with reference to FIG. 15, the control unit 11 determines the display position of the label image 41 so that the label image 41 approaches the position of the observer from the display position when the observer is not recognized. The control unit 11 shifts the process to step S15.

After executing the process of step S16, or in the case of NO in steps S13 and S15, the control unit 11 generates a label image 41 of a single swimming fish based on the position of the single fish recognized from the captured image and the position of the observer. is determined (step S503). Specifically, similarly to step S502, the control unit 11 determines the display position so that the label image 41 approaches the position of the observer. The control unit 11 shifts the process to step S18.

As described above, according to the fifth embodiment, the display position of the label image 41 can be determined in consideration of the position of the observer, and the label image 41 can be presented more preferably.

(Embodiment 6)
In the first embodiment, the screen 4 is provided on the transparent wall of the water tank of the aquarium, and the label image 41 related to the fish swimming in the water tank is projected (displayed) on the screen 4. FIG. In the present embodiment, when observing animals (creatures) from inside a vehicle, such as in a safari park, the window of the vehicle is configured as a transmissive display 604 and the label image 41 is displayed on the window of the vehicle. FIG. 17 is a schematic diagram illustrating a configuration example of an image display system according to Embodiment 6. FIG. The image display system according to this embodiment has an image output device 1 , a camera 602 and a transmissive display 604 . For example, the image output device 1 may be configured as a personal computer as in the first embodiment, or may be configured as an in-vehicle device such as an ECU (Electronic Control Unit).

The camera 602 is an imaging device that images the surroundings of the vehicle, such as an omnidirectional camera. The camera 602 captures omnidirectional 360-degree images from front to back, right to left, and the like when the direction of travel of the vehicle is forward. Instead of using an omnidirectional camera as the camera 602, for example, cameras 602 may be provided at the front, rear, and left and right sides of the vehicle, and a plurality of cameras 602 may be configured to capture images in all directions. Further, in the present embodiment, since the transmissive display 604 is provided only on the side windows of the vehicle, the camera 602 may be configured to image only the left and right directions of the vehicle instead of capturing images in all directions.

The transmissive display 604 is a display device such as an organic EL (Electro Luminescence) display or a liquid crystal display, and is a display device in which areas other than display objects such as the label image 41 are transparent. The transmissive display 604 is provided, for example, on the side window of the vehicle shown in FIG. Instead of the transmissive display 604, a projector system using the transmissive screen 4 as in the first embodiment may be used to display an image on the vehicle side window. The transmissive displays 604 are provided in a plurality of windows on the left and right sides of the vehicle, respectively, and display images for observers (safari park visitors, etc.) riding in the vehicle.

Note that the transmissive display 604 may be provided not only on the side windows of the vehicle but also on the front and rear windows of the vehicle.

FIG. 18 is an explanatory diagram showing an overview of the sixth embodiment. FIG. 18 illustrates how the camera 602 captures an image of an animal present around the vehicle. Based on FIG. 18, the outline of this embodiment will be described.
The image output device 1 sequentially acquires captured images of the surroundings of the vehicle from the camera 602 as moving images. As described above, the camera 602 is an omnidirectional camera and captures 360-degree omnidirectional images around the vehicle. The image output device 1 acquires an omnidirectional captured image, and also acquires direction information indicating the direction in which the image is captured by the camera 602 . For example, as shown in FIG. 18, when the camera 602 captures an image in the right direction of the vehicle, the captured image is acquired, and azimuth information indicating that the image is captured in the right direction is acquired. The image output device 1 similarly acquires captured images in other directions.

The image output device 1 recognizes animals existing around the vehicle from the acquired images. For example, as in the first embodiment, the image output device 1 holds a discrimination model 162 (a discriminator) in which image feature amounts of animals are learned in advance, and inputs a captured image to the discrimination model 162 to obtain a discrimination result of the animal. get. Thereby, the image output device 1 recognizes animals around the vehicle from the captured image.

The image output device 1 also recognizes the position of the animal relative to the vehicle based on the azimuth information acquired together with the captured image. In the case of the example of FIG. 18, the image output device 1 recognizes an animal from the captured image in the right direction, so it recognizes that the recognized animal is positioned in the right direction.

The image output device 1 determines the display position of the label image 41 based on the recognized position of the animal. For example, from the direction in which the animal is positioned, the image output apparatus 1 selects which transmissive display 604 out of a plurality of transmissive displays 604, 604, 604, . to decide. In the example of FIG. 18, the animal is located on the right side of the vehicle, so the image output device 1 determines that the label image 41 should be displayed on the transmissive display 604 provided on the right window of the vehicle.

Note that the image output device 1 determines the position of the animal in more detail, and displays the label image 41 in the vicinity of the animal for the observer as in the first embodiment. The display position of the image 41 may be controlled more finely.

Further, the image output device 1 may recognize whether or not animals form a herd (organism group), and may control to display the label image 41 for each herd, as in the first embodiment.

As described above, the image display system described in Embodiment 1 can be applied to present the label image 41 of the animal to the observer in a mobile object such as a vehicle.

FIG. 19 is a flow chart showing an example of a processing procedure executed by the image output device 1 according to the sixth embodiment.
The control unit 11 of the image output device 1 acquires a captured image from the camera 2 that captures the surroundings of the vehicle (step S601). For example, the camera 602 is an omnidirectional camera capable of capturing omnidirectional images of 360 degrees around the vehicle. The control unit 11 acquires from the camera 602 azimuth information indicating the imaging azimuth of the camera 602 together with the captured image.

The control unit 11 recognizes the animal from the acquired captured image (step S602). For example, the control unit 11 uses the identification model 162 to recognize an animal whose image feature amount has been learned from the captured image. The control unit 11 also recognizes the relative position (orientation) of the animal with respect to the vehicle by referring to the orientation information acquired together with the captured image.

The control unit 11 determines the display position of the label image 41 based on the position of the animal recognized from the captured image (step S603). For example, based on the relative position (orientation) of the animal with respect to the vehicle, the control unit 11 displays the label image on any of the transmissive displays 604, 604, 604, . Decide whether to display 41. The control unit 11 displays the label image 41 on the determined transmissive display 604 (step S604), and returns the process to step S601.

As described above, according to the sixth embodiment, the image display system described in the first embodiment can be applied to present the label image 41 of the animal to the observer in a mobile object such as a vehicle.

(Embodiment 7)
FIG. 20 is a functional block diagram showing the operation of the image output device 1 of the form described above. By executing the program P by the control unit 11, the image output device 1 operates as follows.
Acquisition unit 201 acquires captured images of a plurality of living things. The recognition unit 202 recognizes a group of organisms of the same kind from the captured image. A content acquisition unit 203 acquires a content image related to the organism group. A determination unit 204 determines the display position of the content image on a display surface provided between the plurality of living organisms and an observer based on the recognized position of the living organism group. The display unit 205 displays the content image on the display surface based on the determined display position.

The seventh embodiment is as described above, and other aspects are the same as those of the first to sixth embodiments, so corresponding parts are denoted by the same reference numerals, and detailed description thereof will be omitted.

The embodiments disclosed this time are illustrative in all respects and should not be considered restrictive. The scope of the present invention is indicated by the scope of the claims rather than the above-described meaning, and is intended to include all modifications within the scope and meaning equivalent to the scope of the claims.

1 PC (image output device, display device)
REFERENCE SIGNS LIST 11 control unit 12 main storage unit 13 communication unit 14 display unit 15 input unit 16 auxiliary storage unit P program 161 label DB
162 identification model 2, 602 camera 3 projector 4 screen 604 transmissive display

Claims (10)

an acquisition unit that acquires captured images obtained by imaging a plurality of living things;
a recognition unit that recognizes a group of organisms of the same type from the captured image;
a content acquisition unit that acquires a content image related to the organism group;
a determination unit that determines the display position of the content image on a display surface provided between the plurality of living organisms and an observer based on the recognized position of the organism group;
a display unit that displays the content image on the display surface based on the determined display position;
The captured image is a moving image,
The recognition unit
extracting feature points corresponding to the organism from each of the first and second images that are continuous in time series in the moving image;
detecting the movement of the creature based on the difference between the coordinate values of the feature points extracted from each of the first and second images;
A display device that recognizes the group of organisms based on the movement of each of the plurality of organisms. an estimating unit for estimating the moving direction and moving speed of the organism group based on the movement of the organism detected by the recognizing unit;
The display device according to claim 1, wherein the display unit moves and displays the display position based on the estimated moving direction and speed. The display surface is a transmissive screen provided on one surface of the space containing the plurality of living things,
3. The display device according to claim 1, wherein the display unit projects the content image onto the transmissive screen. The display surface is a transmissive display provided on one surface of the space containing the plurality of living things,
3. The display device according to claim 1, wherein the display unit displays the content image on the transmissive display. a determination unit that determines whether the display position of the content image overlaps with the recognition position of another organism different from the organism group corresponding to the content image,
5. The display device according to any one of claims 1 to 4, wherein the display unit changes the display position so that it does not overlap with the recognition position of the other creature. A determination unit that determines the scale of the organism group based on the recognition result of the recognition unit,
The display device according to any one of claims 1 to 5, wherein the display unit changes the display mode of the content image according to the scale of the organism group. The recognizing unit recognizes, from the captured image, the group of organisms and an observer observing the space containing the organisms from the outside,
The display device according to any one of claims 1 to 6, wherein the determination unit determines the display position based on the position of each of the organism group and the observer. Acquiring a captured image that is a moving image of a plurality of living things,
extracting feature points corresponding to the organism from each of first and second images that are chronologically continuous in the moving image, and difference in coordinate values of the feature points extracted from each of the first and second images; recognizing a group of organisms of the same type from the captured image by detecting the movement of the organism based on the movement of each of the plurality of organisms, and recognizing the organism group based on the movement of each of the organisms;
obtaining a content image of the organisms;
determining the display position of the content image on a display surface provided between the plurality of organisms and an observer based on the recognized position of the organism group;
A display method, comprising causing a computer to execute a process of displaying the content image on the display surface based on the determined display position. Acquiring a captured image that is a moving image of a plurality of living things,
extracting feature points corresponding to the organism from each of first and second images that are chronologically continuous in the moving image, and difference in coordinate values of the feature points extracted from each of the first and second images; recognizing a group of organisms of the same type from the captured image by detecting the movement of the organism based on the movement of each of the plurality of organisms, and recognizing the organism group based on the movement of each of the organisms;
obtaining a content image of the organisms;
determining the display position of the content image on a display surface provided between the plurality of organisms and an observer based on the recognized position of the organism group;
A program for causing a computer to execute a process of displaying the content image on the display surface based on the determined display position. a transmissive screen provided on a transmissive wall forming one side of a space containing a plurality of living things;
an imaging device that captures an image of the interior of the space;
a projection device installed outside the space for projecting an image onto the transmissive screen;
an image output device that outputs an image to the projection device,
The image output device is
an acquisition unit that acquires a captured image from the imaging device;
a recognition unit that recognizes a group of organisms of the same type from the captured image;
a content acquisition unit that acquires a content image related to the organism group;
a determination unit that determines a projection position of the content image on the transmissive screen based on the recognized position of the organism group;
a control unit that controls the projection device to project the content image onto the transmissive screen based on the determined projection position;
The captured image is a moving image,
The recognition unit
extracting feature points corresponding to the organism from each of the first and second images that are continuous in time series in the moving image;
detecting the movement of the creature based on the difference between the coordinate values of the feature points extracted from each of the first and second images;
An image projection system that recognizes the group of organisms based on the movements of each of the plurality of organisms.

Images