JP2020034609A - Display unit, display method, program, and image projection system - Google Patents

Abstract

To provide a display unit that can suitably present a content image related to creatures to an observer.SOLUTION: A display unit 1 comprises: an acquisition unit that acquires a picked-up image obtained by picking up images of a plurality of creatures; a recognition unit that recognizes a creature group consisting of the creatures of the same type from the picked-up image; a content acquisition unit that acquires a content image related to the creature group; a determination unit that determines, on a recognition position of the creature group, a display position of the content image on a display surface provided between the plurality of creatures and an observer; and a display unit that displays the content image on the display surface on the basis of the determined display position.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a display device, a display method, a program, and an image projection system.

  There has been proposed a technique of presenting an image related to an object behind a screen or a display by using a transmissive projector screen or a transmissive display in which the background can be seen through. For example, in Patent Literature 1, a projector image is provided inside an aquarium water tank, and a transmission screen is provided on a transmission wall of the water tank such as a glass window for an observer (viewer) to observe the inside of the water tank. A display device is disclosed. According to Patent Literature 1, an image of the inside of the aquarium is taken by a camera installed outside the aquarium, a fish swimming in the aquarium is recognized from the taken image, and an image such as a description of the fish is displayed at an appropriate position in the screen by the projector. indicate.

JP 2005-338115 A

  However, for example, when considering an aquarium tank, a plurality of fish are swimming in one tank. Therefore, in the case where a plurality of fish are swimming in a school, when images such as explanations are displayed for each individual fish, the images corresponding to each fish overlap and the fish of interest are given to the observer. There is a possibility that it cannot be seen.

  In one aspect, an object is to provide a display device or the like that can appropriately present a content image regarding a living thing to an observer.

  In one aspect, a display device includes: an acquisition unit configured to acquire a captured image of a plurality of living things; a recognition unit configured to recognize a group of living things including the same kind of living thing from the captured image; A content acquisition unit that acquires an image, and a determination unit that determines a display position of the content image on a display surface provided between the plurality of living things and an observer based on the recognition position of the group of living things, A display unit that displays the content image on the display surface based on the display position.

  In one aspect, a content image related to a living thing can be suitably presented to an observer.

It is a schematic diagram which shows the example of a structure of an image display system. FIG. 2 is a block diagram illustrating a configuration example of a PC. FIG. 9 is an explanatory diagram illustrating an example of a record layout of a label DB. FIG. 9 is an explanatory diagram illustrating a display example of a label image. FIG. 10 is an explanatory diagram related to a process of changing a display position of a label image. FIG. 10 is an explanatory diagram regarding a process of changing a display mode of a label image. 9 is a flowchart illustrating an example of a processing procedure executed by a PC. FIG. 11 is an explanatory diagram relating to a fish school recognition process according to the second embodiment. FIG. 14 is an explanatory diagram relating to a label tracking process according to the second embodiment. 13 is a flowchart illustrating an example of a processing procedure executed by the PC according to the second embodiment. FIG. 14 is an explanatory diagram relating to fish school recognition processing according to Embodiment 3. 15 is a flowchart illustrating an example of a processing procedure executed by a PC according to Embodiment 3. FIG. 14 is an explanatory diagram illustrating a configuration example of a screen according to a fourth embodiment. 15 is a flowchart illustrating an example of a processing procedure executed by a PC according to Embodiment 4. FIG. 19 is an explanatory diagram showing a display example of a label image according to Embodiment 5. 15 is a flowchart illustrating an example of a processing procedure executed by a PC according to Embodiment 5. FIG. 15 is a schematic diagram illustrating a configuration example of an image display system according to a sixth embodiment. FIG. 15 is an explanatory diagram illustrating an outline of a sixth embodiment. 15 is a flowchart illustrating an example of a processing procedure executed by the image output device according to the sixth embodiment. It is a functional block diagram showing operation of an image output device of the above-mentioned form.

Hereinafter, the present invention will be described in detail with reference to the drawings showing the embodiments.
(Embodiment 1)
FIG. 1 is a schematic diagram illustrating a configuration example of an image display system. In the present embodiment, a transmissive screen 4 (display surface) is provided on a transmissive wall such as a glass window forming one surface of an aquarium aquarium, and a label image (contents) for explaining fish in the aquarium is provided on the screen 4. An image display system (image projection system) for displaying an image) will be described. The image display system includes an image output device (display device) 1, a camera 2, and a projector 3. Each device is communicatively connected by a network such as a LAN (Local Area Network).

  The image output device 1 is an information processing device capable of performing various types of information processing and information transmission / reception, and is, for example, an information processing device such as a personal computer and a server device. In the present embodiment, it is assumed that the information processing device 1 is a personal computer, and is hereinafter replaced with PC1 for simplicity. The PC 1 stores label images for explaining various fish swimming in the aquarium as contents, and outputs the label images to the projector 3 and projects (displays) them on the screen 4. In the present embodiment, for the sake of convenience of explanation, even organisms other than fish, such as mammals, mollusks, and arthropods, are referred to as “fish” in the aquarium.

  In the present embodiment, the PC 1 acquires a captured image of the aquarium from the camera 2, recognizes various fish from the captured image, and performs display control so as to project a label image of the recognized fish to a location near the screen 4. In this case, the PC 1 determines whether the fish is swimming alone or in a school in the water tank. When recognizing a school of fish, that is, a school of fish, the PC 1 performs display control so as to project one label for the entire school of fish, instead of projecting a label image for each fish constituting the school of fish.

  The camera 2 is an imaging device that captures an image of the inside of an aquarium (space) containing a plurality of fishes through a transparent wall such as a glass window for an observer (viewer) to observe the inside of the aquarium, For example, it is installed on the ceiling of an aquarium. The camera 2 captures an image of the inside of the water tank through the transparent wall, and sequentially transmits the captured image to the PC 1 as a moving image.

  The projector 3 is a projection device that projects a label image. The projector 3 is installed at a position opposite to the water tank via the screen 4, that is, outside the water tank, and projects the label image on the screen 4 provided on the transmission wall. When a projector is installed inside a water tank and an image is projected from the inside of the water tank to a transmission screen as in the invention described in Patent Literature 1, there is a possibility that the label image may be shielded by fish swimming in the water tank. On the other hand, by installing the projector 3 outside the water tank instead of in the water tank as in the present embodiment, it is possible to prevent a situation in which the projection light from the projector 3 is blocked by fish in the water tank.

  The screen 4 is a transmission screen provided on a transmission wall of the water tank located between the fish in the water tank and the observer, and is a transparent screen film for diffusing irradiation light. For example, the screen 4 is attached to the entire surface of the transmission wall of the aquarium, and reflects an image light emitted (projected) from the front to form an image by reflecting the light forward.

  In the present embodiment, an explanation will be given assuming that an image is projected (displayed) using the projector 3 and the screen 4. Good. The same effect can be obtained even when the transmission type display is installed on the transmission wall of the water tank and the PC 1 displays an image on the transmission type display.

FIG. 2 is a block diagram illustrating a configuration example of the PC 1. The PC 1 includes a control unit 11, a main storage unit 12, a communication unit 13, a display unit 14, an input unit 15, and an auxiliary storage unit 16.
The control unit 11 has one or more arithmetic processing units such as a CPU (Central Processing Unit) and an MPU (Micro-Processing Unit), and reads and executes the program P stored in the auxiliary storage unit 16, It performs various information processing and control processing related to the PC 1. The main storage unit 12 is a temporary storage area such as a random access memory (RAM), and temporarily stores data necessary for the control unit 11 to execute arithmetic processing. The communication unit 13 includes a processing circuit and the like for performing processing related to communication, and transmits and receives information to and from the outside. The display unit 14 is a display device such as a liquid crystal display, and displays an image provided from the control unit 11. The input unit 15 is an operation interface such as a keyboard and a mouse, and receives an operation input.

  The auxiliary storage unit 16 is a nonvolatile storage area such as a large-capacity memory or a hard disk, and stores a program P and other data necessary for the control unit 11 to execute processing. The auxiliary storage unit 16 may be an external storage device connected to the PC 1. The auxiliary storage unit 16 stores a label DB 161 and an identification model 162. The label DB 161 is a database that stores data of label images that are contents to be projected on the screen 4. The identification model 162 is identification data for identifying a fish in an aquarium from an image captured by the camera 2, and is a learned model constructed by, for example, machine learning, as described later.

  FIG. 3 is an explanatory diagram illustrating an example of a record layout of the label DB 161. The label DB 161 includes an ID column, a type column, and a label image column. The ID column stores an ID for identifying each fish swimming in the aquarium. The type column stores the type of each fish in association with the ID. The label image sequence stores a label image for guiding each fish to the observer in association with the ID.

  FIG. 4 is an explanatory diagram illustrating a display example of the label image 41. FIG. 4 conceptually illustrates a state in which a fish swimming in an aquarium is recognized from an image captured by the camera 2 and a label image 41 relating to the recognized fish is displayed near a position where the fish is recognized. An outline of the present embodiment will be described with reference to FIG.

  First, the PC 1 sequentially acquires, as a moving image, images captured from the camera 2 through the transmission wall of the water tank provided with the screen 4. FIG. 4 schematically shows one frame of a moving image acquired by the PC 1. Here, as shown in FIG. 4, it is assumed that there are a fish forming a school of fish, such as "maji", and a fish swimming alone, such as "nanyoumanta", in the aquarium.

  The PC 1 recognizes each fish swimming in the aquarium from the captured image. Here, the PC 1 not only recognizes individual fish, but also identifies fish swimming alone and fish swimming in a fish school.

  For example, the PC 1 uses the identification model 162 set in the auxiliary storage unit 16 in advance to identify a fish swimming alone and a fish forming a school of fish and swimming. The discrimination model 162 is a learned model constructed by machine learning, and is a discriminator that has learned the image feature amount of a fish swimming alone and the image feature amount of the entire fish school including a plurality of fish of the same type. For example, the identification model 162 is a CNN (Convolution Neural Network) generated from teacher data, but the method of machine learning and the configuration of the classifier are not particularly limited. The identification model 162 may be generated according to another algorithm such as another neural network, an SVM (Support Vector Machine), a Bayesian network, or a decision tree. For example, the identification model 162 is generated based on teacher data that associates an image of a single fish or the entire fish school with a label value indicating the name of the fish (single or fish school).

  The PC 1 inputs the captured image acquired from the camera 2 to the identification model 162, and obtains, as an output value, an identification result of identifying an object included in the captured image. As a result, as indicated by the dotted rectangular frame in FIG. 4, the PC 1 recognizes the fish swimming in the aquarium after distinguishing a single fish or a school of fish.

  The PC 1 determines the display position (projection position) of the label image 41 to be displayed (projected) on the screen 4 corresponding to each fish based on the recognition position where each fish (single or school of fish) is recognized in the captured image. For example, as shown in FIG. 4, the PC 1 determines the display position immediately below the position where the fish is recognized so that the label image 41 is displayed near the fish.

  The PC 1 reads the label image 41 corresponding to the recognized fish from the label DB 161, outputs the label image 41 to the projector 3, and projects (displays) it on the screen 4 provided between the fish in the aquarium and the observer. Although the label image 41 is the name of a fish or the like, information other than the name may be included in the label image 41. Further, the label image 41 is not limited to text, but may be a picture image or the like.

  As described above, when the fish in the aquarium forms a school of fish and swims, the PC 1 does not display the label image 41 for each fish in the school of fish, but one label image 41 for the entire school of fish. Is displayed. Thereby, a large amount of label images 41 are not projected on the screen 4, and it is possible to avoid a situation where the label images 41 overlap and the fish cannot be seen.

FIG. 5 is a diagram illustrating a process of changing the display position of the label image 41. FIG. 5 illustrates how the display position of the label image 41 is changed so as not to be superimposed on another fish.
In the above, it has been described that the PC 1 displays the label image 41 at a position near the fish in the aquarium. Here, when the label image 41 is superimposed on another fish, the PC 1 changes the display position of the label image 41 so as not to be superimposed.

  In the example of FIG. 5, “whale shark” is included in the imaging range in addition to the various fish of FIG. 4. When the label image 41 is displayed in this state in the same manner as in FIG. 4, the label image 41 of “Maggie” on the upper left is displayed at the position of the thick dotted rectangular frame in FIG. 5 and partially overlaps with “Whale Shark”, It becomes difficult to see.

  After recognizing each fish from the captured image and determining the display position of the label image 41, the PC 1 compares the position of each fish with the display position of the label image 41, and superimposes the display position of the label image 41 on another fish. It is determined whether or not to perform. When determining that the label image 41 is superimposed, the PC 1 changes the display position so that the label image 41 is not superimposed. In the example of FIG. 5, the PC 1 changes the label image 41 of “Fuji” from the lower side to the upper side of the school of fish to avoid superimposition.

  In this way, by changing the display position of the label image 41, it is possible to avoid a situation where the label images 41 overlap and the fish cannot be seen.

  In the above description, the display position is changed according to whether or not the label image 41 is superimposed on the fish. However, the present embodiment is not limited to this. For example, when the position of the fish recognized from the captured image overlaps the edge (outer edge) of the screen 4 (display surface), the PC 1 may change the display position so that the label image 41 is moved to the center of the screen 4.

  For example, in FIG. 5, a fish school of “maji” is located at the right end. However, a part of the school of fish is out of the installation range of the screen 4, and the school of fish overlaps the edge of the screen 4. In such a case, when the entire school of fish including the portion outside the range of the screen 4 is recognized and the label image 41 is projected on the screen 4, the label image 41 may be projected out of the range of the screen 4. is there.

  Therefore, the PC 1 determines, for example, whether or not the recognition position of the fish (the school of fish) recognized from the captured image is superimposed on the edge (outer edge) of the screen 4. Is changed so that the display position of the label image 41 is closer to the center. Thereby, the label image 41 can be displayed at an appropriate position even for the fish located at the end of the screen 4.

FIG. 6 is a diagram illustrating a process of changing the display mode of the label image 41. FIG. 6 illustrates how the display mode of the label image 41 is changed according to the size of the fish school recognized from the captured image.
FIGS. 4 and 5 show the manner in which the label image 41 is displayed for any fish in the same manner. However, for example, if the aquarium is large and the school of fish swimming in the aquarium also becomes large, even if the label image 41 is displayed for the large fish school like other fish, There is a possibility that the observer may be hard to notice or difficult to see.

  Therefore, when recognizing the school of fish from the captured image, the PC 1 determines the size of the school of fish. For example, the PC 1 determines the scale according to the size of the recognition range of the fish school in the captured image (the dotted rectangular frame in FIG. 6). Alternatively, the PC 1 may determine the scale according to the number of fish forming the fish school.

  When it is determined that the fish school is a fish school of a predetermined size or more, the PC 1 changes the display mode of the label image 41 corresponding to the fish school. For example, the PC 1 displays the label image 41 of the fish school larger than the label image 41 of another fish (single or fish school). This allows the observer to easily recognize the label image 41 even for a large school of fish.

  In the above description, the label image 41 of a large school of fish is enlarged. However, the label image 41 of another fish may be reduced and the label image 41 of the school of fish may be relatively emphasized. Further, the change of the display mode is not limited to enlargement or reduction. For example, the PC 1 may display the same label image 41 at a plurality of locations for a large school of fish so that the observer can easily find any of the label images 41. Thus, the PC 1 only needs to be able to change the display mode of the label image 41 according to the size of the school of fish, and the display mode is not particularly limited.

FIG. 7 is a flowchart illustrating an example of a processing procedure executed by the PC 1. The processing executed by the PC 1 will be described with reference to FIG.
The control unit 11 of the PC 1 acquires, from the camera 2, an image of a plurality of fish (creatures) swimming in the water tank (moving space) (Step S11). The control unit 11 performs a process of recognizing a fish from the captured image based on the identification model 162 (Step S12). For example, as described above, the discrimination model 162 is a learned model constructed by machine learning, and is a discriminator that has learned image feature amounts of a single fish and an entire school of fish. The control unit 11 inputs the captured image obtained from the camera 2 to the discriminator, and obtains an identification result in which a single fish swimming alone in the aquarium and a group of fish swimming in a group in the aquarium are identified.

  The control unit 11 determines whether or not the school of fish has been recognized from the captured image as a result of the processing in step S12 (step S13). When it is determined that the school of fish has been recognized (S13: YES), the control unit 11 displays (projects) a label image related to the school of fish on the screen 4 based on the recognition position in the captured image that recognized the school of fish ( The projection position is determined (step S14).

  The control unit 11 determines whether or not the school of fish is a school of fish of a predetermined size or more based on the recognition result in step S12 (step S15). For example, the control unit 11 determines the scale according to the recognition range of the fish school in the captured image or the number of fish forming the fish school. When it is determined that the fish school is a fish school of a predetermined size or more (S15: YES), the control unit 11 changes the display mode of the label image of the fish school (step S16). For example, the control unit 11 performs enlarged display, multiple display, and the like of the label image.

  After executing the process of step S16, or in the case of NO in steps S13 and S15, the control unit 11 determines, with respect to another fish that has not been recognized as a fish school, that is, a fish swimming alone, in the captured image in which the fish is recognized. Based on the position, the display position at which the label image relating to the fish is displayed on the screen 4 is determined (step S17).

  The control unit 11 determines whether or not the display position of the label image determined in step S14 or S17 overlaps with another fish (single or fish school) (step S18). When it is determined that the label image is superimposed (S18: YES), the control unit 11 changes the display position so that the label image is not superimposed (step S19).

  After executing the process of step S19 or in the case of NO in step S18, the control unit 11 outputs the label image of the fish (single or fish school) recognized in step S12 to the projector 3, and determines the label image in steps S14, S17, and the like. The image is projected (displayed) on the display position (step S20). The control part 11 returns a process to step S11.

  As described above, according to the first embodiment, the fish school is recognized from the captured image, and the label image 41 is displayed for each fish school. This reduces the risk that the label image 41 is displayed for each individual fish forming the fish school and overlaps with the essential fish, and the label image 41 relating to living organisms can be suitably presented to the observer.

  Further, according to the first embodiment, the school of fish is recognized using the identification model 162 constructed by machine learning, and the label image 41 is displayed. By using the machine learning algorithm, the fish in the aquarium can be recognized with high accuracy, and the label image 41 can be presented more appropriately.

  According to the first embodiment, the transmission screen 4 is provided on the transmission wall (observation surface) of the water tank, and the label image 41 is projected (displayed) from the front of the water tank, that is, from outside the water tank. Therefore, unlike the video display device described in Patent Literature 1, there is little risk that the image is projected from the inside of the water tank and overlaps the fish, and the label image 41 can be more suitably presented.

  Further, according to the first embodiment, a configuration in which a transmissive display is provided on the transmissive wall of the aquarium in place of the projector 3 and the screen 4 may be employed. Also in this case, it is possible to preferably present the label image 41 while avoiding overlapping of the images.

  Further, according to the first embodiment, by changing the display mode of the label image 41 according to the size of the school of fish, it is possible to provide for the convenience of the observer.

  Further, according to the first embodiment, by changing the display position of the label image 41 so as not to be superimposed on another fish, it is possible to more effectively prevent the label image 41 from overlapping the fish and becoming invisible. You can do it.

(Embodiment 2)
In the first embodiment, a description has been given of a mode in which the characteristics of a school of fish are learned using a deep learning technique, and the school of fish is recognized using a trained model. In the present embodiment, a form in which the movement of each fish is specified by using a method such as an optical flow and a school of fish is recognized from the specified movement of each fish will be described. Note that the same components as those in the first embodiment are denoted by the same reference numerals, and description thereof is omitted.
FIG. 8 is an explanatory diagram relating to the school of fish recognition processing according to the second embodiment. FIG. 8A illustrates a state in which a feature point is extracted from a captured image, an optical flow is calculated, and a motion of an object (fish) moving in the aquarium is detected. FIG. 8B illustrates a state where a school of fish is recognized based on the movement of a plurality of objects. An outline of the present embodiment will be described with reference to FIG.

  As in the first embodiment, the PC 1 sequentially acquires captured images of the aquarium from the camera 2 as moving images. Here, the PC 1 specifies the movement of the object moving in the aquarium by comparing each image (frame) included in the moving image using, for example, an optical flow technique.

  Specifically, the PC 1 extracts feature points from each of two images (first and second images) that are continuous in time series in the moving image. The feature points are calculated from the brightness (luminance) of each pixel in the image. The PC 1 calculates a vector value representing the motion of the feature point, that is, an optical flow, based on the difference between the coordinate values of the feature point between the two images. FIG. 8A illustrates how the optical flow is calculated. In FIG. 8A, a white circle represents a feature point, and an arrow represents a vector corresponding to an optical flow.

  The PC 1 detects an individual object moving in the water tank, that is, a fish, from the optical flows corresponding to one or a plurality of feature points. For example, as shown in FIG. 8A, when optical flows related to a plurality of adjacent feature points indicate vector values of the same direction and the same speed, the PC 1 detects the plurality of feature points as one moving object (fish). . Note that the PC 1 may detect the moving object from the movement (optical flow) of one feature point.

  After detecting the movement of each object moving in the aquarium, the PC 1 compares the movements of a plurality of objects and detects (recognizes) an object forming a group and moving in the aquarium, that is, a school of fish. For example, as illustrated in FIG. 8B, when the optical flows of a plurality of objects located within a predetermined range indicate vector values in substantially the same direction and substantially the same speed, the PC 1 detects the plurality of objects as a school of fish.

  The PC 1 performs image recognition on one object (fish) constituting the detected school of fish, and specifies what kind of fish the school of fish is. The PC 1 similarly performs image recognition on other fish (single or fish school) and specifies what kind of fish it is. The PC 1 outputs the specified label image 41 of each fish to the projector 3 and projects the label image 41 at a position near each fish.

FIG. 9 is an explanatory diagram relating to the label tracking processing according to the second embodiment. FIG. 9 conceptually illustrates a tracking display in which the label image 41 is moved in accordance with the detected movement of the school of fish.
As described above, the PC 1 detects the movement of the fish (single or fish school) in the aquarium. The PC 1 estimates where to move next from the detected movement of the fish, and performs tracking display for moving the label image 41 to the estimated position. The PC 1 estimates the moving direction and the moving speed from the optical flow indicating the movement of the fish, and controls the projector 3 to move the display position of the label image 41 in accordance with the estimated direction and speed. As described above, not only the fish school is recognized by detecting the movement of the fish, but also the display control can be performed to move the label image 41 to a suitable position.

FIG. 10 is a flowchart illustrating an example of a processing procedure executed by the PC 1 according to the second embodiment.
After acquiring the captured image from the camera 2 (step S11), the control unit 11 of the PC 1 executes the following processing. The control unit 11 extracts a feature point included in the acquired captured image (Step S201). For example, the control unit 11 extracts a feature point according to the brightness of each pixel included in the imaging pixel.

  The control unit 11 controls the inside of the water tank based on the difference between the coordinate values of the feature points extracted from each of the two images (first and second images) that are successive in time series among the images sequentially acquired as a moving image. The movement of a moving object (fish) is detected (step S202). For example, the control unit 11 calculates an optical flow for each feature point, and detects the motion of each moving object based on the optical flows of one or more adjacent feature points.

  The control unit 11 determines whether a plurality of objects are moving in a group based on the detected movement of each object (step S203). If it is determined that the group is moving (S203: YES), the control unit 11 performs image recognition on any of the objects constituting the group and specifies the type of fish (step S204). The control unit 11 shifts the processing to step S14.

  After executing the process in step S16, or in the case of NO in steps S203 and S15, the control unit 11 performs image recognition on an object moving alone, and specifies the type of fish (step S205). The control unit 11 shifts the processing to step S17.

  After executing the processing in step S19 or in the case of NO in step S18, the control unit 11 outputs the label image 41 to the projector 3 and projects (displays) it (step S206). In this case, the control unit 11 estimates the moving direction and the moving speed of the fish (single or fish school) corresponding to the label image 41 based on the movement of the fish detected in step S202. The control unit 11 performs tracking display in which the display position of the label image 41 is moved according to the estimated moving direction and moving speed. Control unit 11 returns the process to step S11.

  As described above, according to the second embodiment, the movement of fish is detected by a technique such as optical flow, and a school of fish is detected from the detected movement of each fish. Thereby, even if the user does not have the learned model, the fish school can be recognized and the label image 41 can be presented.

  Further, according to the second embodiment, it is possible to estimate where to move from the detected movement of the fish, and perform tracking display of the label image 41. Thereby, the displacement between the fish and the label image 41 can be avoided, and the label image 41 can be suitably presented.

(Embodiment 3)
In the present embodiment, a description will be given of a mode in which all fish are recognized from a captured image and a school of fish is identified from the positional relationship between the recognized fish.
FIG. 11 is an explanatory diagram relating to a fish school recognition process according to the third embodiment. In the second embodiment, the form in which the individual fish is specified after the detection of the group of fish has been described. However, in the present embodiment, the individual fish is specified first, and the group of fish is recognized from the positional relationship of the specified fish.

  Specifically, the PC 1 performs image recognition on the captured image, and recognizes all fish swimming in the aquarium. In the example of FIG. 11, the PC 1 recognizes each individual as a “bluefin tuna” not only for “blue dolphins” and “nanyoumanta” but also for four “bluefin tuna” forming a fish school.

  Thereafter, the PC 1 recognizes a fish school formed by a plurality of fish of the same type from the types and positional relationships of the plurality of fish. For example, the PC 1 recognizes a school of fish depending on whether or not fish (individuals) of the same type and a predetermined number or more are located within a predetermined range. In the example of FIG. 11, a plurality of fishes of the type “BFT” are common in the upper image area of the captured image. When the plurality of fishes are located within the predetermined range, the PC 1 recognizes the plurality of fishes as a “bluefin tuna” fish school.

FIG. 12 is a flowchart illustrating an example of a processing procedure executed by the PC 1 according to the third embodiment.
After acquiring the captured image from the camera 2 (step S11), the control unit 11 of the PC 1 executes the following processing. The control unit 11 performs image recognition on the captured image, and recognizes all of the plurality of fish in the water tank (step S301). The control unit 11 determines whether or not a plurality of fish form a fish school based on the recognized positional relationship of each fish (Step S302). For example, as described above, the control unit 11 determines whether or not a fish school is formed by determining whether or not fish of the same type and a predetermined number or more are located within a predetermined range. When it is determined that a school of fish is formed (S302: YES), the control unit 11 shifts the processing to step S14. When it is determined that no fish school is formed (S302: NO), the control unit 11 shifts the processing to step S17.

  As described above, according to the third embodiment, all the fish are recognized from the captured image, the fish school is identified based on the type, number, and positional relationship of the recognized fish, and the label image 41 is displayed. Thereby, the label image 41 can be suitably presented to the observer.

(Embodiment 4)
In the present embodiment, an embodiment will be described in which the screen 4 is provided only in a part of the transmission wall of the aquarium, and the label image 41 is projected only when the fish passes through the area where the screen 4 is provided.
FIG. 13 is an explanatory diagram illustrating a configuration example of the screen 4 according to the fourth embodiment. In the present embodiment, the screen 4 is provided not in the entire surface of the transmission wall of the water tank, but in only a partial area. For example, as shown in FIG. 13, the screen 4 is provided on the left half surface of the transmission wall when viewed from the front.

  The range in which the screen 4 is provided is not limited to the example of FIG. 13, and may be changed to, for example, a right half surface, an upper half surface, or the like. Further, the ratio of the area of the screen 4 to the entire surface of the transmission wall is not particularly limited.

  The PC 1 projects (displays) the label image 41 only when the fish passes through the area where the screen 4 is provided in front view. For example, the PC 11 recognizes all fish (single or fish school) in the aquarium from the captured image as shown in FIG. Then, the PC 11 specifies a fish passing through the installation area of the screen 4. For example, when the label image 41 corresponding to each fish is projected (displayed), the PC 11 determines whether or not the display position of the label image 41 is within the installation area of the screen 4. When determining that the label image 41 is within the installation area of the screen 4, the PC 1 controls the projector 3 to project the label image 41 on the screen 4.

FIG. 14 is a flowchart illustrating an example of a processing procedure executed by the PC 1 according to the fourth embodiment.
After executing the processing in step S19 or in the case of NO in step S18, the control unit 11 of the PC 1 executes the following processing. The control unit 11 determines whether or not the display position of the label image 41 determined based on the recognition result of the captured image is in an area where the screen 4 is provided on the transmission wall of the aquarium (step S401). When it is determined that it is within the area where the screen 4 is provided (S401: YES), the control unit 11 projects (displays) the label image 41 on the screen 4 (Step S402). After executing the process of step S402, or in the case of NO in step S401, the control unit 11 returns the process to step S11.

  In the above description, the fish passing through the area of the screen 4 is identified after all the fish in the aquarium are recognized, but the present embodiment is not limited to this. For example, the camera 2 may capture an image of only the area where the screen 4 is provided, and the PC 1 may recognize a fish from an image having a reduced imaging range, thereby identifying a fish passing through the area of the screen 4. That is, the PC 1 only needs to be able to recognize the fish from the image area of the captured image corresponding to a partial area of the transmission wall provided with the screen 4 and project the label image 41 on the screen 4. It is not indispensable to take an image of and recognize all fish.

  As described above, according to the fourth embodiment, the display area of the label image 41 can be narrowed down to a partial area of the transmission wall of the aquarium, and a flexible configuration can be provided in accordance with actual operation.

(Embodiment 5)
In the present embodiment, a mode in which the display position of the label image 41 is determined according to the position of the observer will be described.
FIG. 15 is an explanatory diagram illustrating a display example of the label image 41 according to the fifth embodiment. FIG. 15 conceptually illustrates how the display position of the label image 41 is controlled so as to approach the position where the observer is.

  In the present embodiment, the PC 1 recognizes not only fish in the aquarium but also an observer who observes the aquarium from the captured image. For example, the camera 2 sets the inside of the aquarium and a predetermined range in front of the aquarium where the observer is located as an imaging range, and the PC 1 determines an observer around the aquarium from an image obtained by the camera 2 capturing the imaging range. recognize. Alternatively, the camera 2 may be configured as an omnidirectional camera, and may image the entire room of the aquarium in which the aquarium is installed to recognize an observer in front of the aquarium. Alternatively, the camera 2 for imaging the inside of the water tank and the camera 2 for imaging the front of the water tank may be separately provided, and the observer may be recognized from the image captured by the latter camera 2.

  The PC 1 controls the display position of the label image 41 according to the position of the observer recognized from the captured image. In the example of FIG. 15, the observer is located substantially at the center in the horizontal direction of the water tank. In this case, the PC 1 controls the display position such that the label image 41 is located closer to the center as compared with the case where no observer is present. In the example of FIG. 15, the PC 1 shifts the label images 41 of “bluefin tuna”, “nanyoumanta”, and “maji” located on the right side to the left, and the label images 41 of “maji” and “madara dolphin” located on the left side Shifts to the right. In this way, the PC 1 controls the display position so that the label image 41 approaches the position of the observer.

  By the above-described processing, the label image 41 can be shifted to a position that is easy for the observer to see, and the label image 41 can be more appropriately presented.

FIG. 16 is a flowchart illustrating an example of a processing procedure executed by the PC 1 according to the fifth embodiment.
After acquiring the captured image from the camera 2 (step S11), the control unit 11 of the PC 1 executes the following processing. The control unit 11 recognizes a fish (single or a school of fish) from the captured image and also recognizes an observer observing the inside of the aquarium (step S501). The control unit 11 shifts the processing to step S13.

  When it is determined that the fish school is recognized from the captured image (S13: YES), the control unit 11 displays the label image 41 corresponding to the fish school based on the fish school position and the observer position recognized from the captured image. Is determined (step S502). For example, as described in FIG. 15, the control unit 11 determines the display position of the label image 41 from the display position when the observer is not recognized so that the label image 41 approaches the observer's position. The control unit 11 shifts the processing to step S15.

  After executing the processing in step S16, or in the case of NO in steps S13 and S15, the control unit 11 determines the label image 41 of the fish swimming alone based on the position of the single fish recognized from the captured image and the position of the observer. Is determined (step S503). Specifically, similarly to step S502, the control unit 11 determines the display position so that the label image 41 approaches the position of the observer. The control unit 11 shifts the processing to step S18.

  As described above, according to the fifth embodiment, the display position of the label image 41 can be determined in consideration of the position of the observer, and the label image 41 can be more appropriately presented.

(Embodiment 6)
In the first embodiment, the screen 4 is provided on the transmission wall of the aquarium aquarium, and the label image 41 relating to the fish swimming in the aquarium is projected (displayed) on the screen 4. In the present embodiment, when observing an animal (living organism) from inside a vehicle like a safari park, a mode in which the window of the vehicle is configured as a transmissive display 604 and the label image 41 is displayed on the window of the vehicle will be described.
FIG. 17 is a schematic diagram illustrating a configuration example of an image display system according to Embodiment 6. The image display system according to the present embodiment includes an image output device 1, a camera 602, and a transmissive display 604. The image output device 1 may be configured as an in-vehicle device such as an ECU (Electronic Control Unit) in addition to the configuration of a personal computer as in the first embodiment, for example.

  The camera 602 is an imaging device that captures an image around the vehicle, and is, for example, an omnidirectional camera. When the traveling direction of the vehicle is forward, the camera 602 captures 360-degree omnidirectional images extending in the front, rear, left, and right directions. Note that the camera 602 may not be an omnidirectional camera, but may be, for example, provided with cameras 602 at the front, rear, and right and left sides of the vehicle, and a plurality of cameras 602 may capture images in all directions. Further, in this embodiment, since the transmissive display 604 is provided only in the window on the side of the vehicle, the camera 602 may be configured not to capture images in all directions but to capture images only in the left-right direction of the vehicle.

  The transmissive display 604 is a display device such as an organic EL (Electro Luminescence) display or a liquid crystal display, and is a display device in which the area other than the display object such as the label image 41 is transparent. The transmissive display 604 is provided, for example, in a window on the side of the vehicle shown in FIG. 17 and displays an image output from the image output device 1. Instead of the transmissive display 604, an image may be displayed on a window on the side of the vehicle by a projector method using the transmissive screen 4 as in the first embodiment. The transmissive display 604 is provided in each of a plurality of windows on the left and right sides of the vehicle, and displays an image for an observer (e.g., a park attendant at a safari park) riding the vehicle.

  Note that the transmissive display 604 may be provided not only at the window on the side of the vehicle but also at the front and rear windows of the vehicle.

FIG. 18 is an explanatory diagram showing an outline of the sixth embodiment. FIG. 18 illustrates a state in which the camera 602 captures an image of an animal existing around the vehicle. An outline of the present embodiment will be described with reference to FIG.
The image output device 1 sequentially acquires, from the camera 602, captured images of the surroundings of the vehicle as moving images. As described above, the camera 602 is an omnidirectional camera, and captures images in 360 degrees around the vehicle. The image output device 1 acquires an omnidirectional captured image and also acquires azimuth information indicating the imaging azimuth of the image captured by the camera 602. For example, as illustrated in FIG. 18, when the camera 602 captures an image in the right direction of the vehicle, the camera 602 acquires the captured image and azimuth information indicating that the image is an image captured in the right direction. Similarly, the image output device 1 acquires a captured image for other directions.

  The image output device 1 recognizes an animal existing around the vehicle from the acquired image. For example, as in the first embodiment, the image output device 1 holds an identification model 162 (identifier) in which the image feature amount of the animal has been learned in advance, and inputs the captured image to the identification model 162 to determine the animal identification result. get. Thus, the image output device 1 recognizes animals around the vehicle from the captured image.

  Further, the image output device 1 recognizes the position of the animal relative to the vehicle based on the azimuth information acquired together with the captured image. In the case of the example of FIG. 18, the image output device 1 recognizes the animal from the captured image in the right direction, and thus recognizes that the recognized animal is located in the right direction.

  The image output device 1 determines the display position of the label image 41 based on the recognized position of the animal. For example, the image output device 1 displays which transmission display 604 among a plurality of transmission displays 604, 604, 604... Provided in each window on the left and right sides of the vehicle from the direction in which the animal is located. To determine. In the example of FIG. 18, since the animal is located to the right of the vehicle, the image output device 1 determines that the label image 41 should be displayed on the transmissive display 604 provided in the right window of the vehicle.

  In addition, the image output device 1 determines the position of the animal in more detail, and similarly to the first embodiment, the label in one transmissive display 604 so that the label image 41 is displayed near the animal for the observer. The display position of the image 41 may be controlled more finely.

  Further, the image output device 1 may recognize whether or not the animals form a group (organism group) and control to display the label image 41 for each group, as in the first embodiment.

  As described above, the image display system described in the first embodiment can be applied to present a label image 41 of an animal to an observer on a moving object such as a vehicle.

FIG. 19 is a flowchart illustrating an example of a processing procedure executed by the image output device 1 according to the sixth embodiment.
The control unit 11 of the image output device 1 acquires a captured image from the camera 2 that captures an image around the vehicle (step S601). For example, the camera 602 is an omnidirectional camera capable of capturing 360 degrees around the vehicle. The control unit 11 acquires, from the camera 602, azimuth information indicating the imaging azimuth of the camera 602, together with the captured image.

  The control unit 11 recognizes an animal from the acquired captured image (Step S602). For example, the control unit 11 uses the identification model 162 to recognize an animal whose image feature has been learned from the captured image. The control unit 11 refers to the orientation information acquired together with the captured image and also recognizes the relative position (azimuth) of the animal with respect to the vehicle.

  The control unit 11 determines the display position of the label image 41 based on the position of the animal recognized from the captured image (Step S603). For example, based on the relative position (orientation) of the animal with respect to the vehicle, the control unit 11 transmits a label image to any of the transmissive displays 604 provided on the window on the side of the vehicle. 41 is displayed. The control unit 11 displays the label image 41 on the determined transmissive display 604 (step S604), and returns the process to step S601.

  As described above, according to the sixth embodiment, the image display system described in the first embodiment can be applied to present a label image 41 of an animal to an observer on a moving object such as a vehicle.

(Embodiment 7)
FIG. 20 is a functional block diagram showing the operation of the image output device 1 of the above-described embodiment. When the control unit 11 executes the program P, the image output device 1 operates as follows.
The acquisition unit 201 acquires a captured image of a plurality of living things. The recognition unit 202 recognizes, from the captured image, a group of living things including the same kind of living thing. The content acquisition unit 203 acquires a content image related to the creature group. The determination unit 204 determines a display position of the content image on a display surface provided between the plurality of living things and an observer based on the recognition position of the living thing group. The display unit 205 displays the content image on the display surface based on the determined display position.

  The seventh embodiment is as described above, and the rest is the same as the first to sixth embodiments. Corresponding parts are allotted with the same reference numerals, and their detailed description is omitted.

  The embodiment disclosed this time is an example in all respects and should be considered as not being restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

1 PC (image output device, display device)
Reference Signs List 11 control unit 12 main storage unit 13 communication unit 14 display unit 15 input unit 16 auxiliary storage unit P program 161 label DB
162 Identification model 2, 602 Camera 3 Projector 4 Screen 604 Transmissive display

Claims (14)

An acquisition unit that acquires a captured image of a plurality of living things,
From the captured image, a recognition unit that recognizes a group of living things composed of the same kind of living things,
A content acquisition unit that acquires a content image related to the creatures,
A determining unit that determines a display position of the content image on a display surface provided between the plurality of creatures and an observer, based on the recognition position of the creature group,
A display unit for displaying the content image on the display surface based on the determined display position. The display device according to claim 1, wherein the recognizing unit recognizes the living organisms based on a learned model in which an image feature amount of the living organisms has been learned. The captured image is a moving image,
The recognition unit,
Extracting a feature point corresponding to the living thing from each of the first and second images successive in time series in the moving image;
Based on the difference between the coordinate values of the feature points extracted from each of the first and second images, the movement of the living thing is detected,
The display device according to claim 1, wherein the group of living things is recognized based on a movement of each of the plurality of living things. An estimating unit that estimates a moving direction and a moving speed of the creature group based on the movement of the creature detected by the recognition unit,
The display device according to claim 3, wherein the display unit moves and displays the display position based on the estimated moving direction and speed. The recognition unit,
Recognizing each of the plurality of living things included in the captured image,
The display device according to claim 1, wherein when the plurality of living things of the same type are located within a predetermined range in the captured image, the living thing is recognized as the group of living things. The display surface is a transmission screen provided on one surface of a space containing the plurality of living things,
The display device according to claim 1, wherein the display unit projects the content image on the transmissive screen. The display surface is a transmission type display provided on one surface of the space containing the plurality of living things,
The display device according to claim 1, wherein the display unit displays the content image on the transmissive display. The display surface is provided in a partial region of one surface of the space containing the plurality of living things,
The said display part displays the said content image on the said display surface, when the said creature group is recognized from the image area of the said picked-up image corresponding to the said partial area | region. 2. The display device according to claim 1. A determination unit that determines whether the display position of the content image is superimposed on a recognition position of another living thing different from the living thing group corresponding to the content image,
The display device according to any one of claims 1 to 8, wherein the display unit changes the display position so that the display position does not overlap with the recognition position of the other living thing. Based on the recognition result by the recognition unit, comprising a determination unit that determines the size of the organism group,
The display device according to any one of claims 1 to 9, wherein the display unit changes a display mode of the content image according to a scale of the creature. From the captured image, the recognition unit recognizes the group of living things and an observer who observes the space containing the living things from outside,
The display device according to any one of claims 1 to 10, wherein the determination unit determines the display position based on the positions of the living organisms and the observer. Obtain captured images of multiple organisms,
From the captured image, recognize a group of organisms composed of the same kind of organism,
Obtaining a content image related to the creatures,
Based on the recognition positions of the creatures, determine the display position of the content image on a display surface provided between the plurality of creatures and the observer,
A display method, comprising: causing a computer to execute a process of displaying the content image on the display surface based on the determined display position. Obtain captured images of multiple organisms,
From the captured image, recognize a group of organisms composed of the same kind of organism,
Obtaining a content image related to the creatures,
Based on the recognition positions of the creatures, determine the display position of the content image on a display surface provided between the plurality of creatures and the observer,
A program for causing a computer to execute a process of displaying the content image on the display surface based on the determined display position. A transmissive screen provided on a transmissive wall forming one surface of a space accommodating a plurality of living things,
An imaging device for imaging the inside of the space;
A projection device that is installed outside the space and projects an image on the transmission screen,
An image output device that outputs an image to the projection device,
The image output device,
An acquisition unit that acquires a captured image from the imaging device;
A recognition unit that recognizes the living thing from the captured image,
A content acquisition unit that acquires a content image related to the living thing,
Based on the recognition position of the living thing, a determination unit that determines the projection position of the content image on the transmission screen,
A control unit that controls the projection device so as to project the content image on the transmission screen based on the determined projection position.

Images