JPH09293133A - Abstract medium supplementing method and its system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reproduce the state of listening to language, voice, sound, etc., to feel as if being at a site (virtual space) by selecting concrete shortage information corresponding to an abstract medium based on the instruction of a user so as to supplement by means corresponding to human five senses. SOLUTION: Concrete shortage information corresponding to the abstract medium expressed by language, voice, sound or characters is stored by the class to select shortage information by the sort from the information by user's instruction to successively or simultaneously execute supplement. This concrete information is derived from five senses of sight (picture), hearing (voice, sound), smelling (scent, etc.), taste (taste), touch (warm/cold window), etc. Namely this system creates a landscape picture from a landscape describing sentence such as being provided with the data base of the parts of the picture constituting landscapes such as 'tree', 'mountain' 1, 'sea', 'sky' 18, etc., to retrieve the picture part group to arrange in the picture to create a landscape picture based on the input of the user.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an abstract medium intended for supplementing concrete lack information such as an image, a sound image, a fragrance, etc. with an abstract medium given by language, voice, sound or characters. The present invention relates to a supplement method and supplement system for media.

[0002]

2. Description of the Related Art Regarding complementing conventional abstract media,
The simultaneous display of characters and images in karaoke etc. is known.

The inventors have also previously proposed a method of complementing missing information in a video (Japanese Patent Application No. 7-110973).
issue).

[0004]

The image displayed in the above-mentioned conventional karaoke is only automatically given the image associated with the lyrics. Therefore, there is a problem that the lyrics are exclusive and cannot match the user's sensitivity.

Further, it is relatively easy to complement missing information for an image and video, but a complementary system is lacking for complementing abstract media such as characters, languages, voices or sounds, which are overwhelmingly generated in a large amount. However, there were problems such as not only doing it, but also making no proposal about this.

[0006]

SUMMARY OF THE INVENTION Therefore, the present invention is based on a character, a language, a voice or a sound that describes a landscape or any other object.
The words related to the composition of landscape and other materialized information are extracted, and the words are related to the names of objects such as mountains, rivers, birds, and trees (hereinafter referred to as objects), the season / time / place / state, etc. Object conditions), images that the user wants to get out, and things related to mood (hereinafter referred to as affective words) are categorized, and objects that are categorized into each category are organized by series and accumulated a large amount of data (stored data). By arbitrarily extracting and using these data, the above-mentioned conventional problems have been successfully solved. The term "series-based" refers to differences in properties such as winter mountains, snow mountains, summer mountains, and spring mountains in mountains, and streams, rivers, rapids, and the like in rivers.

That is, the invention of the method is a method of complementing an abstract medium, which is characterized in that concrete lack information corresponding to the abstract medium is selected and displayed based on a user's instruction, and corresponds to the abstract medium. This is an abstract media complementing method characterized by selecting concrete lack information based on a user's instruction and complementing it by means corresponding to the human five senses.

In the system invention, concrete lack information corresponding to the abstract medium is stored according to the information classification, and the lack information is selected according to the classification from the information according to a user's instruction, and sequentially. Alternatively, it is an abstract media complementing system characterized by complementing at the same time, and concrete lack information corresponding to the abstract media is stored by classification and by sequence, and classified by user and by sequence. It is an abstract media complementing system characterized by automatically or selectively complementing lacking information.

As described above, when the present invention is provided with media represented by abstract depictions such as language, voice, and sound, the present invention complements the specific information that is missing. Then, as an example, a case where natural language information is replaced by information such as a landscape image will be described.

The above-mentioned language is created by an individual using each language and expressed by patterns such as letters, voices, sounds, fingers and lips.

Next, the image is the sensor output signal 2
It is a visualization of the dimensional distribution. That is, since the image information and the video information are input via the physical sensor, the objectivity is strong and the individuality is strong. This is because the physical sensor output is objective, and the constantly changing target world is captured as it is. On the other hand, from the above definition,
For example, a linguistic expression depicting a scene can be said to be a description of the scene due to the interest of an individual.

The image is divided into three types: a document image, an image in the document, and a real world image. Among them, the mutual conversion between the real world image and the language is as follows.

That is, a language is generally generated by a human being and also an individual, and the efficiency of expression is poor due to its peculiar expression form, and when describing the target world, only a very limited part can be described. Moreover, even if the intention is to avoid the subjective judgment, there is a high possibility that an expression peculiar to the creator depending on the preference of the sentence creator is mixed. In addition, the symbolic representation means conceptualization, which means that individual objects are represented by a general category that abandons individuality.

What is meant by these things is that the recognition / understanding process limits the interpretation of an image and, for the first time, it has a content equivalent to a language expression. The concept of responding to individual objects in an image by recognition is
A two-to-many one-to-one mapping is applied in which a part of the mapping is selected according to the interest. When the image of the linguistic expression is applied twice, the reverse one-to-many mapping is unavoidable. That is, it is possible to reproduce the world in a standard environment only after individualizing the generalized object, restoring the lost background using common sense, and arranging it in common sense.

Next, in the recording of the sketch, a big difference from the recording by the word is that the positions of the objects on the screen and their relative positional relationship can be reproduced "unintentionally". In words, it is necessary to clearly indicate what is on the screen and where it is, but in a sketch, the position is also indicated by drawing on the screen.

A big difference between recording words and sketches is that words can record the entire impression. Generally, the impression that a picture has is considered to be given by factors such as color, composition, touch, theme, and the object being drawn, and is expressed using impression words such as "warm" and "vivid."

As described above, it can be seen that the amount of information that can be expressed is different between language media and images. For example, when comparing "illustration" in a novel with "landscape description", the difference in the amount of information between them is obvious. Usually, scenes are portrayed such as "The ship returned to the harbor at dusk whistling like a lonely whistle." Or "When I passed through the forest, a majestic waterfall appeared in front of me." As can be seen from these example sentences, the "landscape depiction sentence" only represents a characteristic part of the landscape.

However, in a language medium, in order to supplement a small amount of information, the whole impression can be recorded using sensitivity words. Humans get various impressions from an image, and it is thought that this gives an impression of the entire image by complicatedly entangled with various elements such as the color of the image, composition, and objects in the image. In words, the impression that this human gets from the image,
It is possible to express it with only one word, a sensitivity word. To put it in an extreme way, it can be said that the emotional words in the sentence contain a huge amount of information comparable to images. In other words, sensitivity words play a very important role in the words that compose sentences. Therefore, as described above, when complementing a landscape image from a landscape description,
It turns out that we have to consider human sensitivity.

Humans receive various impressions from various landscape images. For example, if you look at a "picture of a cow on a ranch,"you'll feel "noisy." If you look at a "picture of a large view of Mt. Fuji in the middle,"you'll feel "dignified" or "heavy." . Also, even if you take a picture from the same place, the impression you get from the picture changes with the season and time.

As described above, it is considered that the landscape images give different impressions to humans because there are some elements in the landscape image that provoke human sensitivity.

In the present invention, the elements that are considered to affect the sensitivity in the landscape image are divided into the following three elements. (1) Property of object (2) Color (3) Composition

The property of the object is, for example, "it feels calm when there is a cow" or "the grave is scary", and is mainly based on human knowledge and experience.

This kind of sensitivity based on human knowledge or experience is something that is generally known to everyone, and when sensitivity is taken into consideration, the 雖 is similar to the keyword in the keyword method. It is a thing. For example, even in an image search system that is not particularly conscious of sensibility search, it is possible to use sensibility words based on such human knowledge and experience as keywords.

Next, it is well known that color influences human sensibilities. For example, he said, "People wearing white clothes feel neat" and "red is passionate", and relatively many studies have been conducted to use them for image retrieval. Some studies have obtained good results by using general parameters such as hue, saturation, and lightness for general paintings, and establishing a rule that describes the relationship between those parameters and impression words. In addition, as a study using natural language, there are cases where the color of flowers or fruits is used as a key to a search in a botanical pictorial book.

In addition to the above, in the search of design images, attention is paid to the number of colors, color classification such as warm color / cold color, and contrast. There are other studies that try to use colors in the search for design drawings. Next, according to some research, the relationship between 130 colors in the Munsell symbol space and 15 sensibility words was obtained by conducting a questionnaire by a design expert and used for the search of design drawings.

In these studies, a wide variety of images such as portraits, landscapes, and design images are searched as search targets. Also, other than image search,
There are also examples of applying the relationship between color and sensitivity to coloring landscape paintings. In this research, we use fuzzy theory to create uncolored landscape paintings.
I am trying to color it according to the image of the human season and time.

Finally, as for the composition, even in the picture of the same mountain,
There is a sense of stability and weight when a triangular mountain like Mt. Fuji is in the middle, and a dynamic feeling when there are many steep, steep mountains. It also gives a different impression, such as a gentler feeling in a more gentle mountain.

As described above, it has been known for a long time that composition influences sensibility, and many studies have been conducted in the field of art such as landscape painting and landscape photography. However, little research has been done to use this relationship between composition and sensitivity in fields such as "image retrieval" and "image creation." In the research of arranging parts images according to the composition rules of Chinese flower and bird painting, the composition knowledge is modeled, and the same parts are arranged using different compositions.

In another study, attention is paid also to the pattern shape of the design, and the relationship between the impression word and the pattern shape is performed. The study states that curved designs give the impression "warm" and "soft", while linear designs give the impression "cold" and "hard".

Other studies have been conducted on a chair database to utilize the relationship between chair shape and sensitivity for retrieval. In this research, a factor analysis using the SD method is performed on the chair shape, and the factors affecting the kansei by the chair shape are considered.

As described above, the research focusing on the composition is still small compared to the color, and the target image is also limited. As a practical matter, no studies have been conducted yet focusing on the relationship between landscape composition and sensitivity.

However, as described above, the composition of a picture is an important factor that influences human sensitivity.
It is necessary to use it for things such as "image search" and "image creation".

All objects existing in this world consist of lines. Research on the important lines that form the basis of this composition principle has been done for a long time both in East and West. For example,
The German painter Albrecht Dürer classifies the elements of lines related to the human body into 13 lines of 6 types in his book "Equity of the Human Body".

Although there are various theories as described above, it is a common theory among contemporary artists to classify into 3 types and 7 lines as shown in FIG.

(1) Form mainly based on the characteristics of the line (a) Straight line (b) Curve (2) Form mainly regarding the amount, length, and size of the line (a) Long line, thick line (b) Short line , Thin line (3) Line-oriented format (a) Horizontal line (b) Vertical line (c) Diagonal line

Next, the format mainly based on the property of the line will be described.

The first character of the straight line is a masculine feeling that the person is tense and the force of going straight, that is, one point is a certain direction in full speed and total tension without being obstructed by anything and traveling the shortest distance. It represents power and its unique simplicity. Also, in contrast to the curve described below, there is an image that a straight line is stiff. In addition to the first character, another property is newly added depending on the "quantity" and the "direction" of the property.

[0038] Also, the word "personality has come out"
As you can see, "a person who is horny and hard", the curves are gentle, and it is easy to express a warm roundness. You can express a relaxed, peaceful landscape and rich gentleness. Then, as in the case of the straight line, another property is newly added depending on the "quantity" and the "direction".

As described above, there are (a) long line, thick line, (b) short line, and thin line as the elements showing the property of the line, which are described in (3) (a) and (b) above. The meaning becomes clear by combining with any of (c). In other words, depending on the position of each line in (3), the character of (a) term in (2) becomes more "emphasized" than the property of (b) term. That is, when the long line and the thick line are in the vertical position, the property as the vertical line is strengthened more than the short line and the thin line. If you are in a diagonal position, after all, long lines, thick lines are shorter than thin lines,
That power is greatly strengthened.

In addition, the horizon extends freely,
There is quietness. In addition, it has a very stable property, and when it is used when the screen changes too much and becomes unstable, it calms down quietly, but when it is used easily, the screen has no tension. As described above, when the horizon is thick, strong, and long, it exhibits its original properties and power, and if it is short or thin, its effect is weakened.

Further, the vertical line originally implies the attractive force and gravity of the earth, and represents the situation (that is, the locus) of the falling of the object. Therefore, in addition to expressing a stable feeling as well as a natural feeling, it also includes speed, obedientness, naive feeling, and lonely feeling. And the image that you feel will be different depending on the amount of lines mentioned above. The line is thick, strong,
When it is large, it makes you feel dignified, rigorous, and powerful, but on the contrary, when the line is long and weak, it makes you feel lonely, sad, and pitiful. This can be seen from the fact that the thick Japanese cedar trees that are several hundred years old feel dignified and majestic, but the thin plows feel weak and lonely.

It is said that the remarkable appearance of the characteristics of the horizontal line and the vertical line is due to the motility of the muscles that move the human eyeball. That is, since the resistance is higher in the vertical direction than in the horizontal direction, as a result, a strong sense of tension and solemnity are emphasized from the vertical line, while an expansive and calm feeling becomes dominant from the horizontal line.

The diagonal lines are considered to represent feelings of movement, resistance and the like. In other words, it is a form that displays the force of nature, the force of human will to counter gravity, and the continuation of its resistance. Also, you can feel the sharpness of strong fighting. Therefore, when there are a lot of diagonal lines, for example, rocks with a large amount of water and old trees on a mountain with many twisted branches give a feeling of violent force and intense fighting.

Also, the feeling that the force flows along the inclination, the dynamic feeling that the exercise progresses is promoted, the restless and uneasy feeling, the feeling of being defeated, and the feeling of irritating somehow. Therefore, it is difficult to find a sense of stillness and rest.

The thickness or length emphasizes such a property as in the case of horizontal lines or vertical lines, but in the case of diagonal lines,
That angle is also an important factor. For example, as the inclination angle becomes a right angle, that is, an acute angle, that is, when the angle approaches horizontal, the property of the horizon such as safety and quietness is inclined.

In a landscape image, the position of the horizon has various influences on human sensitivity. For example, when the horizon is low and the sky is large, it gives a "spacious" impression, and conversely, when the horizon is high and the sky is not so clear, the viewer is given "compression". For example, if you have a sea and want to place a ship there, you need to consider the balance of the picture.

When taking a photograph of a landscape, the photographer considers and balances the following points. The center of the screen is the weakest point and not the most intriguing position. because,
Because the centers are located at equal distances from each perimeter of the screen, they are too regular and descriptive and poorly varied. Therefore, descriptive, non-changing archival photographs, specimen photographs, etc. center the object on the screen.

In a landscape image, all the objects in the landscape must be constructed in harmony, proportion, and balance on the screen. This is what is called balance. If the screen is constructed by ignoring this balance, the screen will be solid and unstable.
In order to create a well-balanced landscape image, first draw three straight lines as shown in Fig. 2 and number the intersections.

The theme motif is placed anywhere from 1 to 4 in FIG. If there are two motifs, place them at 1 if the first one is 2, then at 4 and so on. The third is placed in a place where the three motifs do not line up in a straight line. Similarly, in the case of setting the fourth and fifth motifs below, the motifs should be arranged so that they do not form a straight line with the previous two motifs. Using these methods, photographers take well-balanced landscape photographs.

In addition to the above, the number of motifs present in an image also affects human sensitivity. If there are many motifs in the image, it will be lively, and if there are few, it will be quiet. This is the same as feeling quiet when there are few people and busy when there are many people in the real world.

Repeating the same shape creates a rhythm.
If a certain shape, whether it is a sharp triangle or a circular arc, appears repeatedly on the screen, you can enjoy a rhythm by itself. Its shape is characterized by making it easy to distinguish it from others, and simple shapes tend to produce rhythm.

There is a composition in which two motifs are confronted clearly and strongly. There are various contrasting elements such as the plain against the mountains, the rough sea and the cape, the narrow space and the spacious space, the past and the future, the man and the woman, etc. To produce.

There is peace in the space surrounded by walls and large trees, and in the paths of dead ends. In contrast to the infinite space, in a closed space, it becomes a place like a static sun pool,
Become a community venue. However, if there is no open loophole, it becomes a suffocating place.

Having described the basic requirements for media, the method and system of the present invention will now be described.

[0055]

BEST MODE FOR CARRYING OUT THE INVENTION The present invention appropriately and selectively displays concrete information stored in advance in an abstract medium represented by language, voice, sound or characters. Specific information in the above includes visual (image), auditory (voice,
It is derived from the five senses, such as the flow of a river, sounds of various natures or animals, odors (fragrances, etc.), tastes (tastes), and touches (cold breeze). In the above, sight and hearing are typical. For example, a text that describes a landscape represents a landscape picture that matches the text, and represents the stream of a stream or the singing of birds included in the landscape picture. Therefore, if the objects are systematically organized and stored in advance and selectively used by the user, the objects will not be missed.

[0056]

Embodiment 1 That is, the present invention is a system "Automatic Picture Crea" for creating a "landscape image" from a "landscape depiction sentence".
"Picnyck" is an abbreviation for "tion system based on Composition Knowledge." A picnic is a database of various image parts that make up a landscape, such as "trees,""mountains,""sea," and "sky." Based on the user's input, using the theory described later, search the image parts group that seems to be suitable,
Then, arrange them on the screen to create one landscape image. Figure 3 shows an overview of the system.

The object in this embodiment is
It is what constitutes a landscape such as a "forest" or a "sea." An object is a concept, and has the following information as knowledge that does not have the substance of an image.

(1) ID number Each object is a unique I from 0 to 39.
I have a D number. (2) ID of Parent Object For example, “ship” exists on “sea” and “cow” exists on “ground” or “grassland”. By mistake, cows do not fly in the air and ships do not climb mountains. Therefore, when arranging image parts on the screen, it is necessary to have information that what object should be on which object. In the system of the present invention, an object that can be on top is called a parent object, and each object is an I of the parent object.
Stores D.

Currently, the system defines 40 objects (Table 1).

[0060]

[Table 1]

The image parts are images actually existing in the database and have the following information.

(1) ID Number Each image part has a unique ID number, like the object. (2) Group Each image part belongs to some object and has the ID of the object to which it belongs. That is, a mountain image part has a mountain object ID. (3) Size Size represents how many meters the height of the image part is in the real world. This knowledge is necessary to create a landscape image from various image parts so that the image parts do not become inconsistent with reality. If this knowledge does not exist, "people larger than mountains" may appear in the landscape image.

Generally speaking, there are "mountain scenery" and "sea scenery" even if they are called landscape images. There are probably "seascapes" and "sandy beaches" as well as "sea and islands".
Then, the existing objects and their arrangement are different for each landscape. The system of the present invention has necessary knowledge as a template for synthesizing the image parts group retrieved from the database. The template is, for example, "Mountain landscape template".
There are multiple such as "and seascape template",
An optimal template is selected from the input objects, and a landscape image is created using the knowledge of the selected template.

In the system of the embodiment, eight types of templates as shown in Table 2 are defined.

[0065]

[Table 2]

The template has the following information.

(1) Search Keyword The search keyword is a key for determining which template to use, and stores the ID of the object that characterizes the template.

(2) Subordinate Template Some templates have a tree structure, and if there is a subordinate template under their own template, store its ID. If there is no lower template, -1 is stored.

(3) Essential Objects In the system of the present invention, the minimum objects that make up each landscape image are called "essential objects". Each template stores the IDs of essential objects in an array. The order of this ID array is significant, and when synthesizing the image parts, the essential objects are arranged in the order of this ID array. For example, a template has sky, mountains, and ground as required objects, and when creating a landscape image, the sky image parts,
Arrange the mountain image parts and the ground image parts in this order.

(4) Position of essential object The template has position information for arranging the above-mentioned essential object. This position information is represented by an axis (axis) such as the horizon and a position (distance) from the axis, respectively for the x coordinate and the y coordinate. By making a relative expression in this way, it is possible to cope with a change in the position of the horizon depending on the input sentence.
The axis is defined as follows. x coordinate 1. Left edge of screen, 2. Right edge of screen, 3. Screen center y coordinate 1. Top of screen, 2. Bottom of screen, 3. On the horizon, 4. Below the horizon, 5. Horizon and center of bottom of screen

(5) Horizon Position Templates each have default horizon position information. If there is no sensitivity word in the input, use the default horizon position. And
If any sensibility word is input, move it up or down based on the default position. A template is determined from the complemented word group. Templates have search keywords, match words with keywords, and select a template that seems appropriate. Templates have a tree structure, and if there are multiple templates that are considered appropriate, the template with the lowest hierarchy is used.

In the system of the present invention, "landscape depiction sentence"
There are several steps involved in creating a "landscape image". The flow of this system is shown and each step is explained.

The words recognized by the system of the present invention are classified into the following three types.

(1) Object As described above, a landscape such as "forest,""sea," or "cow" is constructed. (2) Objective condition words Words about time such as "morning" and "daytime" and "spring" and "summer"
Condition words that do not include human subjectivity, such as seasons and weather related words such as "sunny" and "rain." (3) Kansei words Kansei words such as "rough", "quiet" and "magnificent" used to describe landscapes.

If any of the conditions is not specified, the system sets a default. This is because when the user does not specify an objective condition, a landscape that is inconsistent with the reality such as "the ground is snowy but the distant mountains are lush" is not created. Also,
It is also possible for the user to prevent this default setting.

When a landscape image is created from a scene depiction sentence,
There are things to consider. That is the difference in the amount of information that media have. For example, when comparing the "illustration" in the novel with the "landscape description", the difference in the amount of information is clear.

[0077] Usually, a scene is depicted such as "the ship returned to the port at dusk whistling like a lonely whistle," or "when I passed through the forest, a majestic waterfall appeared in front of me." . As can be seen from these example sentences, the "landscape depiction sentence" only represents a characteristic part of the landscape. In addition to what is represented by a sentence, for example, there will be no words in the first sentence, but there will be a "sea" if it is a "port." Of course, you can see the "sky", and maybe "gulls".

If all the things in the picture are represented by letters, no matter how much paper there is, it will not be enough.

Therefore, from "landscape depiction sentence" to "landscape image"
When you create, you have to imagine and supplement the parts that do not exist in the sentence.

We have classified this complement into four types.

(1) Basic Complement: Complement the objects that are considered to be the minimum necessary for constructing a landscape image. (2) Higher-order complement The upper-level (parent) objects such as "sea" when only "port" is given and "forest" when only "cicada" is given are complemented. (3) Subordinate supplementation "Cicada" when there is a "forest", "ship" when there is a "sea"
Complement those that are subordinate (child) objects such as. (4) Related Complementation This is the complementation when there is a certain object such as when there is a "waterfall" and when there is a "river".

The complement is performed from the input word group, and the system of the present invention uses the complement table.
The complementation table stores the complementation degree of an object with another object. If the degree of complementation is 0, there is no need to complement at all. If it is 1, it must be complemented. If it is 0.5, it may be necessary to supplement it in some cases.

As described above, in the case of the basic complement, the higher rank complement, and the related complement, the target object (for example, “sea” when the input is “ship”) must be complemented. Is 1. On the other hand, the degree of complementarity is 0 for objects such as "mountains" and "ships" or "sea" and "waterfalls" that are not supposed to exist at the same time. This degree of complementation was subjectively entered by us.

Complementation is performed for each input object, a threshold value is determined, and an object having a complementation degree equal to or higher than the threshold value is complemented.

The threshold can be changed by the user.
If the threshold value is raised, a landscape image faithful to the words in the sentence is created, and if the threshold value is lowered, a imaginative landscape image is created.

For example, suppose that there is a complementary table as shown in Table 3. In Table 3, the vertical is the input object,
The horizontal is the object to be complemented. This time,
When the threshold is 0.5, the objects such as “mountain”, “sky”, and “ground” are complemented to the input “There is a refreshing forest”, but when the threshold is set to 0.3, “semi” is also added. Will be complemented. In addition, when the same object is complemented, the complement degree is higher.

[0087]

[Table 3]

The objects after being complemented each have a degree of complementation, but the object whose degree of complementation is 1 and the other objects are handled differently in the search unit.

An object having a complementation degree of 1 is an object that is always required when creating a landscape image suitable for a landscape depiction text input by the user, so image parts belonging to that object are always searched. To be done.

In the search, the degree of satisfaction of the condition input by the user of each image part belonging to the object is obtained,
Search for the most satisfying image parts. For example, FIG.
Consider a case where an image part belonging to the object "mountain" is searched for under such conditions.

As shown in FIG. 4, each image part and each condition are linked by a link, and each has a membership value. This time, the average of membership values for each condition was used as satisfaction.

In this example, when the satisfaction of each image part is calculated, the result is as shown in Table 4, and the mountain "Mt.1" having the highest satisfaction is searched.

[0093]

[Table 4]

When a plurality of image parts having the same degree of satisfaction exist, a random number is used for the determination.

An object whose degree of complementarity is less than 1 is an object that needs to be complemented in some cases, as described above. These objects are searched in consideration of the condition input by the user, and are not searched unless there is an image part that meets the condition.

As a search method, first, even one of the membership values for the condition input by the user is 0.
The image parts that have the value of are excluded from the search target. Next, the degree of satisfaction of the remaining image parts was obtained by the method described above, and the image part with the highest degree of satisfaction was searched.

By using this method, it is possible to create a flexible landscape image according to the real world such that the character "cow" does not appear in the input scenery depiction sentence but "cow" does not appear. Becomes

The image parts placement unit places the retrieved image parts group on the screen based on the template information. The placement of the image parts is performed by the following procedure.

First, the position of the horizon is determined. Since the template has information on the position of the default horizon, this position of the default horizon is used when there is no sensitivity word in the user input. Then, if there is any sensitivity word in the input by the user, the position of the horizon is moved up or down accordingly.

Next, the image parts belonging to the objects described as essential objects in the template are arranged. As described above, the essential objects are arranged in the order in which they are stored in the array. For example, FIG.
Consider the following case.

First, an “empty” image is arranged at the top of the screen. Then, the image of the “ground” is placed below the horizon and the image of the “mountain” is placed above the horizon, and the placement of the essential objects is completed.

The system of the present invention has a management table in order to prevent a landscape image (for example, a ship flying in the sky or a cow climbing a tree) inconsistent with reality from being created.

The management table is composed of a two-dimensional array having the same size as the landscape image, and the members of the array correspond to one pixel of the image. Then, the array stores the object ID number to which the image part on the corresponding pixel belongs (FIG. 8). The management table is
It is updated every time one object is placed, and is used by the system to grasp the current image information. After the placement of the required objects is completed, the placement of other objects is performed. The objects are arranged by the following method (FIG. 9).

For example, when there is an image of "cow" in the search result, the parent object of the cow is "ground" and "meadow", so it can be anywhere on the "ground" or "meadow" on the screen. If it is too close to the edge of the screen, the balance of the entire image will be poor, but it is not interesting to place it in the middle.

When arranging an image on the ground, if the image is simply arranged at a designated place, the perspective of the resulting image becomes unreasonable and an unrealistic landscape image is produced. I will end up. It is also possible that the image parts in the database do not have a uniform reduction ratio, such as "images of cows larger than mountains." Therefore, the system of the present invention is arranged in consideration of the perspective of the screen and the size of the object in the real world.

First, the reduction rates of the image parts are made uniform. As mentioned above, each image part has its own height and actual size (size).
It has the information that the unit is meters. Also, 1
The height of the image (STD-DOT) when the image of a meter object is at the bottom of the screen is defined in advance by this system. Thus, when arranging a certain image part at the lower end of the screen, it is possible to obtain how large the image should be displayed.

The reduction ratio (std-ratio) when arranging a certain image part at the lower end of the screen is obtained by the following formula.

[0108]

As a method of expressing perspective, the area from the lower end of the screen to the horizon is divided into 40 parts, and the size of the image is reduced by 0.95 times when approaching the one-section horizon side. That is, assuming that the designated coordinates are in the nth partition from the bottom of the screen, the image parts are displayed at the following reduction ratio (ratio).

Reduction rate = reduction rate of parts arrangement × 0.95 ⁿ⁻¹

As a result, images can be arranged in consideration of perspective and the size of the actual object.

When arranging the image parts on the screen, it is necessary to consider the parent object of the object to which the image parts belong.

If it is impossible to dispose at the designated place, the position is moved. The movement of the place is shown in FIG.
As shown in, move the vortex counterclockwise around the initial position.

Next, a method for determining the membership between image parts and affective words will be described. In the system of the present invention, the membership value is determined based on the relationship between composition and sensitivity.

There is a strong relationship between the composition of a landscape image and sensibility. For example, a landscape with a low horizon and a large sky gives a "spacious" impression, and an image composed of curves gives a "soft" impression.

In the present invention, words directly associated with such composition rules are called "composition-sensitive words".
In the embodiment, the following 13 words are used as composition sensitivity words (Table 5).

[0117]

[Table 5]

Certainly, these words are directly linked to the composition rules. However, I think that many of these words lack practicality.

For example, the word "spacious" or the like is actually used in a landscape depiction sentence such as "spacious grassland". However, words such as "dynamic" and "obedient" are not often used to describe landscapes.
When you say "dynamic mountain", you can imagine what kind of mountain it is, but usually, rather than such description, a description such as "dynamic mountain" or "rough mountain" is made. . Further, in the system of the present invention in which the input is a landscape depiction sentence, it is too inflexible to recognize only 13 sensitivity words as composition sensitivity words.

Therefore, other than the "composition affective word", the affective words actually used for describing the landscape are called "general affective words", and the relation between the "composition affective words" and the "general affective words" is obtained, so that more We decided to make the Kansei words available in the system. This time, "composition affective words" and "general affective words"
The relationship was calculated by questionnaire.

The questionnaire method will be described below.

First, the affective words to be recognized by the system of the present invention were determined as "general affective words". From the Japanese dictionary, we have extracted the sensitivity words that seem to be suitable for describing the landscape. However, sensibility words that are rarely used in daily life such as "Ayoyo" and "Abundant" were excluded at this point.
At this stage, 132 affective words were extracted. Next, we grouped words that seemed to have similar meanings, such as "elegant" and "graceful", "quiet" and "quiet", and grouped the sensitivity words into 45 groups. Summarized. And to represent a word from that group,
Used in the questionnaire.

For the questionnaire, a list of 45 representative affective words of the previous group was prepared, one "composition affective word" was presented to the subject, and an impression similar to the "composition affective word" was given. I took the method of having any number of "sensitivity words" selected from the list. This questionnaire was conducted for one person for all "composition sensitivity words" and the
I went to a person. Table 6 shows the representative 45 words of the general affective words used in the questionnaire.

[0124]

[Table 6]

As described above, there are two types of "composition sensitivity words", "related to line element" and "related to horizon height". The former is 12 words excluding "feeling of pressure", and the latter is 2 words of "broad" and "feeling of pressure".
The word "spacious" refers to both "elements of the line" and "height of the horizon." These two types of "composition sensitivity words"
The results of the questionnaire were analyzed using different analysis methods. “Composition sensitivity word” for “line element” is 1
There are two words, which are associated with "line elements" such as "straight line,""curve,""horizontalline,""verticalline," and "slanted line" (Table 7).

[0126]

[Table 7]

Here, a "triangle" as a special shape in which lines are combined is also treated as an "element of a line". First, the number of appearances of each "general affective word" is set to 2
Divide by 0 to find the similarity between the “general affective word” and the “composition affective word”. FIG. 11 shows the similarity between the general affective word "rough" and each "composition affective word".

Since the "composition sensitivity word" is associated with the "line element", the relationship between the "general sensitivity word" and the "line element" can be obtained. Since "straight lines" and "triangles" are associated only with one "composition sensitivity word" that is "stiff" and "stout", the similarity with "stiff" and "stout" is directly taken as "straightness". It can be “triangularity”. Other "line elements" are associated with a plurality of "composition sensitivity words". For example, since two "composition sensitivity words" of "silence" and "spacious" are associated with the "horizontal line", the one having a higher degree of similarity to the two "composition sensitivity words" is defined as "horizontal degree". .
Here, the degree of straightness is called "straightness", and the degree of curve is called "curveness".
Using the method as described above, the relationship between "general sensitivity words" and "line elements" was obtained.

Next, a method of analyzing the shape of an image and obtaining the degree of strength of each line component, such as straightness, curvilinearity, and horizontal linearity, of each image will be described.

According to the present invention, six elements of straightness, curvilinearity, horizontal linearity, vertical linearity, diagonal linearity, and triangularity are extracted from the target image data. Although it is only human's subjectivity to judge the constituent elements of these images, in the present invention, the following algorithm was used when this work was performed by the computer. However, this algorithm is only an example, and it may be necessary to tune according to human sensitivity.

In the present invention, analysis is performed on line segments that cross the image left and right, such as mountain ridges. First, an object to be analyzed is cut out from the original image and its contour is extracted (FIG. 12).

The contour line is converted into the form of bitmap data (consecutive point sequence), which is stored in a one-dimensional array to analyze the constituent elements (FIG. 13).

Regarding the linearity, first, a process of extracting a line segment from the point sequence is performed. The change difference in the vertical direction of the point sequence is calculated. The point at which the sign of this change difference is inverted and the point at which two or more 0's continue in the change difference are extracted and used as the vertices. A line segment group having the apex thus obtained as the starting point and the ending point is derived.

The extent to which these line segments deviate from a straight line is defined as shown in FIG. 14, and this is added for each line segment. The deviation from the straight line thus calculated was calculated for all line segments and averaged. Using the average value of this deviation, the linearity was calculated by the following formula.

[0135]

The linearity is normalized to a value of 0 to 1, and the closer to 1, the higher the linearity.

The curvature is considered to have a complementary relationship with the straightness. Therefore, the straightness is defined as the curve.

Regarding the horizontal linearity, the change difference of the point sequence is 2
The length is added to the portion continuously showing 0 or more pieces. The ratio between the integrated result and the width of the point sequence was defined as the horizontal line degree. That is, in the case of a straight point sequence having no change in the vertical direction, the horizontal linearity becomes 1, and the horizontal linearity approaches 0 as the undulation of the point sequence increases.

With respect to the vertical linearity, the absolute values of the differences between the point sequences are added, and this value is used to analyze how vertical the point sequence has a component. When all point sequences have a change difference by the height of the figure to be analyzed (height heightdot, width widthdot) (for all i | X _i −X _i-1 | = height)
Was set as the standard, and this was defined as the vertical linearity of 1. That is, this is the case where the value obtained by accumulating the absolute values of the differences between the point sequences is width * height. Therefore, the vertical linearity is

[Equation 1] Is calculated.

However, when this value is actually used, the value tends to be too close to 0, which causes a deviation from the human sense. Therefore, in practice, multiplying this value by 1/8,

[Equation 2] Is used.

Regarding the slanting degree, the slanting degree becomes higher in a line segment including many line segments having an inclination of 45 °. Therefore, for each point in the point sequence, the length of the point sequence in which adjacent points have an inclination of 45 ° is added, and the ratio of this to the horizontal width of the image is defined as the slanting degree.

Regarding the triangulation, the first condition is that the peak having the maximum height in the line segment group is between 1/3 and 1/2 of the horizontal width of the image. Data that does not satisfy this condition has a trigonality of 0. With respect to the data satisfying this condition, the deviation from the triangle connecting the vertex and both ends of the line segment group was defined as the triangulation (FIG. 15). This deviation is calculated in the same way as when the linearity is calculated.

With respect to the image parts, the composition element is analyzed by the above-mentioned method. From the result, the membership value to the affective word is obtained as follows.

Membership values A _ij (i = 1.I: j = 1 ..) for J composition elements E _j (j = 1.j) for I image parts P _i (i = 1.I). j) is obtained as in the example of Table 8,

[Table 8] In addition, membership values B _kj (k = 1.K: j = 1 ....) to the composition element E _j for K sensitivity words F _k (k = 1.K).
Suppose J) is calculated as in the following example.

[0145]

[Table 9]

At this time, the membership value C _ki for the sensitivity word F _k of the image part P _i is

(Equation 3) It is calculated by the following equation.

“Composition sensitivity word” relating to “horizontal height”
Is the two words of "spacious" and "compressive". As mentioned above, a low horizon and a wide sky are "spacious"
I feel that, on the contrary, I feel that I feel "compressed" when the horizon is high and the sky is narrow.

Therefore, based on the result of the questionnaire, how much each "general affective word" is similar to "broad" and "compressive feeling" was sought. Similar to the case of “line element”, the degree of similarity was obtained by dividing the number of appearances of “general affective word” by 20 subjects.

Next, the similarity for "spacious" was subtracted from the similarity for "compressive" to determine the "horizontal movement coefficient" (μhorizon) of each "general sensitivity word" (Table 10). The "horizon movement coefficient" has a value from -1 to +1. If the value is positive, it lowers the horizon, and if it is negative, it raises the horizon (plus or minus may give the impression that it is the opposite, but it is This is because the upper left of the image is the reference point (0.0) in the system of the invention.).

[0150]

[Table 10]

The width w to move the horizon is the maximum width (HORIZON-MAX) to move in the system of the present invention.
Has been set,

(Equation 4) And The maximum width of movement at this stage is set to 70 pixels.

In the above embodiments, as an example, the system for selecting various images from language, voice, sound, or characters by abstract description has been described. In addition to the above, sounds (voice, sesseragi, bird's voice, wind sound, etc.), fragrance (forest incense, flower incense, dead grass odor, etc.), tactile sense (bright breeze, storm front, wave breeze, etc.) Can be expressed.

Although each of the above embodiments is carried out alone or in combination,
It goes without saying that the present invention mainly uses images. Therefore, each of the concrete objects is added to the image individually or in combination.

[0154]

[Embodiment 2] The hardware configuration of the system of the present invention is as shown in FIG. In order to implement this system on a UNIX (X) -window and make it a more human-oriented user interface, GU
For I, OSF / Motif programming was used.
Next, image parts used as data are created as follows. First, a scanner captures a predetermined landscape image. Then, from among these, images such as "mountain" and "cow" which are likely to be used in the system of the present invention are cut out using an application such as X-point or XV and stored in the hard disk.

The input interface of the system of the present invention is composed of a scene depiction sentence input window and some buttons as shown in FIG. First, the user inputs the scene depiction sentence in the form of text in this window. Then, when the search button is pressed, a search is executed for the text entered in the scene description text input window.

Next, when the edit button is pressed, the image output window is displayed without searching. This is used when you want to output a landscape image created in advance.

In the file menu, save the scenery depiction text written in the window, read the scenery depiction text from the file, renew the text input window,
There is a function called system termination. Within this option is the ability to set the completion level. When the setting of the complement level is selected, a scroll bar appears as shown in FIG. 18, and the complement level can be freely changed in 10 steps from 0 to 9. When the level of complementation is 0, only minimal complementation is performed and a landscape image faithful to the input sentence is created. Then, if the level of complementation is increased, a landscape image with rich imagination is created. Inside the system, a value of 1-level of complementation / 10 is sent as a threshold value of the degree of complementation.

By default, the complementary level is set to 5.

As shown in FIGS. 19, 20, and 21, when comparing three types of mountain landscapes, it can be seen that mountains having different shapes are output according to the sensitivity words. For example, "magnificent" will output a triangular and massive Mt. Fuji (Fig. 19), "rough" will result in a jagged ridgeline (Fig. 20), and "idle" will result in a mountain with many curves (Fig. 21). ing. Also, it can be seen that the height of the horizon also changes according to the sensitivity word. Comparing Fig. 19 and Fig. 20, it is clear that the former is "magnificent" because the horizon is lower and the sky is larger.
Feel. Further, in FIG. 19, “cow” is complemented from the object “grassland”.

Next, the output results for the scenery of the sea will be shown, and at the same time, the difference in output when the level of complementation is changed is also shown.

In FIG. 22, since the complementing level is low, the scenery image is faithful to the text. Then, when the complement level is raised, the ship is complemented and displayed on the sea, as shown in FIG.

Examples of picnic applications according to the present invention are as follows.

Regarding the visualization of a novel, for example, while reading "Bocchan" and can think of the scenery of the scene in my mind, I cannot actually see the scenery. By using the picnic of the present invention, a favorite scene in the novel can be visualized. Furthermore, by combining it with the above-mentioned okes, a sound scene can be created.

In the future, when books are digitized into electronic books,
By using it together with picnic and okesu, you can feel as if you are in a book.

In addition, when giving a presentation, an image suitable for one's own presentation is required. By using the picnic, an image one can imagine can be created.

[0166] In the future, with the further spread of WWW, it seems that anyone can easily have a home page. In other words, a picnic will be useful as a simple multimedia creation tool when creating a home page consisting of multimedia information.

In the embodiment of the present invention, imaginary complementation of a lacking medium based on information of existing media, which is unknowingly performed by humans, is called "media complementation". , We proposed a picnic, which is a system that automatically creates landscape images from landscape description sentences by using the complement of this media.

Then, when creating a landscape image, a landscape image considering human sensitivity is used by utilizing the knowledge of the relationship between human composition and the composition of scenery accumulated by artists such as photographers and painters for many years. Tried to produce.

By using the picnic of the present invention, it is possible to visualize a favorite scene in a fictitious world only in the language of novels and to easily create an image used for a presentation as desired. .
As a result of evaluating the image created by the picnic, it was found that the picnic can construct a landscape image in consideration of human sensibilities.

Further, in creating a landscape image from a landscape depiction sentence, attention was paid to the relationship between the composition of the landscape image and human sensitivity, and above all, the shape of the object and the height of the horizon. However, there are other elements of composition that affect sensibility. For example, since the number and rhythm of objects also affect human sensibilities, it is easy to introduce such other composition elements.

Since it has been known that other factors such as the colors in the landscape image and the image of the object itself also affect the human sensibilities, only the composition will be determined as a factor for determining the membership of the image in the future. However, it is desirable to consider other factors as well.

In the present invention, a picnic, which is a system for automatically creating a landscape image from a landscape description sentence, is proposed and constructed by utilizing the complement of media. Therefore, we have previously proposed and constructed Okesu, a system that creates a sound scene from a landscape image. At present, by combining this picnic and okez and inputting a landscape depiction sentence, a landscape image and a sound landscape are automatically created, and a user can virtually move around in the system.

With the background of further development of virtual reality technology, it is possible to develop a system for complementing media for various human senses beyond visual and auditory senses.

[0174]

The following panel test was conducted to show the effect of the present invention.

First, a panel was asked to actually use the system, and then a questionnaire was conducted. The questionnaire items are as follows.

Q1. Do you think the concept is interesting? Q2. Do you think that the scenery was created according to the text? Q3. Do you think the displayed mountains match the text? Q4. Do you think the position of the horizon matches the text? Q5. Do you want to actually use this system? We asked the above questions to be rated on a 5-point scale. 1
"I don't think so at all." 5 is "I think so," 3
Was "neither" and "sometimes good and sometimes bad", and 2 and 4 were set to intermediate values. FIG. 24 shows a summary of the questionnaire results. This FIG.
As can be seen from the above, a very good evaluation was obtained for the created landscape image.

There were the following opinions regarding the system of the present invention.

(1) The concept that a flexible image can be created by adjusting the complementary level is very interesting. (2) Although it fits the text, there were some things that were different from the scenery I imagined. (3) Depending on the sensitivity words, there were some scenes that seemed strange. (4) I want you to increase the types of objects.

The above (2) will be dealt with by allowing the user to modify the image output in the future. Then, for example, after the output, the mountains can be changed to different mountains or the position of the cow can be moved to create a more favorite image. Regarding (3), this time, we focused only on the relationship between the composition of the landscape and the sensibility, but there are other factors that affect the human sensibility. For example,
It will be even more effective if it takes into account other factors such as color in the future, such as the color of the object, and can handle sensitivity words that cannot be expressed only by composition.

Further, it is easy to increase the types of objects and the number of image parts, and it is possible to further improve the degree of complementation.

As described above, various kinds of concrete information are supplemented to the abstract media of language, voice, sound, or characters, so that the information environment is improved and the higher the degree of complementation, It has the effect of being able to approach as much as possible (not the environment).

Therefore, by listening to a language, a voice, a sound, or the like, or by touching a character, a state of being in the actual place (virtual space) is reproduced, and there are various effects such as better understanding.

[Brief description of drawings]

FIG. 1 is a diagram showing types of line elements in the present invention.

2A and 2B are diagrams showing a balanced arrangement.

FIG. 3 is a conceptual diagram of the system of the present invention.

[FIG. 4] Similarly, template determination.

FIG. 5 is a diagram showing a part of image parts in the same manner.

FIG. 6 is a conceptual diagram showing retrieval of image parts.

FIG. 7 is a conceptual diagram showing the arrangement of essential objects.

FIG. 8A is a view of a landscape image. (B) A diagram of a management table.

FIG. 9 is a flowchart of the same system.

FIG. 10 is a movement diagram of the same place.

FIG. 11 is a diagram showing a relationship between a general affective word and a composition affective word.

FIG. 12 (a) is also an original image diagram. (B) The image figure which extracted the analysis object similarly. (C) Similarly, a contour diagram.

[FIG. 13] Similarly, bitmap data

FIG. 14 is a diagram similarly showing a deviation from a straight line.

FIG. 15 is a diagram similarly showing a degree of triangulation.

FIG. 16 is a system configuration diagram of the same.

FIG. 17 is a diagram of an input interface.

FIG. 18 is a complementary level setting diagram.

[Fig. 19] Similarly, "There is a majestic mountain across the grassland."
Input diagram for.

[Fig. 20] Similarly, input example diagram for "There is a rough mountain".

[Fig. 21] Similarly, an input example diagram for "There is a peaceful mountain".

[FIG. 22] Similarly, an input example diagram of a low complement level for “There is a beach where you can see the lighthouse”.

[FIG. 23] Similarly, an input example diagram of a high complement level for “There is a beach where you can see the lighthouse”.

FIG. 24 is a diagram showing a result of the same questionnaire.

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Kenichi Okada 4-25-12 Hongo, Bunkyo-ku, Tokyo (72) Inventor Tatsuma Okubo 3-14-1, Hiyoshi, Kohoku-ku, Yokohama-shi, Kanagawa Keio University Faculty of Science and Engineering Department of Measurement Engineering, Matsushita Laboratory (72) Inventor Naoto Oki, 14-1, Hiyoshi, Kohoku-ku, Yokohama-shi, Kanagawa 3-14-1, Hiyoshi, Keio University Faculty of Science and Engineering 3-14-1, Hiyoshi Keio University Faculty of Science and Engineering Department of Measurement Engineering Matsushita Laboratory

Claims (4)

[Claims] 1. A method of complementing an abstract medium, characterized in that specific lack information corresponding to the abstract medium is selected and displayed based on a user's instruction. 2. A method of complementing abstract media, characterized in that concrete lack information corresponding to the abstract media is selected based on a user's instruction and complemented by means corresponding to the five senses of a human being. 3. Specific lack information corresponding to an abstract medium is stored for each information category, and the lack information is selected for each category from the information according to a user's instruction, and complemented sequentially or simultaneously. Complementary system of abstract media characterized by doing. 4. Specific lack information corresponding to an abstract medium is stored by classification and by series, and the lack information is automatically or selectively complemented by classification and series that the user has recognized. An abstract media complementing system characterized by that.