Lecture notes as presented at Wiretap 6.07: Time-based Space by Maja Kuzmanovic on 2000-07-09

Time Based Space

• a space that is responsive and shapeable, as opposed to absolute and static space
• a time that is embedded in a changing space, and can have complex structure and duration
• a body in motion, a body aware of its impact on its surroundings, a body capable of changing the normalized flow of time and space
• a hybrid environment where the media and the matter fuse together, creating a malleable substance, opening new perspectives and challenging perception
• an experience of constant mutation, without a beginning or an ending

From these broad areas there are 4 key concepts that connect several projects that I will present on TBS Wiretap:

• spaceTime
• bodyMotion
• mediaFusion
• experienceMutation

Different approaches to SpaceTime in 4 ongoing projects: GoToØ, GroWorld, Media Sauna and T-Garden

GoTo0 is a storytelling research project that has its outcome on several platforms:

• virtual reality application
• web-based SMIL/HTML application
• hard-copy, non linear book
• multidisciplinary festival.

This text focuses on the first part of GoTo0, the VR, CAVE application. The project is a collaboration between the Multimedia and Human-Computer Interaction Theme at CWI, Center for Mathematics and Computer Science in Amsterdam, the Netherlands, and the Virtual Environments Group at GMD, National Institute for Information Technology in Sankt Augustin, Germany. We are currently working on the development of a short storytelling demo for a CAVE environment, and a hypermedia application for the World Wide Web, that will allow the CAVE story to grow.

Background

The theme of Goto0 is the link between virtual communication and bodily sensations. This theme has been chosen based on a research about the existence of virtual realities throughout history. Historic virtual realities are worlds that did not exist until someone shaped them, worlds that have all characteristics of reality, albeit an alternative one. These realities have often a religious or mystical background.

For their creators and 'inhabitants', such as mystics, shamans and alchemists, these virtual worlds were channels for a communication with a remote, immaterial entity: their god(s). Many similarities between these ancient views and today's VR can be found. For example:

1. The practitioners were constructing and exploring their individual identities through novel ways of communication (with God).

2. The structure of their worlds is exploratory, based on their itinerant journeys towards God(s).

3. They referenced the known models of the world, as metaphors to describe something imageless and virtual.

4. These Virtual realities were exclusive domains of secret sects, 'the chosen ones', just as today's VR technology too often remains in the hands of a few wealthy institutions, and it is seldom accessible for experiment, that is certainly needed for the understanding of its full possibilities. This technology offers the possibilities to extend our senses, and bridge the gap between the immaterial world of thought and the sensual world of bodily experience.

5. Shamans, alchemists and mystics all search for unity, interconnectedness and spiritual ecstasy; while at all times conscious that they are deeply rooted in their physical bodies, in flesh. Starting from the body, they explore their virtual worlds in the same way we explore cyberspace today. It is approached as an uncharted territory, that can take them far, but they will always remain anchored in reality, through their bodies.

6. They use reality as a reference point, from which they set out on a vague journey towards a vague goal, during which they both construct the virtual world and draw its maps, turning the virtuality inside out, making it a part of their everyday lives. These maps are their stories and songs.

Storytelling

The way the mystics and alchemists transfer information about the immaterial worlds has a specific poetic style. This inspires a search for a language that is opposed to documentation of facts, and opposed to describing laws and moral messages that the audience should follow. Such language can easily catch up with the inconsistencies in non-linear storytelling. It is a language that speaks in nuances and atmospheres, that progresses through subtle changes, rather than through fast actions with a culminating ending. Such an ending demands a collection of facts, laid out in a logic and causal fashion. No randomness or RAM-ness allowed. In interactive stories, the algorithmical organisation of the content is more suitable. It allows change from one reader to the other, it can incorporate both a fact and its opposite, and still seem consistent. The story spirals from one plot to another, and leaves plenty of room for participants' association.

Interactive Storytelling

In my previous work, I experimented using this language in interactive film, installations and web-based work. However, something was always missing. Hypermedia has proven to have some of the qualities needed for future content development: it encompasses complexity, non linearity, the collaborative and fragmentary nature of contemporary contents. However, the immersion in screen based hypermedia is minimal: the user is often bound to a mouse, keyboard or joystick to explore the stories. While in reality humans interact both with their bodies and minds, in multimedia the user is conceptually, but not physically a part of the story environment.

One way for the users to be fully involved in digital stories is the use of immersive technologies like virtual reality, that is developed for a more natural human-computer interaction. A mouse or a stylus are not a suitable interface for an immersive environment. An ideal situation would make interaction in VR as transparent as in the physical reality. We should be able to use all our senses for input and output of information, and still move freely through the world, leaving the leash of cables behind. Until the present day, if we want tactile feedback, we are usually tied up to a cumbersome device like the Phantom, that is too disruptive in a CAVE like environment. The CAVE at GMD in Sankt Augustin has a vibrating floor, that allows the participants to feel the haptics of the environment, but not to influence it. Developing wireless data-suits combining wearable computers and smart fabrics, that facilitate sending and receiving of tactile impulses might be one of the solutions.

There is still a lot of research needed for the understanding of tactile communication, both in virtual worlds and in reality. As Sadie Plant says: “Looking itself is at issue now… Zeroes and ones are utterly indiscriminate, recognizing none of the old boundaries between passages and channels of communication, and spilling out into the emergence of an entirely new sensory environment in which begins to be evident that 'touch' is not skin, but an interplay of senses, and keeping in touch or getting in touch is a matter of a fruitful meeting of senses, of sight translated into sound and sound into movement, taste and smell.”

Hypermedia Storytelling in VR

As the VR environment is continuous (albeit pixelized), so should be the connections between all sensory experiences. These experiences have to be closely linked to the passages between the different content parts: the hyperlinking in VR needs the development of sensually satisfying transitions. The metaphor of passing from one room to another, or moving through a tunnel does not have the same impact as e.g. the transitions in movies, where they form an essential part of the plot. Furthermore, they give the rhythm and the atmosphere to the story. In interactive storytelling, transitions are perhaps even more important: they represent the space and the time of a link. Blends, morphs and warps might be the most appropriate metaphors for a hyperlink in VR. They explore the insides of a link, and explicitly show the participants the impact of their choices on the environment. More interaction, more crossing of hyperlinks, more transitions, more in-between positions.

Very active users mould the environment as clay, and create a very personal view of the StoryScape. At some point, the predefined chunks of content could warp into an endless knot of transitions. It is possible that being immersed in such an environment might make the users motion sick, but this might also mean that we are developing a new physical sense, that will make the language of virtual reality understandable for our bodily senses, as happened with montage in cinema, that in the beginning made the audience confused and sometimes even nauseous. This VR sensation is a feeling of being in constant mutation, without a progress or a goal, other than to mutate again.

However, the resources for this project are limited, and the transitions we conceived are mostly visual and auditory, that are triggered by the participants' movement. This movement can be seen as a dance, an improvisation on an unwritten choreography (that emerges from the contact with the stories), through which the participants create their personalized environments. Assuming that the participants do interact, below is a sketch of what they might expect to experience in GoTo0

The Stories

The story they will find themselves immersed into is the following: Norn and ZUZ have an inspiring virtual contact, at the time when the communication technology is infected by a virus, that will eventually destroy the global networks. Norn works on the development of a bodily, telepathic search engine, the GoTo0, that will allow her to re-establish the contact with her virtual community, and with ZUZ above all.

GoTo0 is a system of associatively and contextually linked stories, sharing 15 common themes such as pain, rhythm, change, interiority etc. The stories represent Norn's thoughts and memories, and her search for the union with her virtual partner ZUZ. In the future, the internet users will have the possibility to expand these stories, through the GoTo0? web-application. We will create merely a framework, that is open-ended enough to allow the users to actively participate by adding new story fragments, connect them with the existing ones, add sound and visual media elements (that will be used as textures in the CAVE application).

Navigation

The timeline of the narrative is generated dynamically, based on the participants' path through the theme, that depend on their movement in the CAVE. Both the interface and the interaction are kept as simple and transparent as possible, so that they don't disrupt the feeling of immersion, but still give enough subtle clues for navigation. . The only interface elements are the barely visible swarming particles, that appear as suggestions of possible paths. The particles are superimposed on a texture of changing symbols.

The participants' movements across this texture are tracked, and influence the path through the stories. By turning their head, the users trigger events within one particular story fragment. For the interaction in GoTo0 we used a metaphor of a daydream: the participants are casually passing through the environment, while the media elements, such as video textures and sounds are projected from different directions. They do not disrupt the environment, as they are submerged under a semi-opaque shell. This can be compared to the mirrored images seen in a polished shop window, or incidental fragments of conversations in a noisy street. By moving their head and focusing on a specific media element, the participants 'call' it to the surface and are able to experience its contents. This also triggers the changes in the overall atmosphere of the story, as more contextually linked media elements might surface, pulled by the 'magnetic' force of the chosen content. Other, less contextually imminent media will disappear in the deeper layers of the environment.

Design

Visually speaking, the storySpace is undifferentiated at first. As the participants move, they 'crack' the environment, as walking on ice. Depending on how much time has passed, and how much of the environment is formed, the more or the less media textures the participants will perceive visually: in the beginning, there are only sound and tactile feedback. The more the environment opens up, the more visual the experience becomes:

The environment should be experienced as a very fragile system. It consists of a meters thick structure, made of thin layers of an ice or glass like substance. With every step, one falls deeper through the structure, that breaks open in big chunks at first, but the 'vibration' caused by the step, will break the big chunks into smaller fragments, so that the surface becomes increasingly less flat. When we map a video texture on the big portions of the environment, they are fragmented and mirrored in multiple perspectives.

Some of the media elements remain visible in the storyScape, while others disappear when a story-fragment is abandoned. When most of the environment becomes covered with these media textures, they form a cracked, broken up picture morphing from Norn's face to ZUZ's and back again.

Time in GoTo0

Global History

• multiple paths of different participants through time influence the behavior of the swarming particles, and the 'life' of the story fragments (fragments die out if not visited for a longer amount of time)

Individual History

• history of the person's movement in the CAVE, it has a linear past, but the present and future are a rhizome of possibilities

Thematic History

• associative network of stories that spirals back and forth through time

Spatial History

• the visual experience of the VE. Its evolution is based on the behavior (movement) of the participants, and their relationship to speed and acceleration. It is linear, but scaleable, and its duration can vary from session to session.

From virutality back into the real world, the GroWorld project for the Burning Man festival (www.burningman.com) in the Nevada desert. The desert is a space with strange space-time structures. The time is slowed down by the heat and the slow movement of dust, only to explode is short bursts of sandstorms and heavy winds. BM festival is situated there as a temporary reality, based on rhythms of appearance and disappearance. The GroWorld project is concerned with our living, growing space of the future, a space that is becoming increasingly engineerd, a reality in making, but not in becoming, where time shrinks down into laboratory instruments and is allowed to exist only in highly controlled conditions.

This work wishes to inspire discussions about the decrease of the global bio-diversity, due to the practices of transnational ‘life industries’. These industries range from pharmaceutical to agricultural, dealing with genetic manipulation of human, animal and plant cells, birth control and patenting organic materials such as seeds and DNA samples. There are numerous issues arising from the commercialisation of all life on Earth, that are pressing to be brought to attention of both local communities and global networks of professionals involved in developing technologies for these industries.

After religious revealing of the invisible realm of God, and the scientific revealing of the micro and the macro natures, our era needs revealing of another realm, an invisible realm of global interconnectivity and collaborative creativity. Are we again revealing an illusion? Do we think that we're reaching a time where free will and uncensored communication is becoming reality? Or should our revelation be that we're deluding ourselves, drifting in cyberspace, while our bodies and their surroundings are being patented and sold on genetic markets.

The installation consists of multiple parts that can be viewed as a whole or presented separately:

1. Night Shadows

Philosophy

• Body in today’s world is not merely a physical being of an individual, but consists of (at least) 3 parts: physical body, virtual body and data body. The first one is the body we have lived with throughout history. The second is a body that can be seen as our being in virtuality, and which does not necessarily map to our physical body. The virtual body is a body of great potential, promising new sensations, new sexuality, gender reassignment, experiencing multiple dimensions etc. However, the third body, data body, the total collection of files connected to a person, poses a great threat to the previous two, and could change the way we perceive our bodies in the future. As CAE writes: “Data have become the centre of social culture, and our organic flesh is nothing more than a counterfeit representation of the original data.” This means that this data can be used (and very often misused) by commercial and repressive apparatuses that could increasingly gain control over our bodies and lives. Obtaining this data does not take more effort than to brush against someone’s body, or to pick up someone’s eyelash. If all the systems are connected, our data body will provide an institution with all information it wants to have about an individual. Wherever you go, you leave traces behind you. Resistance? Futile.

Description

• The installation is a wall painted with light sensitive paint, and a large flash that is activated by motion/pressure sensors, situated in the vicinity of the wall. It will function only in very dark spaces. As the people pass in front of the wall, the flash lightens up, leaving the shadow/contour of the person on the wall for a short period of time. The image slowly disappears as new people pass by. Your trace remains behind you, long after your departure. Misformed by the wind and oversized by the light, the elements you cannot influence, but that decide for you what your presence in this temorary realm of shadows is going to be.

2. TransDimensional Biosphere

Philosophy

• It is often said that the information revolution is taking us away from the real world, that our experiences are increasingly mediated, and that we are beginning to see the world through the glasses of ‘flat-space’. The public space, that is historically an arena for free creativity is beginning to disappear behind the screens. Although mentally we do not feel isolated, physically, our bodies begin to show signs of atrophy. RSI, headaches, loss of clear vision, social disorientation etc. Maybe we should try to invert the two worlds: instead of pushing reality into virtuality, we could let the virtual worlds bleed out into the three-dimensional physical existence. Marcos Novak labelled this occurrence ‘eversion’. What happens when we enter in such an ‘everted’ space and infect it with the movements of our bodies? Could we inflate the 2d organisms and dance with them in reality? Could they grow, mutate and decay in our reality, following both the laws of virtuality and those of nature?

Description

• The TDB is a forest constructed out of luminescent wire. The two dimensional shapes, resembling line drawings of vanished flora and fauna (including homo sapiens) are spread in space, hanging densely from the ceiling and ‘growing’ from the ground. From far away, it looks like a database of two-dimensional drawings representing endangered organisms. By entering the forest, the participants bring the biosphere back to life. When the wires are moved in space, they create fields of light that can be seen as 3d shapes derived from the drawings. Throughout the forest, there are motion- and pressure sensors installed, activating small motors that move the wires in different paths (around their axis, horizontal/vertical, rotating…). Wires hanging loose from the ceiling are set in motion by the wind and the touch of the participants, as they pass underneath or next to them. The shapes vibrate, rotate, move and by doing so gain the third dimension, through the participant’s interaction with the environment. The construction of this hybrid space is a choreography in which multiple visitors can be engaged, both by manipulating the ‘line-drawings’ and by creating 3d shapes through movement.

3. Growth Bunker

Philosophy

• GB is a dystopian vision of a possible future, where the growth of organic life (def. To increase in size by a natural process) can be viewed solely in a virtual rarity cabinet. The building blocks of all life-forms are patented by transnational corporations, and stored in databases for cloning and manipulating purposes. The risks of a natural birth are too big, leading to an increasing control on the creation of life. The human body is merely an incubator machine, deprived of its sexual drifts, engaged in voluntary eugenics. It is a make-able society that discourages ‘the becoming’ as an unpredictable process. Even looking at the growth process is in this (maybe not so distant) future society seen as a sexual perversion. Therefore, a hidden bunker has been built, in which the growth-nostalgic individuals can feed on ‘erotic’ images of growing organisms (cells, seeds, roots, embryos).

Description

The bunker is a 6x6x6 foot physical space situated in the centre of the Transdimensional Biosphere (Installation 2.) The outside is covered with dark material, which does not reveal the happenings inside the space. Inside, the walls and the ceiling form a semi-sphere, on which 5-10 frame luminescent wire animations of growth are shown. The choice of the animations, and their speed is based on the movement of multiple participants in the bunker (which are sensed through a grid of pressure sensors). The more people participate, the denser the image will become. When there is no one present, the bunker is dark.

foam/sponge project (Chris Salter, Sha Xin Wei, Maja Kuzmanovic and others)

From the largness of a desert, into a small enclosed space of our 'media de-tox lounge'. It is an installation that explores how media influence different bodily rhythms, without actual movement of the body. The sauna resembles the MRI chambre, a semi-circular pipe in which the participants are pushed inside legs first, lying on a soft bed. The have headphones on and the images are projected very close to the eyes. People feel soaked in media so that the sampling-rate of their physical reality changes with the sampling rate of the media around them. in the future we wish to allow bodily- and the media reality to interact more, by coupling the media to the pulsations of blood or breath of the participant.

The room is pitch black and very gradually begins to hum with frequencies hovering on the threshold of hearing. The room activates. Barely perceptible UV light begins to filter into the room from the outside. Depending on the program the visitors choose, their time is slowed down or speeded up.

Projected faintly onto the walls of the room in a narrow strip are images of intense acceleration or extreme slowness: high speed motion through darkness, urban environment, night-air, abstract spaces. Some of the images, which frame the lower half of the space are perceivable only from the visitors peripheral vision, giving a sense of extreme speed/slowness while the body remains still. In time, faint images begin to appear on the ceiling of the sauna as well. When the program is finished, each spectator is slid out to the other side of the pipe.

T-Garden

foam sponge project: Chris Salter, Maja Kuzmanovic, Sha Xin Wei, Laura Farabo, Ozan Cakmakci, Joel Ryan, Dave Tonnesen, Sam Auinger, Ed Severinghouse, Evelina Kusaite, Cynthia Bohner-Vloet, Margot Pluijmers, Rudolf mestdagh, Craig Lindley and many more.

T-Garden is a responsive environment where visitors can put on sound, dance with images and play with media together in a tangible way, constructing musical and visual worlds 'on the fly'. The performance dissolves the lines between performer and spectator by creating a social, computational and media architecture that allows the visitor-players to sculpt and shape the overall environment.

As visitors enter the performance, they find an array of clothing from which they can choose to don. The clothing has specific exaggerated physical qualities of, for example, weight, size and material. This clothing is embedded with wearable sensing devices as well as small audio speakers. Individually, the visitors enter into several private vestibules-rehearsal studios where they can play with streams of sounds and compositional effects that is produced by and played within the clothing. There the visitors can reveal the aural and physical properties of their garment instrument, and gradually learn how to modulate and change the sounds they are receiving.

After practicing, the players then enter a circular room, thick with sound and image. The floor is covered with transforming, polymorphous video and computer- generated textures: organic forms, elemental and microscopic liquid and solid state changes. These phantasmagoric textures appear to breathe and dance according to the sound patterns in the room. In this garden, as the visitors pass near each other, their clothes will appear to howl and squeal - patterns of sound ‘bleeding’ from one body to another. As the visitors move about, their locations and groupings will strengthen and lighten the density of the visual environment while varying the melodic and rhythmic aspects of the sound space. Memory, population density and bodily proximity affect the dynamics of the room, causing growth, decay, infection and contamination in the visual environment. The floor is covered with oversized balls, that form physical barriers for the people and the visuals. The visitors can move the balls around and discover the images without deformations. They are changing the physical environment and sometimes even blocking the passage to their co-players. A social play, shaping a common temporary world should emerge from the interaction with the physical-, visual and aural topologies of the space.

Visitors leave traces and “converse” with each other in musical and visual phrases as they weave their way through the room. The traditional roles of spectator and actor dissolve into a field of performance as gesture immanent in ordinary life, where social play emerges without explicit rules or grammar.

Media and Time Structure

All media (clothing, image, sound) in the T-Garden environment follow one central theme: transmutation. Within this theme, the media will explore the connections between different mutating systems, such as alchemy, ecology, memory, archaeology and recognition. Melodic and rhythmic flows and cycles, morphs, transformations and pliability are some of the characteristics of media that will be explored and developed. Aural and visual density can be influenced and guided depending on the play of visitors. The entire space should appear experientially as if it was shapeable and responsive in a fluid and choreographic manner-where the visitors ’ own bodies can meld inside an alchemical landscape and, like the principal goal of alchemy itself, result in a transformation not only of the media but the visitors themselves.

The T-Garden's interaction structure unfolds by way of the shape of a three dimensional spiral or helicoid. The visitors can experience four different types of time (alchemical, archaeological, experienced and room (garden))during the

performance's duration.

1.Alchemical: At the basis of the t-garden experience for the visitors is embodied experience of real time social and media transformation. Each individual alchemical “slice ” or has its own transformational cycle. These cycles are composed of a series of state changes that move the visual layers of the room between different alchemical transformations (calcination, fermentation, distillation, separation, dissolution, etc) and physical and elemental changes (solid to liquid, organic to industrial, crystalline to pliant, stable to corroded, etc.). Visitors can influence both the time duration of a particular alchemical cycle (speeding it up or slowing it down depending on their room position, movement, tactile contact with others around them)as well as the types of transformations that occur.

This will be achieved technologically by dividing the playing space into separate responsive visual areas and using the sensing data coming off the visitors as well as dynamically tracking in which areas visitors are congregated, how close they are to each other, etc. By being able to influence and play with these alchemical characteristics, visitors can create multiple variations within a single cycle or smaller cycles within a larger one. In other words, the time cycles and content of these alchemical cycles can be modulated and radically changed.

II. Archaeological Time: Archaeological time is the active interpretation of the garden's history by the visitors ’ players which is arrived at by “digging “through the stacked slices or layers of alchemical time in the room.

The garden's archaeology unfolds as each successive alchemical layer is actively revealed over the course of the performance's duration. This is akin to literally and figuratively “peeling ” away layers of the room's media. The impression should be that of a thick “sandwich ” of visual media layers (digital composite images, textures and processes)that lie stacked “beneath ” the physical floor of the performance or exhibition space. The layers are revealed as visitors leave traces or “marks” in the visual media–standing in a particular location for too long can result in the media “floor ” giving away and the visitors falling through the layers, for example. Furthermore, holes and corrosion can develop in certain layers of media when no interaction occurs.

III. Experienced (real) Time: The actual clock time that the visitors are in the garden proper. This can also be noted as the user experience in the room. Useful in terms of tracking the history of bodies in the room during the 1 hour performance cycle: where people have been, what areas have had the most activity, how long have visitors been in a certain area, etc.

IV. Room Time: The “passive” memory of the room or the overall course of the garden's growth and decay. Each 1 hour performance cycle is constructed from a separate archaeology and history, although at times during the course of a day the visitors can experience “flashbacks ” over which they have no influence: traces of the room's previous history across different performance cycles.

The alchemy of T-Garden is a self-creating activity that does not accept the notion of a static world. An alchemical reality is a world in constant development-in becoming. By transforming the media around them, the visitors perform an alchemical journey where physical and virtual (or spiritual), exterior and interior, micro and macro exist in “the collision of two elastic spheres.” The image becomes tangible, the sound malleable and the clothing ethereal. The media use a dynamic language that can be compared to the movement of verbs instead of the symbolism of nouns. This is a language that connects the sensual experience of the visitors with the processes of memory and sedimentation. T-Garden should be experienced as an alchemical world, whose matter is not actually solid, but merely a stress, a strain in the field of time and space. It is the visitors ’ gestures ((that are not so different from everyday gestures of touching, brushing along other bodies, moving and falling, etc.) that define the matter of T-Garden. For without the human gestures, it would remain in a chaotic flux. Space without time, duration without distance, body without motion.