What is a Hologram?
The basic idea of a hologram is an apparition of something that does not exist but appears as if it was just in front of our eyes. These illusion techniques were invented a long time ago. The philosopher and alchemist Giovanni Battista della Porta invented an effect that was later developed and brought to fame by Prof. J. H. Pepper (1821-1900) and applied in theatrical performances. The innovation nowadays consists in the adopted technology to produce them. Taking advantage of the available digital technologies, the challenge we are going to discuss is to use holograms in the museum context, inside showcases, to realize a new form of scenography and dramaturgy around the exhibited objects. Case studies will be presented, with a detailed analysis of the EU project CEMEC, where holographic showcases have been designed, built and experimented in EU museums. In this case, the coexistence in the same space of the real artifact and the virtual contents, and interior setup of the showcase, its dynamic lighting system, the script and the sound, converge to create an expressive unity. The reconstruction of sensory and symbolic dimensions that are “beyond” any museum object can take the visitor in the middle of a lively and powerful experience with such technology, and represents an advancement in the museological sector.
In the last ten – fifteen years the domain of Virtual Museums (VMs) have reached a great improvement in terms of technological solutions, graphic implementation and visualization, interaction interfaces, and methodological pipelines.
However, especially when VMs are conceived to be included and accessed inside real museums (or exhibitions), digital applications often appear to be not sufficiently connected and harmonized with the real collections and the museum’s space.
Multimedia and virtual contents are confined in their own “frame” of exposition, either a screen, a multi-projection set, or a head-mounted display, and, again, a tablet. In such cases, the public can live experiences more or less immersive and engaging. Nevertheless, these experiences are optional and ancillary in relation to the visiting path along the collections, with whom they are not strictly related. For instance, temporary exhibitions commonly include introductive movies at the beginning of the visit, and sometimes tablets at the end, with the aim of offering additional information about the topics on show. However, the setup of an exhibition and the communicative approach of the collections remain very traditional, supported by printed panels and small captions located beside the objects. In that condition common visitors can only look at artifacts and admire their aesthetic consistency. They cannot “experience” anything of the objects’ contexts, of the stories and people that are “behind”.
Even in the case of more sophisticated installations, like VR applications using novel interaction paradigms or offering truly immersive experiences, the dialogue between digital contents and real artifacts can be often weak, sometimes even hostile.
The majority of visitors are usually immediately attracted by digital technologies, especially if they are new, because they are able to solicit their senses and curiosity. Playing with them is a challenge because the dynamism of multimedia is perceived as something that interrupts the silence and the repetitiveness of the museum visit experience, making it more pleasant, unpredictable and fun. This explains why so many visitors, entering in a room with audio-visual/virtual installation on one side and common showcases with real artifacts on the other side, are immediately attracted by the digital rather than by the real contents. This behavior has been very often observed and recorded in occasion of several surveys we have carried on in museums and exhibitions, aimed at evaluating the users’ interaction with real and digital contents.
On the other side, the attention towards the digital contents decreases rapidly if they do not offer a good connection with the real contents of the museum and if the storytelling is not really engaging. Technological innovation is not sufficient to keep the interest for a long time. The visitor entering a museum is going to live a complex and demanding experience, conceptual and physical (a long path, several rooms, showcases, a huge variety of objects coming from different ages and places..), and he does not want to be distracted for too long by an installation that is not completely involving and not coherent with the collection along the way.
In other words, it seems that today more and more museums are embracing the digital challenge but there is still not a real awareness of how to let virtual and real contents work together, to enhance the experience of the visitors in and with the museum’s collection.
Real and virtual continue to be juxtaposed but not combined together in order to produce a powerful experience of mixed reality.
Despite the great contribute that multimedia and VR could offer to the museum’s sector, the exhibited collections still suffer from lack of contextualization, of stories, of sensorial immersion and anthropological dimensions, joining together Past and Present.
A new conception of the museum’s space is needed, and a new curatorial practice as well (from both sides of real and virtual museums) to create such an integration, where users, collections and stories are complementary and interconnected. Such interconnection requires the cooperation of different professionals: content experts, designers, developers, creatives, technicians, cognitive scientists, artists, experts of usability.
The museum would thus become a place of enjoyment, learning and social inclusion, reflecting local, global, individual, collective dimensions and perspectives.
Starting from these premises, the intent of creating a “dramatic dimension” inside the museum’s space in order to push the user’s sensory involvement with objects and stories, has brought some researchers and creatives to test and apply an ancient representation technique, able to interact with real museum’s objects. This technique consists in the Pepper Ghost effect, living inside the showcase, in presence of the artifact, producing an effect of mixed reality. As National Research Council and E.V.O.CA. company, we are working with this kind of approach in the framework of the European project CEMEC, Connecting Early Medieval European Collections, still in progress (Creative Europe programme, 2015-2019).
The selection and use of such an approach derives from a long experience in the domain of VMs, accompanied by a deep reflection in the museological domain. We have experimented and developed several kinds of digital applications for museums: audio-visual contents visualized on screens along the paths of visit, beside the objects, with low level of interaction and characterized by an educational approach; VR environments located in secluded and dedicated spaces, in absence of real objects. Especially in this second case, the experimentation has been very long and complex, both with interaction and with interface design and storytelling. Several surveys on audience engagement with digital installations have been carried on in order to understand the impact of such experiences. What emerged from these evaluations is that visitors mostly love to be engaged and “embodied” in stories, to play as protagonists, but, also, they love to be guided and not abandoned to themselves. Free and unconditioned interaction in VR is something too complex and frustrating for visitors, especially if they have not a preliminary knowledge of the subjects, and if they want to follow a narration.
Narration and emotions play a crucial role in the public’s engagement, in cultural transmission and memorization. For this reason we are working on the creation of new narrative “forms” combining different paradigms and techniques, coming from VR, cinema, theatre, serious game, augmented reality, digital art. Each “language” and paradigm can suggest an inspiration, can open a vision, converging in the creation of something new for the next generation of VMs. These experiences led us to the experimentation of holographic showcases, where perception, mixed reality and dramatization are the essential topics.
The basic idea of a hologram is an apparition of something that seems tangible, in an empty space. Something that does not exist but appears as if it was just in front of our eyes. Indeed such technique of illusion was invented long time ago, in theatres; the innovation of using it today consists in the adopted technology to produce the same effect. Therefore, taking advantage of the available digital technologies today on market, the challenge is to realize a new form of scenography and dramaturgy around the museum’s objects, with a calibrated interaction according to the real needs and suitable for all targets, with a simple technology, compatible with the museum’s preservation and maintenance needs.
In the CEMEC project we have conceived and experimented holographic showcases, working on the definition of a coherent communicative format, where a) integration of real and digital contents, b) structure, c) materials, d) interior setup, e) illumination, f) perception, g) audio-visual grammar, h) narrative approach and dramatization, i) scalability, are designed as a whole. We have evolved a specific research on the communicative approach, engaging experts in humanities, museology, psychology, curators and artists, engineers, working together and supported by the user experience evaluators This integration and the obtained results that have helped us to define some general guidelines, constitutes the innovative value of our proposal.
The so called “box of stories” has been conceived as a small theatrical stage equipped with controls for the direction and synchronization of the individual stage devices: lights, audio speakers, scenography, projections. Everything is managed and controlled by a software specifically developed. As already told, the virtual contents are projected in the same space of the real object and bring it back to life, to its sensory dimension, following a “dramaturgy”. Real and symbolic scenarios can be evoked and represented, accompanied by emerging personages, voices and soundscape.
But why should we use holograms to tell the story of an object, rather than a common 2D screen or even a mobile device with a multimedia? It is not only to surprise and produce an astonishing reaction in the visitors. Through the hologram we change the traditional paradigm: if we include the real artifact inside the holographic showcase, the attention still remain focused on it. It stays as the real protagonist of the story, not only on a conceptual level but also in its physical consistency. Not its virtual replica but the original itself is the center of our attention along the time of the whole experience: all the virtual animations, the fragments of stories originate from its real figures and details. We can mount and dismount, virtually restore, move and transform the object, overlapping virtual projection on the real artifact, and playing with lights, thus creating an experience of mixed reality.
Through holograms we do not immerse ourselves in virtual reality using particular viewers/visors, but it is the virtual reality that enters our space. Our reality becomes an augmented reality. Because of this integration the museum’s object becomes alive, enforced and not weakened by multimedia. This is not a marginal difference, because it greatly influences perception, engagement and comprehension, favoring a deep relation between the visitor and the observed object.
In these years holographic techniques have greatly advanced. Thanks to holography it is possible to duplicate reality, create characters, objects or scenes that do not exist.
Holography is an optical technology for recording and storing visual information in the form of a very fine interweaving of interference fringes through the use of coherent laser light, appropriately projected; the image created by the interference fringes is characterized by an illusion of three-dimensionality. It is more properly a parallax effect in the perception of the image: the image is indeed perceived differently, depending on the point of view; in the case of two eyes, each of them perceives the image from a slightly different position with respect to the other. This difference, called parallax, in the normal vision determines the three-dimensional perception.
The etymology of the term “holography” comes from the ancient Greek ὅλος, holos, (“everything”), and γραφή, grafè, (“writing”) and literally means “I describe everything”. Holography was theorized in 1947 by the Hungarian scientist Dennis Gabor (1900-1979), honored with the Nobel Prize for the Physic in 1971, for this invention. It started to have meaningful applications with the introduction of laser sources of light, highly coherent, in Sixties.
In the registration of a hologram, the light from a laser is divided by a semi-transparent mirror (beam-splitter). The two resulting rays are then expanded and conveyed through appropriate mirrors: one of them goes illuminating the object (wave front of the object), while the second one directly affects a photographic plate (reference wave-front). On it the two wave fronts interfere, and the recording of the interference fringes is the hologram. The tridimensional effect is produced by diffraction. The plate is developed and fixed as in an ordinary black and white photographic process.
Holography preserves much more information than a common photographic plate or film, because it records not only the light intensity but also the phase of light wave, thanks to the use of laser.
However, this holographic technology is still immature to be applied to cinematic and multimedia. For instance, it is hardly applicable in exterior and it is not suitable for large objects. Moreover, the hologram recording and the storage on a photographic plate are obviously not suitable for moving images.
Advanced laboratories in the world are dealing with holography, like as the Massachusetts Institute of Technology (MIT), the College of Optical Science Of Arizona University (UA), the Fraunhofer-Gesellschaft in Germany, The Istituto Nazionale di Ottica (INO) of the National Research Council, Italy. Nowadays, the computational power made available by modern computers offers the possibility to generate holograms also through numerical procedures (DH – Digital Holography; CGH – Computer Generated Holography), able to calculate the interference fringes using mathematical algorithms, starting from a representative model of the object. At MIT, for instance, they use a TOF (Time Of Flight) camera (specifically a Microsoft Kinect) to produce a real time depth map of the shot scene (even in movement); then an algorithm extracts the fringes of interference that compose the digital hologram. Common GPU (Graphic Processor Unit) are able to calculate the hologram. At the University of Arizona they use, instead, another method, deriving a 3D model of the object taking images by mean of a set of video-cameras located all around the object itself. A parallel research is in progress in relation to displays, for which different technologies are experimented (SLM, Spatial Light Modulator, or a plate of special rewritable polymeric material).
However, the holographic images created by mean of the numerical procedures, especially those ones dynamically projected, still have many limits: low quality of the images, stains, noise, low contrast, low resolution or low frame rate, monochromatic aspect, very narrow field of view in the tridimensional perception, lack of interactivity For such a reason other techniques have taken place in multimedia. Beyond Gabor’s principles, today the word “holography” is often used in a generalized and not fully pertinent way. It is used to indicate a number of different methods able to produce translucent and tridimensional images, that are typical of the images obtained by mean of holography.
The Pepper’s Ghost is one of these techniques, particularly used in performances. It is appropriate and easy to implement also inside museums, in terms of integration with original artifacts, image quality, compatibility with preservation needs, scalability, adaptability, robustness, daily management, costs. In this paper, authors will refer in particular to this technique, discussing how it can be used in the context of museum exhibitions and how it can influence the creation of new narrative and representative approaches, focusing mainly on museological and communicative issues.
The Pepper’s Ghost and the illusion of reality
The Pepper’s Ghost effect is an apparition of something that does not effectively exist but appears as if it was just in front of our eyes. It is an illusion of reality.
In the XVI century the Neapolitan philosopher and alchemist Giovanni Battista della Porta (1535-1615) invented the Camera Obscura. In his publication “Magia Naturalis” (1584), he describes for the first time an optical illusion titled: “How we may see in a Chamber things that are not” . His invention was later developed and experimented, up to be brought into theatrical performances by the English scientist John Henry Pepper (1821-1900). He wanted to create magic effects, fascinating the spectators. The technique became very famous and, in honor of his developer, it was called “Pepper’s Ghost”.
The illusion consists in the perception of things, a place or a floating figure in the empty space, in a position where they are not in the reality – given that their real presence is in a secret place, hidden for the observers . Originally, during theatrical performances, a room was created in a position invisible for the public (defined “blue room”), adjacent and perfectly corresponding to the main stage. This hidden room could be located under the main stage or aside. In this hidden room real figures (characters, objects) moved in front of a powerful source of light. Once illuminated, these figures were reflected by an oblique transparent surface (typically a mirror) that was positioned with a corner of 45° between the hidden room and the spectators. Because of a optical effect, the reflected image was automatically projected on the stage, in front of the spectators’ eyes, on a determined surface. In this way, it was possible to create ghost effects – thus an illusion of reality.
Only what was illuminated by the source of light could be reflected by the mirror and projected on the stage: if the source of light in the hidden room illuminated only a figure, leaving the surrounding space in the darkness, only the ghost of that figure would have appeared, floating in the empty space of the stage, eventually interacting with real actors and real scenography.
Today the ancient ghost technique is still used on the stage of concerts, in theatrical performances ), in films or inside a museum’s showcase, at smaller scale.
Dead famous artists or singers relive and perform again on the stage thanks to holograms. This was, for instance, the case of Michael Jackson, who performed as a dead man in 2014 on the Billboard Awards stage. The dark room, the light and the real content we want to reflect as illusion are replaced by a monitor (or by a projector depending on the size) and virtually represented; the transparent mirror is usually a special glass or a film or a common Plexiglas.
Technological solutions and dimensions of a holographic setup, using Pepper’s Ghost effect, depends essentially on the dimensions expected for the ghost, especially if it must interact with real actors/environments. Nevertheless, dimension is just one of the main parameters able to increase the cost of such system. The quality of the infrastructural design and the hardware components can considerably increase the quality of the effect and, consequently, the cost. The final design of a holographic showcase and the final cost can indeed vary considerably – even if it is still an accessible technological solution for museums.
Some materials are fragile, the most delicate component is the mirror placed at 45 degree. In replacement of the solid mirror a thin and transparent film can be used, especially in the case of very large dimensions of the holographic setup (where the mirror would be impossible to manage). However, the transparent film is even more fragile than the mirror, it is difficult to clean and, above all, not simple to reuse in case of movement and repositioning of the Pepper’s Ghost setup: the film needs indeed to be perfectly stretched to be invisible and to avoid image deformation, and, due to its delicate consistency, it is hard, or impossible, to be stretched more than one time, thus the images will result deformed.
A common type of digital showcase is the holo-pyramid display, a combination of four Pepper’s Ghosts, properly illuminated by a screen, for example a tablet. Usually in this case, the curiosity of the spectator is aroused by the vision of a 3D object, floating and turning in the empty space. Frequently the holo-pyramid display is quite small, able to visualize a small virtual object with animations and possible accessorial graphic effects. The space available for something else to happen is very limited, so the storytelling potential; therefore audio contents and interaction between real and virtual contents are usually absent.
Other types of holographic showcases are possible, with different volumes and setup solutions. For the CEMEC project, Connecting Early Medieval European Collections, a small holographic showcase has been prototyped by the designers of the Allard Pierson Museum (APM), Amsterdam, and presented in the context of the CEMEC itinerant exhibition. It uses a mini-projector located at the bottom of the structure, instead of a TV screen; once projected on the ceiling of the small showcase, images are reflected by a flexible oblique transparent film (instead of a solid mirror).
Every time the showcase is mounted in a new location this film must be stretched again, until it becomes perfectly tense and invisible. Technically, this solution seems to work fine, even if it is a little bit critical along the time, because the film is not robust and has not long lasting mechanical properties. The window on the front is very small (about 40 cm wide x 15 cm high x 15 deep ) and it obliges visitors (maximum two at a time) to come very close to the glass,, as if it was a secret box to peek inside.
This small digital showcase is conceived to contain one small museum artifact (a golden brooch, or a belt buckle), while a ghost animation plays over it. A short audio description accompanies the virtual images, showing details of the golden jewelry in 3D, manufacturing procedures, while simple drawings suggest contextual scenes. The small dimensions of the showcase and the kind of user experience that has been designed do not encourage to create a real dramaturgy that would be unbalanced and too powerful in comparison with the strength of the visualization. This kind of holographic showcase is not very invasive of the museum space and it can easily be multiplied along the path of visit, to offer a series of multimedia events to the visitors.
The case of the big holographic showcase, again created in the context of the CEMEC project by CNR and E.V.O.CA. team (and presented in detail in the following sections), is more demanding in terms of space, and its multimedia potential is greater compared to the smallest. First of all because of its dimensions: the front window is 120 cm wide, the depth is 140 cm, the height 80 cm. A larger audience can see it at the same time and is possible to simulate many more events inside of it. Users can see more than one real object (or one big object), the interior space can host even a simple real scenography, and virtual characters can be projected in real scale. Also contextualization scenes can be represented more realistically. Figure n.7, for instance, shows how the real candle holder exhibited in the showcase is incorporated and contextualized in the virtual scene and thus brought to life. This approach opens exciting perspectives in the experimentation of dramaturgical paradigms, as we are going to discuss in the following sections of this paper.
Another wonderful example of Pepper’s Ghost of big dimensions, with a great dramaturgical power, is the Sarcophagus of the Spouses installation, specifically designed by the well known new media artist Franz Fischnaller, in collaboration with Giosuè Boetto, Cineca and the Superintendence for Archaeological Heritage of Lazio and Southern Etruria. This installation was conceived and realized in occasion of the temporary exhibition Etruscan journey towards the afterlife (2014 – 2015), in the History of Bologna – Museum Genus Bononiae, Bologna, Italy. Here virtual archaeology, 3D video mapping, holographic techniques, audio-visual narration, scenography have been combined together. Sure enough the virtual show was expected to involve the entire exhibition space of the room (14 x 14 x 15 mt high) in the narrative context. Therefore, the installation was based on 1) a three-sided holographic pyramid (4 mt wide x 3 mt. high), to visualize the ghost of the Sarcophagus of the Spouses in the middle of the space and in real scale (1,14 mt high x 1,9 mt wide), and 2) a 3D mapping video projection system to visualize the narrative context on the surrounding walls, thus creating a dramaturgy in the whole space.
This ambitious installation required a large area and, above all, a huge hardware apparatus that was possible to manage in occasion of a temporary exhibition but that could be hard to include permanently inside a museum in terms of daily maintenance, in absence of specific managerial and economic strategies.
Another interesting example comes from the National Archaeological Museum of Cerveteri. Here, in 2013, holographic showcases have been implemented by Mizar company in collaboration with Soprintendenza Archeologica per l’Etruria Meridionale and DTC Lazio. A simple touch on the external glass of the showcase, in correspondence of the artifact, activates a Pepper’s Ghost in the showcase that seems to “wrap” and animate the artifact. The animation describes the object in its details, contextualizing it with other objects of the tomb in which it was found.
In “Welcome to Rome” exhibit, (ex Cinema Augustus in Rome, Mizar, 2017), the story of Rome is told along a path of visit that presents exclusively multimedia installations. In each room, deeply dark, there was nothing except an installation. In this condition visitors entering the room were isolated and immediately induced to silently pay attention to the digital audio-visual narration. Many of these installations were showcases using Peppers’ Ghost effects, mapped directly on the real models of some important Roman monuments. What was mostly interesting was the interior setup of the holographic showcase: the space was very deep to enhance the 3D effect of the whole show, and the 3D model, materially reconstructed, worked as dynamic scenography of the virtual show.
The “Keys to Rome” exhibition (2014-2015) presented another occasion to work with Pepper’s Ghost. The exhibition, involving at the same time four European Museum and their collections, was an interactive journey to discover Roman culture, starting from the city of Augustus and spreading out over the entire Roman provinces. In the Imperial Fora Museum we (as CNR team) presented an holobox whose main issue was interaction. It consisted in a Pepper’s Ghost setup connected to a Leap Motion sensor, able to capture the hands’ gestures (https://vimeo.com/109283974) needed to explore and interact with objects. With holobox, visitors were able to see high resolution 3D virtual models of the objects in all four museums. The visitors had also the opportunity to manipulate them using a natural interaction system: Leap technology allowed them to actively play with the item by personally selecting details, looking at the object from different perspectives, loading other interconnected items. The selection of objects was done through another adjacent VR application, “Admotum game” where the context of the objects could be explored: items collected in Admotum scenarios could be “sent” to Holobox, for a detailed exploration.
In conclusion, pure visualization, without additional contents, can be useful to show digital collections, making accessible virtual replicas of real artifacts that are preserved elsewhere. In such visualization, it is of course possible to integrate additional contents, like for instance virtual restoration. However, if the aim is to transmit historical contents, believes, personages, events, contexts of usage, the creation of a story is needed, as some of these examples have shown and as our CEMEC project aims at.
The EU project CEMEC – Connecting Early Medieval Age Collections – gave us the opportunity to experiment and test assumptions and ideas about holographic systems in museums. Started in 2016, the project aims to create a collaborative network, and a cost-effective business model, between 8 European museum collections, 7 universities and 6 technical partners. Drawing on objects from participating museum collections, the project has produced a traveling exhibition focusing on connectivity and cultural exchange during the Early Middle Ages (300 -1000) in Europe and around the Mediterranean, from Ireland to Egypt and from Spain to Hungary and Greece.
Technology has an important role in this exhibition: it serves as a supplementary way of telling the story of the early medieval period to museum audiences. The holographic showcase is indeed the chosen communicative strategy. It has been used here to tell the story of some objects of the VII sec a.C. which are:
- Kunàgota sword. An Avar sword exposed at the National Hungarian Museum of Budapest (NHM), belonged to an Avar chief of the village of Kunàgota. It was never used because it represented a protective object for the afterlife of the buried man.
Mytilene treasure. A set of Byzantine objects from the Byzantine and Christian Museum of Athens, specifically a golden bracelet, a candlestick and a trulla, a tool for water’s spilling,
- belonged to a wealthy family that lived on the Mytilene island, in front of Asia Minor coast . These objects were part of the domestic equipment.
In the holographic showcase, each object has been presented alternating (a) a short presentation, with a more descriptive style, to communicate the basic information; (b) a dramaturgy which means a scenario where the object is contextualized in its original environment. The dramatized scenarios are useful to create a magic atmosphere but also to suggest additional information regarding the function of the objects themselves. They are evocative and symbolic and combine 3D graphic and real actors/objects. Digital contents interact also with the real objects.
The purpose of using such type of multimedia inside a museum is the creation of an artificial system that reflects technologically, symbolically, the range of life. It indeed transmits contents that otherwise would not be perceptible, increasing the awareness and understanding of the flow of history by visitors. This flow is no longer far away, neither extraneous, but it becomes part of our present.From a technical point of view, the holographic showcase foresaw precise working steps:
- infrastructure design (the skeleton of the showcase; the choice of the hardware);
- production of 3D and multimedia contents to be harmonized with real objects;
- implementation of a real time rendering platform in VVVV software, able to synchronize audio-video play-out, lights, external devices: along a multi-track timeline all the audio-visual events are organized and managed according to a precise sequence;
- management of Arduino/MIDI controller, to manage real lights along a timeline and to control buttons for language selection;
- light design (seven led lights inside the showcase switch on and off on real objects);
- Interaction and user experience design.
Special attention has been dedicated to ensuring the portability, reusability and sustainability of the equipment (fig.10), as well as ensuring the safety of the display containing the real museum objects. The holographic showcase has been designed to be scalable – depending on the size and quantity of objects displayed inside and the available space in the museum. It has been indeed Implemented in two solutions:
- stand-alone showcase (dimensions 1.50 x 1.40 x 2.50 mt.);
showcase inserted in a projection wall 4 meters wide and 2.50 high.
The Kunàgota sword tells the story of the Avars.
The first story that has been created for our holographic showcase deals with a sword belonging to an Avar warrior, who lived in the mid-seventh century AD. It is the so-called Kunàgota sword, from the place of discovery, and it is preserved at the Hungarian National Museum in Budapest. The sword was discovered in the steppe-like south-east region of Hungary, inside the tomb of a prestigious Avar chief, as part of his funerary good. His tomb was isolated, just excavated into the soil. Originally the golden sheets applied on the sword belonged to a Byzantine object, most probably a casket for jewels. An Avar goldsmith took the golden sheets from the casket and applied them to the sword and the scabbard when the Kunàgota chief died.
The entire installation is built around the real object, physically present inside the showcase. Two configurations of the installation have been implemented and presented in museums:
- the holographic showcase integrated in a wider projection wall, presented in the Hungarian National Museum in Budapest, in the context of the exhibition “Avars Revived” (February – May 2017); the holographic showcase in its stand-alone version, without projection wall, presented in the Allard Pierson Museum in Amsterdam, in the context of the “Crossroads” exhibition (September 2017- April 2018).
In the first configuration the show is organized as follows: in some moments, the showcase surface becomes part of the overall projection displayed on the wall; in others, the projection wall turns black and the showcase becomes the center of attention, hosting the holograms that appear and
move above and around the Kunàgota sword. Thanks to this alternation, the real object is enabled to tell its story – and with it, the life and the spiritual dimension of those who owned it.
During the large projection on the wall, the user receives the basic information necessary to understand and contextualize the story of the Kunàgota sword in a more common style. The real sword is well illuminated, while, on the surrounding wall, short captions, enlargements of details of the sword’s decoration (virtual replicas), images of the other funerary goods coming from the same tomb, and illustrations of Avar warriors are projected, accompanied only by music. Digital contents are not invasive in relation to the real ones. This preliminary phase is called “neutral vision”.
After this “neutral” fruition, the showcase takes control and a dramatization takes place inside, through Pepper’s Ghost illusion. No traditional storytelling is used here, nor a linear narrative strategy; differently, the same object, through its presence, evokes voices, fragments of life, customs, religious and cultural identities. The manufacturing process of the object is represented and so the historical events from which it came from: the looting in battle by Avars of a Byzantine casket covered in golden plates; its spoliation to cover with gold the sword of a dying Avar chief. The figures themselves engraved in the golden plates become the protagonists. The Virtual content overlaps and integrates the real object in the same space of the showcase. This phase represents the “dramatization phase”. The holographic projection, in its minimal essentiality, produces finally the magical effect of the illusion of reality: the hands of two Byzantine women appear, stealthily scanning the contents of their master’s casket; the dust rises from the hooves of horses running during the battle; splashes of blood are produced by games of swords; the hand of the dying father raises his sword and offers it to his son asking decorations with the Byzantine gold just stolen – accompanying him in the afterlife. Voices and whispers of various characters animate the events. The last chapter, “the Farewell and the Blue Sky of Tengri”, contextualizes the sword in the Avar chief’s tomb and accompanies the visitors through the vision of the afterlife, according to the belief of the ancient Asiatic shamans. Here starts the “vision phase”. The only surviving object, which testifies the cosmic beliefs of Avars, is a rudimentary decoration engraved on a small bone casket found in the Avar cemetery of Mokrin, in Serbia, used to contain an ointment. The drawing shows a tree on a hill, The drawing shows a tree on a hill, on the sides both the sun and the moon and various stylized animals. It probably represents the Tree of the World: the roots reach the underworld, the trunk represents the earthly world, while the branches reach the sky. The Turkish people believed that the heavens were ruled by the god Tengri and the dead traveled in this world – the same kingdom in which the shamans ascend to create a connection with the spirits.
Starting from this drawing, the scene has been virtually and carefully reconstructed; it has been also animated, as a theatrical scenography in motion: each single element enters the stage, one by one, to better catch the visitor’s attention on its symbolic and metaphysical meaning. The grave is shown in the ground among the roots of the tree; the warrior chief lies dead, near his sacrificed horses, ready to ride for the last time to heaven. At this point his spirit appears (an actor filmed in green screen and integrated into the virtual scene) and he whispers for the last time, while animals move around him. He evokes his vision, the moment of the burial, and his next and final departure in the blue sky, that is finally accomplished. It is a first person drama, organized in different scenes, each of them introduced by a short title.
In the second configuration presented in Amsterdam, at APM, the holographic showcase is a stand-alone installation. All narrative phases, including the first one, are played inside it.
The animations of the objects, in this kind of holographic solution, cannot be seen from all around the structure. These objects are shown to us as they relive, under our eyes, the moments that have characterized their history over the centuries – moments that have led to shifts, transformations, passages from one owner to another, and so on. From a narrative point of view, it is an inverted perspective, so that historical events are not telling us the full story of the objects (as usually happens in movies), but the objects themselves tell us about the facts that led them here and now. In this way, user does not only learn facts or interpretation of facts, but lives them again in real time; he participates to them emotionally. The phantasmal vision of the holographic event breaks out from the past in all its power, full of its history.
The Mytilene treasure tells the story of a Byzantine family
In the Byzantine and Christian Museum (BCM) in Athens the holographic showcase hosted three Byzantine objects, as part of the Mytilene treasure.
The Mytilene Treasure is a collection of more than 70 precious objects (silver vessels, gold jewelry and coins, a bronze seal) discovered in 1952 in Kratigos, in Mytilene island. They belonged to a wealthy family living there, or maybe in the Asia Minor coast, in the first decades of the VII century, as revealed by the effigies on the coins and the Imperial control stamps impressed into some objects. Sure enough, after the efflorescence under Justinian (VII-VIII c.), the Byzantine empire was exhausted by the assaults of Persians, Avars, Slavs, Arabs. The population was decimated, economy decreased, and urban centers dwindled. A stamp, gold buckles, which were a privilege of high-ranking officials, let us suppose that the treasure belonged to the family of a high official of the Byzantine administration. The sudden danger of an enemy attack could have obliged this family to escape, after having buried the precious objects of the house into the ground, to preserve them from possible looting. Three objects are today exhibited in the holographic showcase: 1) a silver candle holder; 2) a golden bracelet of a child, with a very rare openwork monogram; 3) a silver trulla (a bowl) decorated with marine themes and probably used in bathing.
The holographic showcase has been presented in its stand alone configuration, without the projection wall. The narrative approach is partly similar to the one used for the Kunàgota sword, but more classical. Thanks to the virtual projections on and around the objects, the public can experience fragments of their story, meet characters, and relive historical events. Explanatory moments alternate with dramatizations where objects are contextualized and shown during their daily use by the family. These characters are performed by real actors, shot in slow motion.
In order to shoot these scenes, it has been necessary to print the 3D replica of the objects in real scale, paying attention to make them plausible, resistant and beautiful, similar to the original ones exhibited in the same space. Original artifacts in the showcase enter in the compositing as element of the virtual scene. For instance, the real candle holder appears laying on a virtual table, with a virtual candle fixed and burning on it, close to a virtual character writing a paper. Such kind of integrations seem very powerful. In addition to computer graphics, the language of video art was used too: slow motion, together with the essentiality of the scenes, create an abstract and suspended atmosphere of strong attractiveness. At the end of the story, the burial of the treasure is suggested only by soil falling down in slow motion, from the top to the floor of the showcase, increasing its mass during the animation. Also the soundscape has a crescendo, until the complete coverage and disappearance of the real artifacts (lights on them switch off). Also in this case real lights behave as part of the animated compositing.
The most challenging reconstruction work in this project regards the Kunàgota word. Indeed, according to scientific studies, the golden sheets which decorate the sword, originally come from a Byzantine small casket. During the late-Roman period and Early Byzantine Empire, there were furniture pieces such as caskets and cabinets decorated with carved wood or other inlaid panels like ivory or golden sheets. The construction of the original casket was based on archeological and geometric data, analyzed and interpreted with the support of experts from the National Hungarian Museum (NHM) of Budapest. The missing parts of the casket were based on comparative studies and observation of artifacts displaying similar mythological themes.
In order to realistically portray the casket and its appearance in the holograms, the physical reaction to light of golden and ivory materials needed to be studied and simulated in the rendering engine. Different types of maps and shaders, that simulate the color of the artifacts and the main appearance of the material surface, were used to give the museum objects a realistic look in the rendering.
The same workflow has been used also for the objects belonging to the Mytilene treasure. In that case, since the artifacts were almost intact and did not need any virtual restoration, we only optimized the geometry of the scanned objects and we simulated the different materials (the gold of the bracelet and the silver of the chandelier and the trulla).
Light management was another crucial aspect for ensuring a realistic look of the virtual artifacts and create a proper visual mood. The screenplay required that virtual objects were rendered on a black background, enhancing the ghost effect. Rendering such virtual objects, characterized by reflective materials like silver and gold, without any environment and an appropriate multi-point light setup, would have ended in artificial visual results. Whereas, to create a realistic illumination, we adopted a HDRR (High Dynamic Range Rendering) approach for lighting calculation. HDRR consists of using an HDR environment texture as light source to simulate a greater range of details than standard computer graphic light emitters.
or all the renderings, we used different HDR textures. For those intended to illustrate the artifacts and their details, we used a studio HDR which simulates a photographic high-key lighting setup with an emotionally neutral mood. It is composed by several light: main, secondary, back and fill light with almost the same intensity and fairly soft.
In other cases, where strong emotional lights were needed, we adopted a more intense lighting scheme, changing the balance of highlights and shadow to increase their hardness and obtain a much powerful effect. This kind of mood was especially used in those shots where the virtual objects were integrated in video with actors.
Setup of the showcase
The holographic showcase requires a specific design for hardware setup and concealment, interior illumination, overall efficacy of the Pepper’s Ghost effect.
Visibility of the real objects inside the showcase plays a crucial role. The holographic showcase stands as an opportunity to visualize the symbolic and sensory dimensions of the exhibited objects beyond their formal aspect, but without overshadowing the perception of their physical consistency. For this reason, having a good view of the real artifacts is essential. The holographic projection inside the showcase must have a millimeter accuracy and need to follow precise visual rules, in order to produce the expected tridimensional effects, the superimposition and matching with the real objects. Therefore, the position of real pieces is forced in a precise space, that is the ghost projection plane. In the classical Pepper’s Ghost setup with an oblique mirror, the projection plane is not close to the showcase’s frontal window, because the depth of the showcase is an essential factor to enhance the holographic effect: the deeper the showcase is, the more the ghost seems to float magically in the empty space. Thus, real objects remain closer to the background rather than to the front side. This distance can produce negative feedback on visitors because an optimal perception from the front side is precluded: very small details can be hardly appreciated. Nevertheless, the 3D effect helps a lot in this situation, and overcome such visibility problems because digital contents work exactly on these aspects, animating and magnifying objects details and moving them to the foreground. Anyway, visibilitycan be solved through:
). Anyway, visibilitycan be solved through:
- a good lighting system upon the museum objects inside the showcase;
- avoiding as much as possible reflections on the frontal glass closing the window;
- a transparent background. Instead of closing the back of the structure with an opaque panel, a glass or a plexiglass panel can be used, so that visitors can walk to the back and look at the objects from a very short distance, admiring every detail. From the back side the ghost is not perceivable, because it can be perceived only looking from the front side. Therefore, the holographic showcase can work as a traditional showcase if accessed from the back, and as an “augmented” one if accessed from the front. Of course, in the general environment of the room, the light must not be too intense (a semi-dark condition is ideal), otherwise the light could invade the showcase and the ghost could become not well perceivable in its details and brightness.
Interaction with virtual contents can be a problem in museum environments, especially along the path of visit and especially when it requires complex actions or the understanding of not immediate interfaces. Interaction must:
- Be not an expedient itself, only because it is associated with the concept of technological innovation. Interaction can be an element of technological innovation, but it is not the only one, if misused it becomes a barrier for the public. Technological innovation also lies in the visualization and dramatization of a scenic space through digital technologies;
- Really create added value for the public, that is, be a structural part of the idea of fruition and experience and not an occasional or pretentious element;
- Be supported by the museum staff, able to observe the visitors and help them in case they do not understand how to act. Sometimes this is a problematic aspect in museums because there is not always a dedicated staff inside of them.
The limited interaction can be rewarded with the wonder of narration and vision and with a perfectly calibrated rhythm of the story. Beside, interaction has not to be intended only as a physical action, but also as a cognitive level action, generated by the multiplication of the perceptive layers, as it happens in the case of holograms, where visitors can experience augmented and mixed reality.
Surely enough, a low level of interaction is needed with multimedia located along the path of visit, allowing, for instance, language selection. Even in case of short contents, people are not so patient to wait until the beginning of their language: they want to choose immediately, otherwise the risk is they could skip the content. If, on the contrary, interaction is required in a holographic showcase, it will be most probably used to manipulate virtual objects. In this case specific devices can be used, for instance Leap Motion for gesture based interaction (as in “Keys2Rome” exhibition mentioned in 2.2). User’s engagement is mostly generated by active exploration, rather than by dramatized stories.
Until now, surveys on the public of holographic showcase developing a story, in museums, have not revealed that interactivity, in term of manipulation and selection, is considered as a attractive issue by the public.
Storytelling, cultural transmission and dramaturgy of the object
If we want to transmit meanings, believes, values, personages, events, contexts of usage, we need to create a story around the exhibited object. The integration and interaction between real and virtual contents greatly enhances this experience.
Cultural transmission can be approached in two ways: with a simple description or with a dramaturgy or evocation. In the first case, the style is more traditional, impersonal and “flat”, lacking of emotional power and personal perspectives, while the rhythm is regular. The impression is impartial, of something familiar and already seen in museums, even if applied to something new – like the holographic showcase. The magic power of the Pepper’s Ghost exists in itself, and is not enhanced by the pathos of a story.
In the second case, objects become the occasional points where history “coagulates” and a dramatization takes place. The story is built to create an expectation and brings the visitors in the middle of an emotional experience.
Some basic rules are used in tales. Usually the story develops throughout three phases in order to catch the attention and curiosity of the public: a) order, b) transgression, c) restored order and security. The classic tale contains a dialectic between law and transgression. In every story the moment of transgression is always important, evil is always more fascinating than right. Beside, the tale is developed including different perspectives, different rhythms, and draws up a space-time system in which the viewer/visitor is always in the right place at the right time; such a condition is reassuring and rewarding, allowing visitors to trust again and having the “illusion” to be inside the story. Everything contributes to create an expressive unity: oral performance/recitation, visual mood, soundscape, lighting, rhythms, “atmosphere”. The object is the protagonist, the starting and the ending point of the tale; It also opens visions on several meanings and dimensions.
The environmental context in which the showcase is located, along the museum’s paths, assumes a fundamental importance and must be considered by authors, especially in relation to the duration of the story, the rhythms, the sound, in order to be coherent with the environment of the exhibition. Reciprocal cooperation with museum’s curators is mandatory to make good and plausible stories that can be welcomed by all the public and by the experts.
Some starting questions to the museum curators can help digital creatives to collect information upon which they will draft the story. For instance:
- Where was the object found? Was it the original place in which it was used?
- Which period is the object referred to?
- Which was the context of use of the object?
- Do we know something about the identity of the owners? What about their social status and job?
- How was the life in the place where the object was used at that time? Who lived there? Which job or activities did those people carry on?
- Where was the object preserved/used during its daily life? In a house, in a palace, in a religious environment, in a shop, in a tomb….
- Was the object used in the daily life or was it exhibited by the owners just for its symbolic value?
- Did the object belong to a woman or a man?
- Which historical events can be connected to the object’s life? Is it possible to mention any episode in particular?
- Was the object belonging to a standard typology or was it unique and special? Was it a cultural transmitter for successive generations/cultural patterns?
- Which materials was it made of? What about the manufacturing process?
- Which is the provenience of the object’s materials?
- Which main characters can be associated to the story of the object? Warriors, merchants, politics, artists, goldsmiths, emperors…?
- Is the object connected by a common significance to the others exhibited beside (in the same showcase or in the showcases nearby)? If yes which are the most notable relations?
There are also some useful questions in order to know the curators’ opinion about digital restoration:
- What about digital restoration of this object? Which are the limits according to you?
- Do you agree with virtual reconstruction of shape and decoration? Do you agree with simulation of the original aspect of the materials?
- Do you think that the same criteria of real restoration should be respected also in the case of virtual restoration, (distinction between original and restored/ different layers of visualization) ?
Storytelling becomes definitely the mean to recreate the context, to penetrate the form and the significance of things, opening a vision on the lives of populations and historical processes. The reconstruction of sensory and of symbolic dimensions that are “beyond” the object’s appearance can thus take the visitor in the middle of a lively and powerful experience. On this, we can report that when the museum’s visitors have been asked about what they liked mostly about such a “box of stories”, the most common answers have been indeed “How the story is told”, “The combination of information, lights and sounds”, “The chance of not reading the usual museum panel to understand the object”, “History becoming alive”, “The modularity of the story: it changed every time” (referring to the phases of the story).
Audio-visual grammar to creating the illusion
Given that the holographic effect is an illusion of reality, we need to simulate something that seems to be happening under the eyes of the spectators. Thus, in the conception of a showcase using Pepper’s Ghost effect in a museum, there are specific rules and constraints that need to be respected especially in the visual grammar. The show, in fact, is much closer to a theatrical dramatization rather than to a movie or a generic multimedia animation.
Here are some rules.
- Coexistence of real and virtual
The most powerful impact, in terms of surprise and enhanced experience, derives from the coexistence of the real objects and the virtual projections (ghost) on the stage. There are many substantial conceptual and technical differences between a showcase without or with the real object inside. In the presence of real objects, the designer of virtual contents has much more constraints to respect to create a convincing integration: scale factor, point of view, camera FOV, calibration, matching of lights and colors, shadows coherence, etc.
2) 3D graphic If the “ghost” must be credible and realistic, 3D graphic is needed. Under this condition, the illusion of reality becomes convincing: the stage is a 3D space where events are happening. 2D graphic can be used as well, but in this case the difference in terms of consistency and style will be evident and the Pepper’s Ghost effect will be similar to an illustration.
3) Background of the ghost
Virtual contents must be like apparitions in the empty space; usually a black background in the digital animation is suggested. If the showcase has glasses both on the front and on the back and it is possible to watch through the space from one side to the other, the black background of the animation will not be visible; it will appear as transparent because black means no light reflected in the ghost.
4) The whole image in the frame
The illusion of reality imposes that virtual contents should be entirely contained in the frame, without cuts on their borders. For example, the ghost can detach from the real artifact and then floating in the darkness, or it can emerge from a distance, always in the darkness, coming in foreground and positioning on the real object. It can even enter in the frame from a side but, once it is inside, it must be completely included in the real space of the showcase (as it happens in the stage of a theatre but differently from a cinema movie).
5) Scale factor
In absence of real objects in the showcase, the scale of representation is not so much important, because there are no dimensional references on the stage. On the contrary, when a real object is inside, the scale of the holographic projection needs to be real, correctly proportioned to create a precise correspondence between real and virtual contents, especially if the two interact each other. This does not mean that the virtual content cannot zoom in and out. Images floating backward and forward in the empty space can be useful in the story and can contribute to the magic effect; the same occurs for very small details detaching from the real object and coming in the foreground, to be better perceived. Nevertheless, the original proportion and correspondence with the dimensions of the real artifact must be visually declared at the begin or during the animation, in order to be kept by the visitor.
6) Position of the real object on the stage
One more constraint is the position of the real object on the stage. The reflected images (ghost) lay on a well defined plane of projection, that is unique. Only the perfect overlapping of the real object with this plane can ensure the possibility to perceive the integration of reality and illusion. By means of virtual masks, it is possible to create the illusion that virtual contents are behind the real object. If we have an animation of these virtual images moving behind, the mask must be painted frame by frame with rotoscoping techniques.
7) Still camera
Due to the illusion of reality,a unique camera position (coincident with the visitor’s eyes) is needed and must be kept. Multi-camera is not allowed, because this would immediately bring us in the cinematographic domain rather than in the theatrical one.
8) Camera point of view and FOV
In presence of real objects in the showcase it is important to keep a coherence between the point of view of real and virtual contents, as they must interact each others. During animations, the virtual objects can move and rotate, but a correspondence must be visually suggested in some moments of the animation, as in the case of scale factor. It is also very important to render 3D graphic contents using a camera with a Field of View coherent with the perception of the real object (i.e. 50mm full frame).
9) Camera depth of field
Everything in the scene must be in focus, because of the illusion of reality. For the same reason also motion blur should be avoided. HFR (High Frame Rate) is recommended to escape jittering (48 or 50 or 60 fr/sec, progressive video).
10) Light and colors matching
In presence of real objects inside the showcase, real lights are directed on them to make them visible. It is very important that virtual lights, set for the ghost, match with the real ones in terms of intensity and color, in order to create a plausible compositing.
11) Shadow coherence
If the real object inside the showcase is contextualized in a virtual scene that emerges all around (as in a chroma key scene) it is important that the shadow of the real object is simulated and rendered in the virtual frame, to avoid a floating and incoherent effect.
The magic effect can be enormously favored if the rhythm of the show is correctly calibrated. Essentiality and slowness can be very useful because they are able to “suspend” the events in a metaphysical dimension, thus enhancing the significance and the preciousness of each detail. In the holograms less is more.
13) Sounds Sounds are fundamental to suggest the sensorial dimension beyond the exhibited object, creating an engaging experience for the public. Unfortunately the integration of sounds inside a museum’s space is usually very critical. Immersiveness would require a wide sound projection, in a large space and with a good volume. Wifi earphones could be a good alternative but they have some critical aspects (hygiene, protection, inclusion and social relation of many users at the same time). To reduce the noise, some museums prefer to use sound showers but they are mono and usually cut off many frequencies, compromising the perception of a sophisticated sound design.
14) Sound spatialization
Stereo sound spatialization (sounds entering from one side and moving towards the opposite direction) is possible if the dimension of the holographic showcase is large enough and the space of the sound projection is wide.
Dolby surround 5.1 can increase even more the immersiveness of the experience but, to be enjoyed by visitors, it requires a dedicated secluded space where the installation should be located – as the audio speakers need to be located all around the public, taking the installation as central reference point.
4.8 Overlapping of Real and Virtual.
A mixed reality environment, where real and virtual objects are overlapped and constantly interconnected, is created inside the holographic showcase. Mismatching or shape differences between real and virtual object could tear-down the illusion of reality and nullify the sensory involvement of the visitors brought by the realism of the holograms. An accurate and meticulous work of 3D graphic is mandatory for a successful integration of virtual contents, tailored to this new communicative visual language.
For instance, the 3D graphics produced for our case studies (see Kunàgota sword and Mitylene treasure case studies), is created by mixing and integrating 3D scanning and 3D modeling techniques. The former is a process of capturing, through active (laser scanner) or passive devices (cameras), shape and appearance of physical objects and processing them into digital 3D replicas. The latter, instead, is the process of designing a 3D representation of an object through computer graphics software, in simulated tridimensional environment.
4.9 Virtual replica and virtual reconstructions.
During animations, the artifact shown in the showcase “comes up to life”. When the story starts, images start moving, floating, zooming in and out highlighting details and reliefs. Of course it is not the real object moving but the virtual one – which is perfectly overlapped thanks to an accurate projection mapping. To guarantee this illusion of reality, the correspondence of the point of view and the visual coherence between the real artifact and its digital replica are essential. The 3D models representing the virtual replica of the authentic objects are created using 3D scanning technology. According to the scale of projection, desired level of accuracy and budget, a Structure From Motion (SFM) approach can be chosen to obtain quickly an accurate 3D model reconstruction with color information. SFM is a photogrammetric range imaging technique which allows three-dimensional models to be obtained from two-dimensional image sequences. According to our experience and the most recent literature, this technique can be successfully adopted for digitizing museum objects even in case of artifacts with complex geometry and complex material properties, like absorptivity, reflectivity and scattering . It indeed offers multiple advantages: not destructive impact on materials, short time of acquisition and post-processing of the data, very detailed textured models as output, economical and portable equipment. Moreover, the acquisition can be done directly in museums, even supported by internal staff of photographers.
In order to obtain 3D digital replicas of museum artifacts we adopted the following workflow:
- At the beginning, some precautions are kept during the acquisition, like (a) the positioning of the museum artifacts to be digitized, (b) the number and point of view of the shoots and (c) lighting setup.
- When the acquisition has finished, the entire dataset of images is imported within Agisoft Photoscan, a SFM software, to find correspondences between images and perform the photogrammetric model. During the image alignment, the software automatically finds tie points and camera parameters. Subsequently, dense cloud is computed.
After the optimization of the dense cloud, the mesh is calculated using the Poisson Surface Reconstruction algorithm which allows to reconstruct a triangle mesh from the point cloud. Last step is the parameterization of the model and the generation of textures. As texture generation parameters, a combination that uses the weighted average value of all pixels from all images of the dataset to blend them in a single atlas texture was used to map the model.
Virtual reconstructions play an important role in the Digital Archaeology domain and the evolution of the 3D technology. Theoretical and methodological discussions around the definition of Virtual Archaeology have significantly increased its use and application for research activities and museum applications.
A virtual reconstruction is the act of rebuilding a destroyed or lost heritage (monument, artifact or architecture) to its entirety, at the time of its creation or at a specific moment of its past life. In our case study, we reconstructed not only the original appearance of the museum objects (Kunàgota sword and Mytilene treasure) but also their usage in order narrate their history and recall their evolution through the time.
Reconstructing the past is a complex task which requires a big work of survey, source analysis, discussion and interpretation, before proposing an hypothesis. Furthermore, in order to guarantee scientific consistency and clarity, the virtual reconstruction workflow should follow theoretical principles, as fixed for instance in the London Charter and in the Principle of Seville, whose purpose is supporting interpretation and simulation of the past basing on a scientific approach.
Monitoring and recording users’ behavior
By monitoring and recording users’ behavior, researchers (and the same museum staff) can obtain a crucial understanding of what museum visitors are actually doing once in front of the holographic showcase. It allows to know exactly what people are looking at, what they like or dislike, and how they are using the installation, what they are learning, giving the unique opportunity to make modifications that will guarantee a better usability and satisfaction in the next future. Moreover, knowing the public creates the opportunity to identify ideal areas for further development, giving to researchers and developers the chance to craft something tailored on the final customers. The information acquired by monitoring user experience is an invaluable asset to any digital cultural heritage project to improve the effectiveness and efficacy of multimedia inside museums.
A great variety of methods are available to conduct UX research on public. Observations, direct questionnaires, focus groups, driven-scenarios, simulations and quizzes. Not all of these are really valuable for UX evaluation of digital installations inside museums. Some can be employed singularly, others work better if mixed and confronted with others. Nevertheless, it is not automatic and easy to assess user experience since the latter is subjective, context-dependent and dynamic over time. Moreover, it is conducted by humans so it can be partially influenced by operators. The researcher has thus to select the right methods, target, tools, and choose a specific area of interest such as game or digital application.
According to the V-Must classification of Virtual Museum (VM) , sustainability is one of the key aspects since one of the negative experiences reported in the creation of VMs is the lack of policy and strategy regarding the future persistence and sustainability of applications.
The holographic showcase setup complies with this important aspect since it is fully reusable from different point of view:
- Maintenance. The holographic structure is characterized by a reusable and long lasting setup.
- The hardware is made with aluminum sections bars and panel that can be adapted to every location and can be easily replaced; it is easy to realize and stable in the everyday management.
- The software to manage the entire application, is completely customizable to control different type of media: video, sound, lights, touch and physical interfaces, sensors for environmental monitoring.
- The computing and networking infrastructure is robust and can be continuously monitored and updated via web using a remote controlling software.
- Compatibility. Among the holographic techniques, the Pepper’s Ghost effect is not only able to produce a high quality 3D perception, it is also fully compatible with the conservation needs and the museographic constraints, as it does not damage the museum’s artifact in the showcase. Furthermore, the objects are completely inaccessible and protected within a secure holographic showcase.
- Exchangeability. The structure can be easily mounted and dismounted in a couple of days and can easily travel together with museum objects for temporary exhibitions, as experimented in the CEMEC project (https://cemec-eu.net) where the Kunàgota sword was shown in different location (Budapest, Amsterdam and Bonn). Another advantage of this kind of installation is that it can work also with physical replica of the museum artifacts. The recent evolution of 3D printing technologies allows 3D scanned models of objects, originally digitized to produced virtual animation, and then to be used also for printing physical copies.
- Scalability. The creative workflow and the overall setup is completely adaptable and scalable since it can be adjusted to different museum contents. The holographic structure (dimension, setup, multimedia duration, type of visualization, integrations of projection wall,….) can be thus tailored on diverse situations we can encounter inside museums.
Furthermore, the use of 3D printing opens up new possibilities: museum objects that cannot be included into the holographic showcase, can be whereas printed with 3D machines; this allows the museum pieces to be as well exhibited, in case of impossibility to expose the original one for (a) security reasons, (b) conservative issues or (c) in case of exchanges with other museums for itinerary exhibitions or simultaneous ones.
Holographic showcases inside museums represent a great opportunity to evolve towards a better and harmonic integration between collections and multimedia and to solve more effectively the traditional lack of communication of the museum’s artifacts. A methodological proposal about how to use holograms in the best way in museums has been presented in this paper. A wide set of precautions and expedients have been described that should be taken in consideration to build a proper visual grammar and powerful stories using this very specific technique. The purpose of using holograms is to create an illusion of reality in front of the visitors’ eyes, through the apparition of tridimensional images floating in the empty space and giving the impression to be interacting with our physical world.
Dealing with holographic projections, however, the authors do not refer to real “holography”, as invented by Gabor in 1947 and further developed in Sixties. That optical technology records a visual information in the form of a very fine interweaving of interference fringes using coherent laser light, appropriately projected. Even if in these years the holographic technologies have greatly advanced, they are still immature to be applied to cinematic and multimedia. In fact, the obtained images are not satisfying in terms of resolution, details, colors, field of view, frame rate, also because viewers are not ready yet. For this reason, the most used technique in the field of performative arts and museums is the Pepper’s Ghost, that became very famous in the second half of the XIX century inside theaters. Today digital technology allows artists and researchers to easily use this technique in a scalable way, with new improvements and automations. What is really challenging is the communicative paradigm, the new kind of narration, more oriented towards simulation and dramaturgy. Several examples of installations using Pepper’s Ghost technique have been presented, together with a more detailed case study. The latter, in the last part of the paper, is referred to the European project CEMEC, Connecting Early Medieval European Collections, still in progress (2015-2019), that has brought the authors to experiment this technique in the context of an itinerant exhibition hosted inside important museums in different parts of Europe. Here, the Pepper’s Ghost effect has been used inside showcases, in presence of the real artifacts, in order to produce an effect of mixed reality and a dramaturgy beyond the museum objects. Not only a technical know-how has been acquired during the four years of work, but also many useful data about the impact of such kind of installations on the public and curators, under specific conditions.
The design of a good user experience for holographic showcase is a great challenge both for curators and developers and it passes through the reflection on certain key factors:
- the context of use
- the environmental conditions (silent or noisy, dark or illuminated, secluded or crowded)
- the target age
- the conditions of content accessibility
- the time of usage
- the possibility of intervention on the physical space hosting the installation
- the expectations of curators and visitors.
All these aspects must influence the conceptual structure of any digital application inside museums, especially the ones which play with holography. It is no more feasible in the 21st century to design and develop digital applications without taking into account the final users and their emotional involvement – either common people or experts and professionals. Consequently, the selection and choice of the style of storytelling, the interaction interface, the sense of embodiment, the choice of technology and the appropriate software architecture derive from the above mentioned issues.
As presented in section 1, it is common to find an occasional static touch screen in a corner of a cultural heritage site, which contains pages of dry information about certain artifacts and collections, perhaps accompanied by a couple of images; likely, you can find a site-specific immersive application set aside from the museum visit path rather than a mobile application not really integrating the real museum objects with their digital replica. This is hardly inspiring, particularly in the eyes of common visitors, who are not expert in this specific field or belong to younger generations – for whom technology has become the very centre of their daily lives.
It is obvious that museum managers and curators need to take seriously the interest and retention of these target groups to secure an intense and lively experience of the cultural heritage venues. Incorporating digital technology “within” the physical space is one such way to do this, providing more linked and deep occasions of fruition and understanding.
When visiting museums, three conditions are still missing today:
- No stories are told about each single museum object about the owners, the place of belonging and manufacturing and the period of its persistence. Only a cold list of facts and factual information are often presented in captions or panels inside museums. The practice of storytelling is yet not fully exploited inside cultural venues.
- No relation are revealed and brought to light between objects of the same collection; museums address their exposition ordering the pieces by age or provenance and not by theme or subject.
No contextual description about museum objects is presented, specifically reconstructions about the function, the usage, the environment in which it was included, the atmosphere of that specific period in history and so on. A long tradition of physical paper-based setting reconstructions can be today retractable in museums; however, with the development of technology and the advancement in multimedia techniques, the same reconstructions can be easily translated in 3D and made interactive, allowing more involvement and immersiveness for the final users.
Why emotional involvement is so relevant for experiencing cultural goods in museum spaces? Technology is a mean which help us shaping the sense of involvement in the digitally augmented reality. The public is usually attracted by the technological solutions.
However, the several surveys we carried out observing the people’s behaviors, showed very clearly that the attention towards technology in museums is not long-lasting; it rapidly decreases if the cultural contents are not able to keep it alive and, similarly, if the interaction is difficult and not natural.
For such a reason, it is necessary to rethink technologies at a deeper level of usage. But how? By working on a cognitive and emotional level. The emotional component is fundamental in any learning experience because it is the “irrational” part of human being and it generates motivation – that is the first aspect that pushes people to face technological solutions and lets the learning process to take place.
If we think about the holographic showcase, the processes of creating emotions, raising curiosity and stimulating a sense of wonder do work at the same way. This innovative solution can be really an answer to the newly born necessities of museum curators to listen to visitors of all target groups to foster the Culture.
Reactions towards holographic showcases have been investigated through specific surveys, carried on in museums of different countries. It was interesting to discover that the holographic techniques usually meets the expectations of peoples of different cultures and provenance, and the needs of the curators of different museums. What is undeniable is the attractiveness and usability of such kind of installations, and the naturalness of their integration inside the museum’s space along which the visitor’s experience develops.
Bringing the virtual inside these showcases, interacting with the real artifacts in an harmonic way, creating a dramatization, is the most useful and pleasant experience that we can imagine with digital technologies inside museums, for a large audience of different ages and provenience. The first aspect of innovation that usually is appreciated refers to the different way of presenting the museum pieces: they are not only well illuminated and supported by a close caption describing few features of them (as in traditional showcases); they are rather “duplicated” in their tridimensional shapes, they are “augmented”, virtually reconstructed and restored in some cases; in some others, embraced into an evocative atmosphere; words, images and sounds fluctuate around the objects inside the holographic showcase, highlighting details or adding contextual parts.
The communicative style can vary to meet the expectations of curators and the cultural patterns of the public of different regions, without giving up the main guidelines and criteria that have been presented above. Some curators may encourage an amazing and bizarre approach in the use of Virtual in their museums, other ones prefer a more classical style. The augmented experience of mixed reality offered by holographic showcase is cognitively interactive in terms of intellectual engagement of the visitors, even where no or few choices or actions are required, because different levels of perceptions, interpretation and suggestions are created in the showcases.
Eva Pietroni. She is a researcher at CNR ITABC of Rome; for the project, she gave his contribution in direction, conceptualization, project administration, methodology, formal analysis, visualization, screenwriting and writing-review of this paper.
Daniele Ferdani. He is a researcher at CNR ITABC of Rome; for the project, he gave his contribution in methodology, formal analysis, visualization, and writing of this paper.
Massimiliano Forlani. He is a 3D graphic artist at E.V.O.CA. society; for the project, he gave his contribution in methodology, formal analysis, visualization, screen-writing and writing-review of this paper.
Alfonsina Pagano. She is a research fellow at CNR ITABC of Rome; for the project, she gave his contribution in methodology, formal analysis, UX investigation, screenwriting, writing-review and editing of this paper.
Claudio Rufa. He is a musician and composer but also a programmer and filmmaker at E.V.O.CA. society; for the project, he gave his contribution in methodology, formal analysis, software, and writing of this paper.
This research project is funded by EU Programme “Creative Europe, – Culture, Cooperation Project” (H2020) of EACEA (Education, Audio Visual and Culture Executive Agency). Grant Agreement 2015-1143/001-001 (2015-2019).
We would like to thank our colleagues of the National Hungarian Museum of Budapest, of the Allard Pierson Museum in Amsterdam of the Byzantine and Christian Museum in Athens, and all the people who helped us in collecting the historical contents regarding the Kunàgota sword and the Mytilene treasure and in preparing and conducting the UX evaluation in their venues.
Conflicts of Interest
The authors declare no conflict of interest.