Category: “Modest corrections”

Studying immersion in virtual tourism

Matthew Tyler-Jones1 Comment

I am less enamoured of the next paper that my expert examiner recommended: Raptis, George E., Fidas, Christos & Avouris, Nikolaos. 2018. Effects of mixed-reality on players’ behaviour and immersion in a cultural tourism game: A cognitive processing perspective. International Journal of Human-Computer Studies 114: 69-79. doi: 10.1016/j.ijhcs.2018.02.003.

This paper describes an attempt to measure attention in a “mixed reality” environment, and hypothesis the impact of such an environment on players of a cultural tourism game. I was hoping that it would be a useful attempt to do the sort of big-budget work I had originally intend to do in my studies – tracking user attention in a cultural-heritage environment with both persistent physical natoms (narrative atoms), and more ephemeral natoms (sound, light and other digital interventions). But although it uses the sort of technology to teach attention that I had hoped to find budget for (in this case Tobii Pro Glasses 2 gaze sampling system) I compares the users reactions to a game that is available PC (i.e. screen based) and also on Microsoft holo-lens. Now Hololens is market by Microsoft as a “mixed reality” system but I am not convinced it is. It is a reasonably sophisticated augmented reality system, but all it does is overlay the user’s environment with an image projected onto the goggle of the headset that they wear. Yes, it models the physical environment reasonably well, so that (when I had a chance to use it) I could “put” a virtual archeological model of a ship on a table then walk around the table to look at the ship from different angles. But I could not interact with the virtual by manipulating the physical. I have seen better “mixed reality” with an x-box and a sandpit.

The game used in this study is a case in point. Holotour, described as “a playful audiovisual three-dimensional virtual tourism application [that] transforms users to travellers, allowing them to see and explore virtual reality environments and experience physical places in space and time without physically travelling there” can be used on a screen or on hololens. It does not involve physical reality at all. It’s a very simple point and click adventure game with the object of collecting hidden objects and adding them to your inventory. The only difference between the on-screen version and the hololens one (as far as I can ascertain from this paper) is whether you use a mouse and cursor to point and click, or or your finger, held up in the field of vision of your goggles. So its not as useful as I had hoped, not tracking visitors’ attention around a physical site.

(This sin’t to say its not a useful paper to somebody – after all, virtual tourism might be all we can do in these covid times.)

I did learn something new (to me) in this paper however a model of cognitive style (or preference – see previous rants about learning styles) called Field Dependence-Independence (FD-I). “FD-I style is a single-
dimension model which measures the ability of an individual to extract information in visually complex scenes.” It may not be as new to me as I think – I recall reading a book, or chapter in a Conceptual Development book, during my first degree (thirty years ago) by (I think) Susan Greenfield about how some people (generally younger and games literate) were better able to follow the story in Hill Street Blues, because that drama was one of the first to feature multiple stories happening on the screen at the same time. I don’t recall her mentioning FD-I but it kind of sounds like the same thing. anyhow “FD individuals tend to prefer a holistic way when processing visual information and have difficulties in identifying details in complex visual scenes. On the other hand, FI individuals tend to prefer an analytical information processing approach, pay attention to details, and easily separate simple structures from the surrounding visual context.” I wonder which I am (from my failure to take in all the info on a game’s screen I am guessing FD.

Heritage Soundscapes

Matthew Tyler-Jones1 Comment

At my viva my external examiner pointed me towards this interesting paper, which she had co-authored – partly, I think, as an example of how I should restructure the discussion of my Chawton experiment in my thesis. But it contains some real gems ( like “the museums studies literature points out the restorative value of an aesthetic experience that is clear of any information acquisition or learning objective and is centred instead on the sensorial experience of being there”) that makes me regret missing it in my literature review: Marshall, M. , PETRELLI, D., DULAKE, N., NOT, E., MARCHESONI, M., TRENTI, E. & PISETTI, A.. 2015. Audio-based narratives for the trenches of World War I : intertwining stories, places and interaction for an evocative experience. International Journal of Human-Computer Studies 27-39.

It’s a case study of a prototype “visitor­ aware personalised multi­point auditory narrative system that automatically plays sounds and stories depending on a combination of features such as physical location, visitor proximity and visitor preferences” Voices from the Trenches for a First World War exhibition at the Museo Storico Italiano della Guerra in Italy. What particularly interest me is that its part of the Mesch project which has some other outcomes which I refer to in my thesis. The paper describes their intent to move away from what they call “the information­ centric approach of cultural heritage.” I am sure a number of my professional colleagues would bridle somewhat at this accusation. After all, did not Tilden tell us in the 50’s that interpretation was more than mere information? But one od the things that my Chawton experiment uncovered was that actually too much “interpretation” turns out to be mere information after all.

The authors summarise previous experiments in responsive soundscapes, such as LISTEN, which “composes a soundscape of music and/or commentaries depending on the detected visitor’s behaviour: visitors that are not close or are moving are classified as unfocussed and for them a soundscape is created, while visitors that are standing still and close to the artwork are classified as focussed and a narrative (e.g. the curator describing the artwork) is played over the headphones.” Though many soundscapes are delivered by headphone, to avoid sound pollution for other visitors, the interesting project SottoVoce is designed around eavesdropping on what other people in our party are listening to. Half the respondents (in groups of two) heard the soundscape from each others phone speakers, while the other half had headphones. “When in loudspeaker mode visitors focussed on what was displayed on the screen of the mobile device and stayed close to the sound source while partners linked via the same audio on their headphones had a more dynamic visit driven by each other’s interest in the exhibits.”

“The ability to convey and evoke emotion is a fundamental aspect of sound” they say, and explain “The affective power of voice and audio storytelling has been recognised as creating a connection to the listener and is even amplified when spoken words are not coupled with the visual capture of the storyteller, creating a sense of intimacy and affective engagement.” An they built their soundscapes using the same sort of mix of music, speech and other sounds that I used (in a limited fashion) at Chawton. Some of the primary source material was recorded to sound more like oral history, with actors reading the words “with palpable emotion” to be more affective. The responsiveness is similar to that of LISTEN, but the “staying still” metric isn’t used, instead a simpler proximity method is used. woven into that soundscape are voice recordings for attentive listening, which is selected by the visitor choosing from a selection of cards. The sound was delivered by loudspeakers but, unlike SottoVoce, not on people’s own devices, rather places around the site. This was what I did for Chawton UNtours too.

The particular challenge with this project was that it was outdoors.The difficulties of maintaining equipment, connecting power and data etc means that most sites resort to delivering via mobile device. But on the other hand: “While engagement in a museum tends to be via prolonged observation, in an outdoor setting multiple senses are stimulated: there is the physical, full­body experience of being there, the sight and the sound of the surroundings, possibly the smell too. The multi-sensory setting places the visitor in direct connection with the heritage and enables engagement at an emotional, affective level rather than at a pure informative level.” (p6) The danger of using a mobile device to deliver interpretation is one I wrote about here, but essentially it stake them out of the where they are, it is the antithesis off presence.

With all this in mind the designers of the project set out five clear principles:

  • To engage at multiple levels, not just cognitive
  • To focus the visitors’ attention on the heritage, not the technology
  • To deal with group dynamics sensibly
  • To be provocative and surprise visitors, but design simple and straightforward interactions
  • To personalize content on the basis of clear conditions

The choice of sound over anything screen-based was an outcome of the second principle. Loudspeakers rather than headphones was also an attempt to focus attention on the heritage: “During a small experiment in a local outdoor heritage site, we observed that audio creates a wider attraction zone where passers­by become aware of the sound source, and a closer engagement zone around the emitting point where one has to stop and listen in order to understand what the voice says.”

So they designed a soundscape that featured music nd sound to attract visitor to a location and then vice recording to hold them there. The narratives are arranged thematically, with different voices (authoritative and intimate) indicating the nature of the content. Quite how the visitor chooses is not really made clear but I expect it is by approaching the voices that most attract them.

The team trialed the idea by observing the visitors behaviour using about 23 minutes of content, but I was disappointed that they did not come up with any solutions to the problems we encounter trying to evaluate the soundscape at The Vyne. It is hard to observe and distinguish between active listening and background listening. The authors seen to assume that if the active listening content is playing, then the partiocilapants are actively listening. The only evidence they have for this is a qualitative questionnaire, which I am not convinced is an accurate measure on engagment. Yes they said they enjoyed an benefitted from the experience, but if they did not know that was what was being tested, what proportion would have even mentioned the soundscape.

Of course they identified a number of challenges, not least fine-tuning the volume to be loud enough to attract attention and yet not so loud to cause discomfort. This is especially true of the different voices, with some by necessity quieter and more intimate. Of course they also predicted issues overs scalability – similar to the ones I planned fro but wasn’t able to properly test at Chawton “how well would such a system work in a busy environment with many groups interacting.”

The challenges of evaluating in-visit digitally enabled heritage interpretation

Matthew Tyler-JonesLeave a comment

I am reading a paper which will help me better present my Chawton study. The paper is brand new: Nikolakopoulou, V. & Koutsabasis, P.. 2019. Methods and Practices for Assessing the User Experience of Interactive Systems for Cultural Heritage. In: Pavlidis, G., ed. Applying Innovative Technologies in Heritage Science. IGI Global, and it is another literature review, but more a sort of meta study of evaluations, called by the authors a “fortiori.”

The methodology of this literature search is exemplary. The authors searched (via Google scholar, ACM digital library, IEEEXplore, Science direct, and Springer link) and found 350 papers, and and in a second search another 150 that looked as though they might be evaluation studies of digital in-visit heritage content. They screened out the majority, for not actually being accessible, or not actually being scientific (I think my thesis would have fallen at this hurdle), leaving just 73. These were screened a second time and a number of duplicate were found and then a number which did not actually write up the evaluation. Leaving just 29 to study in depth.

In summary these papers discussed applications to”explore a virtual or physical space or place” and/or “play a game”. The authors noted the the proportion of games had gone down since a previous survey, which likely corresponds with the time of my change of focus away from a ludic system to something more to do with narrative. They were still roughly 20% of the papers they looked at. Overall, the systems used a variety of digital technologies including 3d game engines, mobile devices, mobile AR, VR, the web, multi-touch displays, location based audio, and physical and kinaesthetic (responding to body movement) interfaces.

There were a lot of studies that looked at usability of the technologies, but the paper points out only using “user satisfaction” as a metric, which is a dangerous trap to fall into. Only a few to a thorough investigation of the User Experience in that way that modern commercial companies test their websites. “A considerable number of systems are evaluated upon learning effects on users” this despite that fact that learning is often not the primary reason for days out (though it may be a validation for the museum), but again few do that in a properly scientific way. There is an interesting paper mentioned (Falvo, P. G. (2015, September). Exhibit design with multimedia and cognitive technologies impact assessment on Luca Giordano, Raphael and the Chapel of the Magi in Palazzo Medici Riccardi, Florence. In 2015 Digital Heritage (Vol. 2, pp. 525-532). IEEE.)) that I may want to take a look at.

The paper highlights the difference between empirical evaluations “in the lab” as it were and in the field. “Field evaluations are contextual, which enhanced the validity of the process and results.” But “recruiting visitors to experience or assess CH content on purpose changes their original purpose of visit, which is
something inherently connected with the visitor experience discussed in several museum and visitor studies […] which suggests a paradox to me, either test random users, and change their reason for visiting, and potentially their responses to the visit; or invite visitors especially for the test and thus make it more like a lab experiment. The reason for visiting a cultural heritage experience is actually part of the experience.

One issue they highlight is that despite being “peer-reviewed and published in important journals and
conferences, it was possible to identify several failures in the quality of reporting on empirical evaluations” in particular reporting on the number of users that participated in the evaluations. I expect that is because, due tho the limitations of evaluation in cultural heritage sites, the time taken to observe a visit, the number of participants is statistically small – small enough to put into question wether the studies are “empirical” at all I know that after a week on site I had less than ten samples. And so I tried very hard not to present what I learned as empirical evidence. Indeed maybe even too hard. That bit need sa re-write post viva.

The conclusion is a reiteration that digital interpretation evaluation should involve more cultural heritage professionals and field studies. It point out that there is more work in newer methodologies “like physical computing and tangible user interfaces” (objects that are the interfaceIt also highlights ongoing issues of a lack of systematic approaches to evaluation, organised “in distinct identifiable catagories” which I imagine would make doing meta-studies like this more meaningful. Also, “Aspects of cultural value, awareness, and presence could only be recognized in very few empirical evaluations. Evaluation studies that consider more qualitative dimensions and more related to general purposes of CH like learning effectiveness have been increased, and they are usually accompanied by comparative evaluations. Comparative evaluations represent a small number compared to the overall number of studies reviewed.”

Personalisation redux

Matthew Tyler-JonesLeave a comment

My external examiner at my viva was Daniella Petrelli, an academic in the field of HCI (Human Computer Interfaces) who I had referenced a few time in my thesis particularly after discovering she was behind a platform to help curators to write the sort of content I had created for Chawton. I found that work too late, after completing the Chawton experiment. Among the “modest” changes that Daniella recommended in my viva is a considerable amount of further reading, including this paper, which to my shame I had not discovered in my literature search, and which would have saved me doing a whole lot of reading and improved my PhD! (Which is of course what a viva is for 🙂)

The paper (ARDISSONO, L., KUFIK, T. & PETRELLI, D.. 2012. Personalization in cultural heritage: the road travelled and the one ahead. User Modeling and UserAdapted Interaction 73 – 99.) Is an incredible useful survey and summary of personalisation techniques employed in cultural heritage up to 2012. I am pretty sure it came out of somebody else’s own PhD literature search. It is very biased of course towards computer enabled personalisation (because it comes out of that discipline) but it looks at 37 separate projects and charts a history of personalisation since the early 90’s. ” when some of the adaptive hypermedia systems looked at museum content (AlFresco (Stock et al., 1993), ILEX (Oberlander et al., 1998)) and tourism (AVANTI (Fink et al., 1998)) as possible application domains” (p7) These early experiments included “a human-machine dialogue on Italian art and combined natural language processing with a hypermedia system connected to a videodisc”, and “automatically generated hypertext pages with text and images taken from material harvested from existing catalogues and transcriptions of conversations with the curator”.

The authors chart the development of web–based interfaces that don’t rely on kiosks or laserdiscs, though WAP (Wireless Application Protocol – which delivered a very basic web service to “dumb” mobile phones) to multi-platform technologies that worked on computers and Personal Digital Assistants. They note two parallel streams of research – “Hypermedia and Virtual Reality threads” adapting the content to the user, and presenting overlays on maps etc. The appearance of PDA’s so personalisation becoming more content aware, with plug in GPS units, but the difficulty of tracking people indoors led to experiments in motion sensing, Petrelli herself was involved in Hyperaudio, wherein “standing for a long time in front of an exhibit indicated interest while leaving before the audio presentation was over was considered as displaying the opposite attitude” (I might need to dig that 2005 paper out, and 1999 paper on HIPS).

There is also an excellent section on the methodologies used for representing information, modelling the user, and matching users and content. When it talks about information for example, it suggests different Hypermedia methodologies, including:

  • “A simple list of objects representing the exhibition as “visit paths” (Kubadji (Bohnert et al., 2008));
  • Text descriptions and “bag of words” representations of the exhibits on display9 (Kubadji and PIL);
  • “Bag of concepts” representations generated by natural language processing techniques to support a concept-based item classification (CHAT (de Gemmis et al., 2008)); and
  • Collection-specific ontologies for the multi-classification of artworks, such as location and culture, and multi-faceted search (Delphi toolkit (Schmitz and Black, 2008))”

The paper also articulates the challenges to heritage institutions wanting to personalise their user experience, including a plethora of technologies and not standards yet reaching critical mass. Tracking users outside (before and after) their heritage experience is another challenge – membership organisations like the National Trust have a distinct advantage in this regard, but have spend most of the decade since this paper was written getting there. Of course heritage visits are made as part fo a group, more than by individuals, and personalisation by definition is about individuals – yet in most of the projects in this survey, the social aspect was not considered. The paper also acknowledges that most of these projects have involved screen based hypermedia while augmented reality and and physical interaction technologies have developed alongside.

Evaluation is a challenge too. In a section on evaluation which I only wish I had read before my project, the paper foreshadows the all the difficulties I encountered. But also says “a good personalization should go unnoticed by the user who becomes aware of it only when something goes wrong.” (p 25) It is reassuring too that the paper concludes “the real issue is to support realistic scenarios – real visitors and users, as individuals and groups in daily interactions with cultural heritage. It is time to collaborate more closely with cultural heritage researchers and institutions” (p27) which is (kind of) what I did. I had better quote that in my corrections and make it look as though I was inspired by this paper all along 🙂.