Informatica 40 (2016) 279–284 279 Recommender System for Virtual Assistant Supported Museum Tours Aleš Tavčar Department of Intelligent Systems, Jozef Stefan Institute, Jamova cesta 39, Ljubljana, Slovenia E-mail: ales.tavcar@ijs.si Csaba Antonya and Eugen Valentin Butila Transilvania University of Brasov, 29 Eroilor Blvd, Brasov, Romania E-mail: antonya@unitbv.ro, butila@unitbv.ro Keywords: virtual assistants, language processing, cultural heritage, recommender systems, artificial intelligence Received: July 18, 2016 Personalization technology and services have been widely adopted in many domains, especially on various web platforms. This kind of technology is also finding its way to cultural heritage applications since it can broaden the interest for visiting cultural sights. Recommendation systems suggest to users some specific domains that they might be interested in but were unfamiliar with due to their lesser popularity. In this paper we propose a web service for virtual museum tours that is based on intelligent virtual assistants that can learn user preferences and provide recommendations regarding museum exhibitions. Povzetek: V pričujočem prispevku predstavimo sistem za virtualne vodene oglede muzejev. Temelji na konceptih virtualnih asistentov, kjer uporabnik postavi vprašanje v naravnem jeziku, sistem najde najprimernejši odgovor in ga prikaže v tekstovni ter grafični obliki z uporabo posnetkov muzeja. Sistem vsebuje tudi priporočilni sistem, ki na podlagi zgrajenega modela preferenc uporabnika priporoča oglede predmetov, za katere meni, da bodo uporabniku zanimivi. 1 Introduction Different museums around the world offer to its visitors a wide selection of artwork, sculptures, vases, ancient objects, and other exhibits. This might be overwhelming for the visitors and most of the exhibits they see might not be of a particular interest to them. This might cause them to lose interest in the exhibition and miss artwork or other objects that are more linked to their personal preferences. This problem has been identified as an increasingly significant trend and addressed in various museums [1][2]. To address this, more and more museums are providing access to their collections in virtual environments (applications, web services, etc.) and are interested in providing new ways to deploy appealing exhibitions to improve the understanding and promote cultural heritage also in younger generations [3][4]. Based on the available digital data, applications can be developed that augment the tours for museum visitors. Ho et al. [5] reported a study where Secondary students were involved in the design and construction of a virtual museum. Virtual tours that exploit VR and AR are already available as an addition to the exhibitions in several museums and offer a more interesting and immersive experience for museum visitors. Several mobile applications and devices are already available and can be integrated in custom virtual reality interfaces [6]. Since more and more content is available in digital form and published on the internet, users would benefit from services and applications that provide virtual tours and suggest specific exhibits that users might like more. Monitoring what the user is doing, what he likes and dislikes, thus building a model of his preferences is an important step forward from most of the current application that mainly deliver static content to the user. Personalizing the virtual tour applications and deliver different content for each user is the next step in a modern user oriented application. There are several demanding challenges [7] when designing such systems: lack of content, learning data and users when deploying applications (cold start); data sparsity; rating mechanisms; motivating users to use the system and provide feedback, etc. In this paper we propose a novel online web service that offers virtual tours in museums that are covered with the Google Arts & Culture project (Google Street View). The application provides a natural language interface using a virtual assistant (VA), where users can type questions or requests in the user interface and the VA takes care of identifying the correct answer and displays the corresponding feed from Google Street View. Moreover, a recommender system is integrated in the web service that provides suggestions regarding exhibits targeted for a specific user. We will present a proof-of- concept realization (http://www.projekt- asistent.si/eheritage) of the system and define the experimental setup. The paper is organized as follows: firstly, a review of the related literature in the field of recommender system for virtual museum tours is provided; in Section 3, the 280 Informatica 40 (2016) 279–284 A. Tavčar et al. description of the virtual assistant is provided, along with all the functionalities available to the user. In Section 4, the recommendation algorithms are presented and the experimental results are presented in Section 5. 2 Related work Petridis et al. [8] presented two different virtual reality applications developed for the Herbert Museum and Art Gallery that enrich the user experience of their visitors. The first application uses mobile devices in order to deliver additional information regarding exhibitions to visitors. The second application is a serious game based scenario of the Benedictine monastery that requires solving puzzles and quizzes. In the literature, several research papers were published on recommendation systems for museum visits. Pechenizkiy and Calders in [9] presented a simple user-focused framework for personalizing museum tours. They focused on efficient learning since the system should be able to quickly provide relevant suggestions only after a small set of user preferences. Mathias et al. [10] proposed a new method for personalized museum tour recommendations. Their research tackles the problem of optimizing museum visits according to visitor’s preferences and artwork importance. Through questionnaires they showed that the system clearly improves the satisfaction of visitors who follow the proposed tour. Huang et al. [11] proposed a method for personalized guide recommendation system for museum visitors. The developed system used association rule mining to discover recommendations from both collective and individual visiting behavior. Using this system the visitors avoid the exposure to excessive exhibit information. Using questionnaires they showed that visitors had a positive attitude towards the provided suggestions. Keller and Viennet [12] have evaluated several recommendation methods to suggest relevant content to museum visitors. Their dataset was collected through a mobile interface that displayed audiovisual content to museum visitors. They were able to bookmark (like) certain content which was later regarded as positive data for the learner. Their system is similar to ours in the way they collected and regarded the data since they do not use five star ratings. Although research has already been performed in this area, our approach is different than the one presented in most other papers in the way it interacts with the user and presents the content. 3 Virtual museum guide One of the most important concepts when creating applications that support virtual tours is the usability and user-friendliness of the user interface. We have developed a virtual assistant that understands queries typed in natural language form and provides descriptions or suggestions regarding the searched concepts. Moreover, the system can model the preferences of the users and can provide targeted recommendations regarding exhibitions in a virtual museum. 3.1 Virtual assistant The core of the whole system is a web based virtual assistant that can, to a certain degree, understand the questions typed by the users in the natural language form. The VA is instantiated from an existing web framework that allows the creation of basic virtual assistants. It provides functionalities for a full customization of the User Interface and the creation of the VA knowledge base. Its content is created from part of the exhibits of the British Museum (http://www.britishmuseum.org/). In addition, to create a better user experience, the background page that is linked to each knowledge base entry is a Google Street View representation of the location where the exhibit is situated. Figure 1 shows the initial page that is displayed when the VA is instantiated to the content of the British Museum. The designed application can be regarded as an intelligent interface between the user and the recordings of the exhibits in Google Street View. The VA analyses the question of the user, finds the appropriate answer or description and points the user to the correct location of the object in the museum. Figure 1: The initial view of the British Museum application. The knowledge base entries can be related to a specific room in the museum, a similar group of objects, or to an important exhibit. The VA displays a description of the object, while the page in the background shows a Google Street View representation of the searched object. The user can freely move around and observe the object from different point of views (see Figure 2). Recommender System for Virtual Assistant Supported... Informatica 40 (2016) 279–284 281 Figure 2: Virtual tour of the Rosetta stone exhibition. As already mentioned, the VA analyses the question posed by the user, it then searches in its knowledge base and displays the answer. If the query is regarding a specific exhibit then the answer is a description of the object that can contain links where additional information can be found. Figure 3 displays a close-up of the VA and the answer when the user searched for information regarding the Rosetta Stone. Figure 3: A predefined answer regarding the Rosetta stone. Every exhibit or description is stored in the museum management web portal. The records regarding specific exhibits are marked and are later used by the recommender system to build a model of user preferences. 3.2 Backend interface The virtual assistants provide a separate interface (see Figure 4) for each application that can be used to add new content, modify and remove existing entries from the knowledge base. In addition, it provides a mechanism to define keywords or attributes that are used to describe items for the recommender system (tag “Lastnosti” in Figure 4). The sets of keywords are usually different for each item and are used to characterize the object. Based on them, the preference profile of the user is build. The virtual assistants deliver content based on the IF THEN conditional logic (see tag “Pravilo”). For the Rosetta stone entry the rule is presented below: 𝐼𝐹 𝑟𝑜𝑠𝑒𝑡𝑡𝑎 𝐴𝑁𝐷 𝑠𝑡𝑜𝑛𝑒 THEN {𝑠ℎ𝑜𝑤_𝑒𝑛𝑡𝑟𝑦_𝑛} The system provides a simple mechanism to define rules for each entry. Sentences are composed of stemmed keywords that must be included in the search query and are linked together with the logical operators AND, OR, and NOT. The rule is defined in the “Pravila” section as follows: a comma between keywords denotes OR, a new line is an AND, and the exclamation point a NOT. Each entry in the knowledge base has assigned one or more answers or descriptions of the searched object. The responses can be formatted and links assigned to words. In addition, it is possible to set a link that will be shown in the background when the entry is triggered. In our case, the link points to the Google Street View location of the searched object. Figure 4: The backend form for defining entries in the knowledge base of the virtual assistant. The second important functionality in the backend is the online editor for defining the recommendation algorithm (see Figure 5). It enables the definition of the recommender directly in the backend interface, without the need of redeploying the code. This functionality enables each user to have a different recommendation algorithm, based on their needs. The description below the editor provides guidelines for defining a custom recommendation mechanism. It lists the set of attributes that can be used in the code, as well as the output variables that need to be defined and are required to produce recommendations. The system is initialized with a simple content-based algorithm that builds the preference vector of the user based on the attributes linked to each object. The algorithm uses the Cosine similarity to compute the distance between the user preference vector and other items in the database. The items with the highest similarity are presented to the users as recommendations. 282 Informatica 40 (2016) 279–284 A. Tavčar et al. Figure 5: The online editor form for defining the recommendation algorithm. 4 Recommender system The virtual assistant by itself is a useful interaction tool and can deliver to the user a lot of information regarding museum exhibitions. However, such a simple system is not very different from an online search even though it can provide better targeted information faster. Therefore, we designed a recommendation system linked to the virtual assistant. Due to the specificity of the problem (no implicit rating from users) we had to design a recommendation system that learns user preferences based on implicit data (queries, time spent on each answer, motion, etc.) and item features. We decided to implement two content- based filtering algorithms, each with a different approach for providing recommendations. The first is a simple version that will serve as a baseline. To generate a recommendation, the algorithm does not compare users directly, but instead it tries to find the best recommendations for the specific and active user profile. The algorithm computes the similarities between the user profile and the items in the dataset that the user has not seen yet. The similarities are computed using the cosine similarity (see Eq. 1) on the profile and item vectors. Vectors that have the same orientation are the same, thus have a cosine similarity of 1. The closest is the computed value between the profile and item vectors to 1, the more similar these two vectors are. 𝑆𝑖𝑚𝑖𝑙𝑎𝑟𝑖𝑡𝑦𝑐𝑜𝑠(𝑝𝑟𝑜𝑓𝑖𝑙𝑒, 𝑖𝑡𝑒𝑚) = 𝑝𝑟𝑜𝑓𝑖𝑙𝑒∙ 𝑖𝑡𝑒𝑚 ‖𝑝𝑟𝑜𝑓𝑖𝑙𝑒‖‖𝑖𝑡𝑒𝑚‖ (1) The algorithm then ranks the similarities and identifies the item with the highest similarity for the active user. Figure 6 shows a schematic representation of the basic recommendation mechanism. It uses a vector of keywords or attributes that describes each item, or in our case the exhibit. The top vase in the figure has assigned the vector: 𝑽𝟏 = {𝒗𝒂𝒔𝒆, 𝒄𝒉𝒊𝒏𝒂, 𝒂𝒏𝒄𝒊𝒆𝒏𝒕, 𝒃𝒊𝒈, 𝒄𝒐𝒍𝒐𝒓𝒆𝒅} The vector of the second vase is: 𝑉2 = {𝑣𝑎𝑠𝑒, 𝒈𝒓𝒆𝒆𝒌, 𝑎𝑛𝑐𝑖𝑒𝑛𝑡, 𝑏𝑖𝑔, 𝑐𝑜𝑙𝑜𝑟𝑒𝑑} The vector of the third vase is: 𝑉3 = {𝑣𝑎𝑠𝑒, 𝒈𝒆𝒏𝒆𝒓𝒂𝒍, 𝑎𝑛𝑐𝑖𝑒𝑛𝑡, 𝑏𝑖𝑔, 𝑐𝑜𝑙𝑜𝑟𝑒𝑑} Since User 1 searched for the first two vases, his preference model might state that he likes big, colored and ancient vases. Based on the similarities of the user preference vector and V3, the third vase is recommended to the user. A similar mechanism is adopted for User 2. Figure 6: A simplified content-based recommendation mechanism for the virtual museum tour system. The recommendation mechanism for the second version is based on item-item similarities. This time, the model of the user is not a vector composed of attribute values; instead, the user model contains the list of items he searched for or liked. The algorithm then computes all pairs of similarities (see Eq. 2) between these and new items that the user has not seen yet. 𝑆𝑖𝑚𝑖𝑙𝑎𝑟𝑖𝑡𝑦𝑐𝑜𝑠(𝑖𝑡𝑒𝑚𝑖 , 𝑖𝑡𝑒𝑚𝑗) = 𝑖𝑡𝑒𝑚𝑖∙ 𝑖𝑡𝑒𝑚𝑗 ‖𝑖𝑡𝑒𝑚𝑖‖‖𝑖𝑡𝑒𝑚𝑗‖ (2) The unseen item with the highest similarity among all pairs is recommended to the user. Recommender System for Virtual Assistant Supported... Informatica 40 (2016) 279–284 283 The main idea is to recommend similar items to the objects the user already liked. This way, the vector model does not get overpopulated with attributes and specific traits remain visible. This approach opens up several variations of the algorithm, which will be explored in future work. 5 Experimental results In order to initially verify the quality of recommendations and the accuracy of the system, we designed a small experiment involving three users. Users were invited to use the virtual assistant and ask various questions, depending on their preferences. All the queries from the users were collected and stored separately as the learning set. Next, both versions of the recommendation algorithm were used to create a user model and 10-fold cross validation mechanism was used to validate the performance of the model. In each fold, part of the learning set is used to create the vector of user preferences and the other part is used as the test set for validating the recommendations. Each item is classified using the model of user preferences. If the model would recommend the item from the test set, then the recommender has correctly classified the item. To fully evaluate the model, the receiver operating characteristic (ROC) curve was created. It is a graphical plot that illustrates the performance of a binary classifier as its discrimination threshold is varied. The curve is created by plotting the true positive rate (recall) against the false positive rate (fall-out) at various threshold settings. The closest is the curve to the top left corner, the better the model performs (perfect classification). A point on a diagonal line results from a completely random guess. Based on the performance of the recommender model on the test set, the ROC curve was plotted. Figure 7 shows the ROC curve for the baseline algorithm, where items are directly compared to the vector profile of the user. The AUC measure for this experiment was 0.67. Figure 8 shows the ROC curve obtained from the item comparison recommendation algorithm, where the model of the user is a list of items he searched for. Items to recommend are generated by computing the similarity between items in the user model and unseen objects. The AUC measure for this algorithm was 0.79, which is significantly better as the baseline version. Future versions will build upon this algorithm, since the problem domain seems to fit well with the designed approach. 6 Conclusions This paper presents a web-based application for virtual tours of museums and other locations related to cultural heritage. The system enables the user to visit the exhibitions of museums using a web browser and delivers the content in a user-friendly manner. The core of the application is the virtual assistant that facilitates the communication with the user using natural language processing. The user would type a question to the virtual assistant in the same way he would ask another human. The system processes the query and identifies the most appropriate answer from its knowledge base. The knowledge of the assistant contains the information regarding exhibits, groups of objects and exhibition rooms of the British Museum, obtained from the Google Arts and Culture project. In addition, each answer is linked to a specific location from the Google Street view service so the user can experience a virtual tour of the museum. Moreover, the virtual assistant uses a recommendation engine that suggests specific exhibits based on the preference model of the user. We evaluated the performance of the recommendation system and Figure 7: ROC curve of the baseline recommender algorithm. Figure 8: ROC curve of the item comparison recommender algorithm. 284 Informatica 40 (2016) 279–284 A. Tavčar et al. showed promising results. For future work, additional algorithms could be implemented that would exploit the characteristics of the data. In addition, the virtual tour application could be improved by adding immersive technologies. 7 Acknowledgement The work was co-funded by: the Slovenian Ministry of Education, Science and Sports and European Union’s Horizon 2020 research and innovation programme under grant agreement No. 92103, project eHERITAGE (Expanding the Research and Innovation Capacity in Cultural Heritage Virtual Reality Applications). 8 References [1] Bowen, J. P. and Filippini-Fantoni, S. Personalization and the web from a museum perspective. In: D. Bearman and J. Trant (Eds.), Museums and the Web 2004: Selected Papers from an Int. Conference, Arlington, Virginia, USA, (2004) 63–78 [2] Rutledge, L., Aroyo, L., and Stash, N. Determining user interests about museum collections. In Proceedings of the 15th international Conference on World Wide Web WWW '06. ACM Press, New York, (2006) 855-856 [3] Prensky, M. Digital game-based learning. Computers in Entertainment, vol. 1, no. 1, pp. 21– 21, 2003 [4] Alawad, A., Aljoufie, M., Tiwari, A. and Daghestani L. Beyond Geographical and Cultural Barriers: The Concept of a Virtual Gallery for Arts, Design & Architecture Schools in Saudi Arabia. Social Sciences and Humanities, vol. 3, no. 4, 2015. [5] Hoa, C. M. L., Nelsona, M. E. and Müeller-Wittig, W. Design and implementation of a student- generated virtual museum in a language curriculum to enhance collaborative multimodal meaning- making. Computers & Education, 57 (1), pp. 1083- 1097, 2011 [6] Choudary, O., Charvillat, V., Grigoras, R. and Gurdjos, P. MARCH: mobile augmented reality for cultural heritage. In Proceedings of the 17th ACM International Conference on Multimedia, pp. 1023– 1024, Beijing, China, 2009 [7] Ardissono, L.Kuflik, T. and Petrelli, D. Personalization in cultural heritage: the road travelled and the one ahead. User Modeling and User-Adapted Interaction, vol. 22(1), pp. 73-99, 2012. [8] Petridis, P., Dunwell, I., Liarokapis, F., Constantinou, G., Arnab, S., de Freitas, S. and Hendrix, M. The Herbert Virtual Museum. Journal of Electrical and Computer Engineering, 2013. [9] Pechenizkiy, M. and Calders, T. 2007. A Framework for Guiding the Museum Tours Personalization. Proceedings UM 2007 International Workshop on Personalization Enhanced Access to Cultural Heritage (CHIP). [10] Mathias, M., Moussa, A., Zhou, F., Torres-Moreno, et al. 2014. Optimisation using Natural Language Processing: Personalized Tour Recommendation for Museums. Proceedings of the 2014 Federated Conference on Computer Science and Information Systems. pp. 439-446 [11] Huang, M. Y., Liu, C. H., Lee, C. Y. and Huang, Y. M. 2011. Designing a Personalized Guide Recommendation System to Mitigate Information Overload in Museum Learning. Educational Technology & Society 15 (4). Pp 150-166. [12] Keller, I. and Viennet, E. 2015. Recommender Systems for Museums: Evaluation on a Real Dataset. IMMM 2015 : The Fifth International Conference on Advances in Information Mining and Management.