INFORMACIJSKA DRUŽBA Zbornik 21. mednarodne multikonference - IS 2018 INFORMATION SOCIETY Proceedings of the 21st International Multiconference - IS 2018 Slovenska konferenca o umetni inteligenci Slovenian Conference on Artificial Intelligence Kognitivna znanost Cognitive Science Odkrivanje znanja in podatkovna skladišča - SiKDD Data Mining and Data Warehouses - SiKDD Mednarodna konferenca o visokozmogljivi optimizaciji v industriji, HPOI International Conference on High-Performance Optimization in Industry, HPOI Delavnica AS-IT-IC AS-IT-IC Workshop Soočanje z demografskimi izzivi Facing Demographic Challenges Sodelovanje, programska oprema in storitve v informacijski družbi Collaboration, Software and Services in Information Society Delavnica za elektronsko in mobilno zdravje ter pametna mesta Workshop Electronic and Mobile Health and Smart Cities Uredili / Edited by Thomas Bartz-Beielstein, Bogdan Filipič, Matjaž Gams, Marko Grobelnik, Marjan Heričko, Mitja Luštrek, Janez Malačič, Olga Markič, Dunja Mladenić, Rok Piltaver, Toma Strle, Aleš Tavčar, Jernej Zupančič http://is.ijs.si 8.–12. oktober 2018 / 8–12 October 2018 Ljubljana, Slovenia INFORMACIJSKA DRUŽBA Zbornik 21. mednarodne multikonference – IS 2018 INFORMATION SOCIETY Proceedings of the 21st International Multiconference – IS2018 Slovenska konferenca o umetni inteligenci Slovenian Conference on Artificial Intelligence Kognitivna znanost Cognitive Science Odkrivanje znanja in podatkovna skladišča - SiKDD Data Mining and Data Warehouses - SiKDD Mednarodna konferenca o visokozmogljivi optimizaciji v industriji, HPOI International Conference on High-Performance Optimization in Industry, HPOI Delavnica AS-IT-IC AS-IT-IC Workshop Soočanje z demografskimi izzivi Facing Demographic Challenges Sodelovanje, programska oprema in storitve v informacijski družbi Collaboration, Software and Services in Information Society Delavnica za elektronsko in mobilno zdravje ter pametna mesta Workshop Electronic and Mobile Health and Smart Cities Uredili / Edited by Thomas Bartz-Beielstein, Bogdan Filipič, Matjaž Gams, Marko Grobelnik, Marjan Heričko, Mitja Luštrek, Janez Malačič, Olga Markič, Dunja Mladenić, Rok Piltaver, Toma Strle, Aleš Tavčar, Jernej Zupančič http://is.ijs.si 8.–12. oktober 2018 / 8–12 October 2018 Ljubljana, Slovenia Uredniki: Thomas Bartz-Beielstein, Institute for Data Science, Engineering and Analytics, TH Köln Bogdan Filipič, Odsek za inteligentne sisteme, Institut »Jožef Stefan«, Ljubljana Matjaž Gams, Odsek za inteligentne sisteme, Institut »Jožef Stefan«, Ljubljana Marko Grobelnik, Laboratorij za umetno inteligenco, Institut »Jožef Stefan«, Ljubljana Marjan Heričko, University of Maribor, Faculty of Electrical Engineering and Computer Science Mitja Luštrek, Odsek za inteligentne sisteme, Institut »Jožef Stefan«, Ljubljana Janez Malačič, Katedra za matematiko, statistiko in operacijske raziskave, Ekonomska fakulteta, Ljubljana Olga Markič, Univerza v Ljubljani, Filozofska fakulteta Dunja Mladenić, Laboratorij za umetno inteligenco, Institut »Jožef Stefan«, Ljubljana Rok Piltaver, Celtra, d. o. o. in Odsek za inteligentne sisteme, Institut »Jožef Stefan«, Ljubljana Toma Strle, Univerza v Ljubljani, Pedagoška fakulteta, Center za kognitivno znanost Aleš Tavčar, Odsek za inteligentne sisteme, Institut »Jožef Stefan«, Ljubljana Jernej Zupančič, Odsek za inteligentne sisteme, Institut »Jožef Stefan«, Ljubljana Založnik: Institut »Jožef Stefan«, Ljubljana Priprava zbornika: Mitja Lasič, Vesna Lasič, Lana Zemljak Oblikovanje naslovnice: Vesna Lasič Dostop do e-publikacije: http://library.ijs.si/Stacks/Proceedings/InformationSociety Ljubljana, oktober 2018 Informacijska družba ISSN 2630-371X Kataložni zapis o publikaciji (CIP) pripravili v Narodni in univerzitetni knjižnici v Ljubljani COBISS.SI-ID=297704192 ISBN 978-961-264-134-4 (pdf) PREDGOVOR MULTIKONFERENCI INFORMACIJSKA DRUŽBA 2018 Multikonferenca Informacijska družba (http://is.ijs.si) je z enaindvajseto zaporedno prireditvijo osrednji srednjeevropski dogodek na področju informacijske družbe, računalništva in informatike. Letošnja prireditev se ponovno odvija na več lokacijah, osrednji dogodki pa so na Institutu »Jožef Stefan«. Informacijska družba, znanje in umetna inteligenca so še naprej nosilni koncepti človeške civilizacije. Se bo neverjetna rast nadaljevala in nas ponesla v novo civilizacijsko obdobje ali pa se bo rast upočasnila in začela stagnirati? Bosta IKT in zlasti umetna inteligenca omogočila nadaljnji razcvet civilizacije ali pa bodo demografske, družbene, medčloveške in okoljske težave povzročile zadušitev rasti? Čedalje več pokazateljev kaže v oba ekstrema – da prehajamo v naslednje civilizacijsko obdobje, hkrati pa so notranji in zunanji konflikti sodobne družbe čedalje težje obvladljivi. Letos smo v multikonferenco povezali 11 odličnih neodvisnih konferenc. Predstavljenih bo 215 predstavitev, povzetkov in referatov v okviru samostojnih konferenc in delavnic. Prireditev bodo spremljale okrogle mize in razprave ter posebni dogodki, kot je svečana podelitev nagrad. Izbrani prispevki bodo izšli tudi v posebni številki revije Informatica, ki se ponaša z 42-letno tradicijo odlične znanstvene revije. Multikonferenco Informacijska družba 2018 sestavljajo naslednje samostojne konference:  Slovenska konferenca o umetni inteligenci  Kognitivna znanost  Odkrivanje znanja in podatkovna skladišča – SiKDD  Mednarodna konferenca o visokozmogljivi optimizaciji v industriji, HPOI  Delavnica AS-IT-IC  Soočanje z demografskimi izzivi  Sodelovanje, programska oprema in storitve v informacijski družbi  Delavnica za elektronsko in mobilno zdravje ter pametna mesta  Vzgoja in izobraževanje v informacijski družbi  5. študentska računalniška konferenca  Mednarodna konferenca o prenosu tehnologij (ITTC) Soorganizatorji in podporniki konference so različne raziskovalne institucije in združenja, med njimi tudi ACM Slovenija, Slovensko društvo za umetno inteligenco (SLAIS), Slovensko društvo za kognitivne znanosti (DKZ) in druga slovenska nacionalna akademija, Inženirska akademija Slovenije (IAS). V imenu organizatorjev konference se zahvaljujemo združenjem in institucijam, še posebej pa udeležencem za njihove dragocene prispevke in priložnost, da z nami delijo svoje izkušnje o informacijski družbi. Zahvaljujemo se tudi recenzentom za njihovo pomoč pri recenziranju. V letu 2018 bomo šestič podelili nagrado za življenjske dosežke v čast Donalda Michieja in Alana Turinga. Nagrado Michie-Turing za izjemen življenjski prispevek k razvoju in promociji informacijske družbe bo prejel prof. dr. Saša Divjak. Priznanje za dosežek leta bo pripadlo doc. dr. Marinki Žitnik. Že sedmič podeljujemo nagradi »informacijska limona« in »informacijska jagoda« za najbolj (ne)uspešne poteze v zvezi z informacijsko družbo. Limono letos prejme padanje državnih sredstev za raziskovalno dejavnost, jagodo pa Yaskawina tovarna robotov v Kočevju. Čestitke nagrajencem! Mojca Ciglarič, predsednik programskega odbora Matjaž Gams, predsednik organizacijskega odbora i FOREWORD - INFORMATION SOCIETY 2018 In its 21st year, the Information Society Multiconference (http://is.ijs.si) remains one of the leading conferences in Central Europe devoted to information society, computer science and informatics. In 2018, it is organized at various locations, with the main events taking place at the Jožef Stefan Institute. Information society, knowledge and artificial intelligence continue to represent the central pillars of human civilization. Will the pace of progress of information society, knowledge and artificial intelligence continue, thus enabling unseen progress of human civilization, or will the progress stall and even stagnate? Will ICT and AI continue to foster human progress, or will the growth of human, demographic, social and environmental problems stall global progress? Both extremes seem to be playing out to a certain degree – we seem to be transitioning into the next civilization period, while the internal and external conflicts of the contemporary society seem to be on the rise. The Multiconference runs in parallel sessions with 215 presentations of scientific papers at eleven conferences, many round tables, workshops and award ceremonies. Selected papers will be published in the Informatica journal, which boasts of its 42-year tradition of excellent research publishing. The Information Society 2018 Multiconference consists of the following conferences:  Slovenian Conference on Artificial Intelligence  Cognitive Science  Data Mining and Data Warehouses - SiKDD  International Conference on High-Performance Optimization in Industry, HPOI  AS-IT-IC Workshop  Facing demographic challenges  Collaboration, Software and Services in Information Society  Workshop Electronic and Mobile Health and Smart Cities  Education in Information Society  5th Student Computer Science Research Conference  International Technology Transfer Conference (ITTC) The Multiconference is co-organized and supported by several major research institutions and societies, among them ACM Slovenia, i.e. the Slovenian chapter of the ACM, Slovenian Artificial Intelligence Society (SLAIS), Slovenian Society for Cognitive Sciences (DKZ) and the second national engineering academy, the Slovenian Engineering Academy (IAS). On behalf of the conference organizers, we thank all the societies and institutions, and particularly all the participants for their valuable contribution and their interest in this event, and the reviewers for their thorough reviews. For the sixth year, the award for life-long outstanding contributions will be presented in memory of Donald Michie and Alan Turing. The Michie-Turing award will be given to Prof. Saša Divjak for his life-long outstanding contribution to the development and promotion of information society in our country. In addition, an award for current achievements will be given to Assist. Prof. Marinka Žitnik. The information lemon goes to decreased national funding of research. The information strawberry is awarded to the Yaskawa robot factory in Kočevje. Congratulations! Mojca Ciglarič, Programme Committee Chair Matjaž Gams, Organizing Committee Chair ii KONFERENČNI ODBORI CONFERENCE COMMITTEES International Programme Committee Organizing Committee Vladimir Bajic, South Africa Matjaž Gams, chair Heiner Benking, Germany Mitja Luštrek Se Woo Cheon, South Korea Lana Zemljak Howie Firth, UK Vesna Koricki Olga Fomichova, Russia Mitja Lasič Vladimir Fomichov, Russia Blaž Mahnič Vesna Hljuz Dobric, Croatia Jani Bizjak Alfred Inselberg, Israel Tine Kolenik Jay Liebowitz, USA Huan Liu, Singapore Henz Martin, Germany Marcin Paprzycki, USA Karl Pribram, USA Claude Sammut, Australia Jiri Wiedermann, Czech Republic Xindong Wu, USA Yiming Ye, USA Ning Zhong, USA Wray Buntine, Australia Bezalel Gavish, USA Gal A. Kaminka, Israel Mike Bain, Australia Michela Milano, Italy Derong Liu, USA Toby Walsh, Australia Programme Committee Franc Solina, co-chair Matjaž Gams Vladislav Rajkovič Viljan Mahnič, co-chair Marko Grobelnik Grega Repovš Cene Bavec, co-chair Nikola Guid Ivan Rozman Tomaž Kalin, co-chair Marjan Heričko Niko Schlamberger Jozsef Györkös, co-chair Borka Jerman Blažič Džonova Stanko Strmčnik Tadej Bajd Gorazd Kandus Jurij Šilc Jaroslav Berce Urban Kordeš Jurij Tasič Mojca Bernik Marjan Krisper Denis Trček Marko Bohanec Andrej Kuščer Andrej Ule Ivan Bratko Jadran Lenarčič Tanja Urbančič Andrej Brodnik Borut Likar Boštjan Vilfan Dušan Caf Mitja Luštrek Baldomir Zajc Saša Divjak Janez Malačič Blaž Zupan Tomaž Erjavec Olga Markič Boris Žemva Bogdan Filipič Dunja Mladenič Leon Žlajpah Andrej Gams Franc Novak iii iv KAZALO / TABLE OF CONTENTS Slovenska konferenca o umetni inteligenci / Slovenian Conference on Artificial Intelligence ................ 1 PREDGOVOR / FOREWORD ....................................................................................................................... 3 PROGRAMSKI ODBORI / PROGRAMME COMMITTEES ........................................................................... 4 Monitoring Bumblebee Daily Activities Using Microphones / Gradišek Anton, Cheron Nicolas, Heise David, Galen Candace, Grad Janez ......................................................................................................... 5 Reconstructing PPG Signal from Video Recordings / Slapničar Gašper, Andova Andrejaana, Dovgan Erik, Luštrek Mitja ..................................................................................................................................... 9 The Influence of Communication Structure on Performance of an Agent-based Distributed Control System / Malus Andreja, Vrabič Rok, Kozjek Dominik, Butala Peter, Gams Matjaž ..........................................13 Complex Decision Rules in DEX Methodology: jRule Algorithm and Performance Analysis / Kikaj Adem, Bohanec Marko ......................................................................................................................................17 Sensitivity Analysis of Computational Models that Dissolve the Fermi Paradox / Nastran Jurij, Šircelj Beno, Bokal Drago, Gams Matjaž ..........................................................................................................21 Context-aware Stress Detection in the AWARE Framework / Trajanoska Marija, Katrašnik Marko, Lukan Junoš, Gjoreski Martin, Gjoreski Hristijan, Luštrek Mitja ........................................................................25 BRISCOLA: Being Resourceful In Stacking Cards - Opponent, Lament Away! / Janko Vito, Mlakar Nejc, Bizjak Jani ..............................................................................................................................................29 Emotion Recognition Using Audio Speech Signal / Smerkol Maj, Luštrek Mitja ........................................33 Improvement of AI through Deep Understanding / Bizjak Jani, Gams Matjaž ...........................................37 Assessment and Prediction of Auxiliary Carabid Species in Agricultural Fields / Debeljak Marko, Kuzmanovski Vladimir, Džeroski Sašo, Tosser Veronique, Trajanov Aneta ..........................................41 Taxonomies for Knowledge Representation of Sustainable Food Systems in Europe / Trajanov Aneta, Dergan Tanja,Debeljak Marko ................................................................................................................45 Uporaba povezave kalkulacijskega simulacijskega modela z analizo tveganja pri podpori odločanja v kmetijstvu / Dergan Tanja, Trajanov Aneta, Debeljak Marko ................................................................49 Hierarchical Multi-label Classification for Activity Recognition / Reščič Nina, Luštrek Mitja ......................53 Aiding the Task of Process-Based Modeling with ProBMoTViz / Peev Gjorgi, Simidjievski Nikola, Džeroski Sašo ........................................................................................................................................57 Evaluation and Prospects of Semi-automatic Video Distance Measurement in Ski Jumping / Kukar Matjaž .....................................................................................................................................................62 Opis zmagovalne rešitve na mednarodnem tekmovanju o napovedovanju izida točk v tenisu / Mlakar Miha, Sobel Scott ...................................................................................................................................66 Kognitivna znanost / Cognitive Science ......................................................................................................71 PREDGOVOR / FOREWORD .....................................................................................................................73 PROGRAMSKI ODBORI / PROGRAMME COMMITTEES .........................................................................74 Razumevanje odraslih partnerskih odnosov v luči spolne zlorabe v otroštvu / Jerebic Sara .....................75 Do healthy participants make advantageous decisions? Evidence from the Iowa Gambling Task / Vukčević Marković Maša ........................................................................................................................79 Palaeolithic toolmaking and the evolution of cognition and language / Gabrić Petar, Banda Marko, Karavanić Ivor .........................................................................................................................................82 Psihonevrobiološka razvojna perspektiva razumevanja nasilja / Simonič Barbara ...................................87 Emergence of Visual Consciousness in ADHD Children / Zupanič Katja, Y. Fakhreddine Rawan, Nadásdy Zoltán ......................................................................................................................................91 European Legal Regulation of Self-learning Systems / Zapušek Tjaša .....................................................93 Legal Practitioners’ Views on Neuroscientific Evidence in Court Proceedings / Siserman-Gray Cristina .99 From AI to our future I: AI and the Drake equation / Gams Matjaž .........................................................103 The Quest for Understanding: Helping People with PIMD to Communicate with their Caregivers / Cigale Matej, Luštrek Mitja, Gams Matjaž, Krämer Torsten, Engelhardt Meike, Zentel Peter ........................107 Kognitivna raven kot del relacijske družinske terapije pri predelovanju spolnega nasilja / Repič Slavič Tanja .....................................................................................................................................................111 Medgeneracijski prenos nasilja / Gostečnik Christian ..............................................................................116 The other side of neurotechnologies / Elersič Kristian .............................................................................119 Braincrafting: why playing video games is good for you / Fatur Katarina ................................................122 In search of the authentic self: explaining phenomenology of authenticity / Urbančič Maša ...................126 Exploring Features of Cognitive Science as Natural Epistemology / Kolenik Tine ..................................130 v Učinki telesne poze na prag termične bolečine za vroče / Novaković Dušanka, Puh Urška, Vidmar Gaj, Moharić Metka, Bresjanac Mara ...........................................................................................................134 The effect of distractors in lateralized change detection working memory task / Slana Ozimič Anka .....137 "Zmota luči v hladilniku in značilnosti opazovanja fenomenov z obrobja zavesti" / Kordeš Urban, Lipič Viktorija .................................................................................................................................................141 Zemljevid prepričanj – od propozicije do doživljanja / Klauser Florian, Kordeš Urban ............................145 Odkrivanje znanja in podatkovna skladišča - SiKDD / Data Mining and Data Warehouses - SiKDD ...149 PREDGOVOR / FOREWORD ...................................................................................................................151 PROGRAMSKI ODBORI / PROGRAMME COMMITTEES .......................................................................152 Preparing Multi-Modal Data for Natural Language Processing / Novak Erik, Urbančič Jasna, Jenko Miha ......................................................................................................................................................153 Towards Smart Statistics in Labour Market Domain / Novalija Inna, Grobelnik Marko ............................157 Relation Tracker - Tracking the Main Entities and Their Relations Through Time / Massri M. Besher, Novalija Inna, Grobelnik Marko ............................................................................................................161 Cross-Lingual Categorization of News Articles / Novak Blaž ...................................................................165 Transporation Mode Detection Using Random Forest / Urbančič Jasna, Pejović Veljko, Mladenić Dunja ....................................................................................................................................................169 FSADA, an Anomaly Detection Approach / Jovanoski Viktor, Rupnik Jan ..............................................173 Predicting Customers at Risk With Machine Learning / Gojo David, Dujič Darko ...................................177 Text Mining Medline to Support Public Health / Pita Costa Joao, Stopar Luka, Fuart Flavio, Grobelnik Marko, Santanam Raghu, Sun Chenlu, Carlin Paul, Black Michaela, Wallace Jonathan ....................181 Crop Classification Using PerceptiveSentinel / Koprivec Filip, Čerin Matej, Kenda Klemen ...................185 Towards a Semantic Repository of Data Mining and Machine Learning Datasets / Kostovska Ana, Džeroski Sašo, Panov Panče ...............................................................................................................189 Towards a Semantic Store of Data Mining Models and Experiments / Tolovski Ilin, Džeroski Sašo, Panov Panče....................................................................................................................................................193 A Graph-Based Prediction Model With Applications / London András, Németh József, Krész Miklós ....197 Mednarodna konferenca o visokozmogljivi optimizaciji v industriji, HPOI 2018 / International Conference on High-Performance Optimization in Industry, HPOI 2018 ...........................................203 PREDGOVOR / FOREWORD ...................................................................................................................205 PROGRAMSKI ODBORI / PROGRAMME COMMITTEES .......................................................................207 On Using Real-World Problems for Benchmarking Multiobjective Optimization Algorithms / Tušar Tea 209 Bridging Theory and Practice Through Modular Graphical User Interfaces / Rehbach Frederik, Stork Jörg, Bartz-Beielstein Thomas .............................................................................................................213 Expensive Optimisation Exemplified by ECG Simulator Parameter Tuning / Breiderhoff Beate, Naujoks Boris, Bartz-Beielstein Thomas, Filipič Bogdan ...................................................................................217 A Hybrid Optimization Strategy with Low Resource Usage for Large Scale Multi-objective Problems / Monteiro Wellington Rodrigo, Reynoso-Meza Gilberto ........................................................................221 Electric Vehicle Routing Problem: State of the Art / Serrar Jihane, Ellaia Rachid, Talbi El-Ghazali .......225 Optimization of End-to-End Deep Learning for Obtaining Human-Like Driving Models / Dovgan Erik, Sodnik Jaka, Filipič Bogdan .................................................................................................................229 A Bi-Objective Maintenance-Routing Problem: Service Level Consideration / Rahimi Mohammad, Talbi El-Ghazali .............................................................................................................................................233 Study on Reducing Turn-Around Time of Multi-Objective Evolutionary Algorithm on an Industrial Problem / Fukumoto Hiroaki, Oyama Akira .......................................................................................................237 Evolution of Electric Motor Design Approaches: The Domel Case / Papa Gregor, Petelin Gašper, Korošec Peter .......................................................................................................................................241 Model-Based Multiobjective Optimization of Elevator Group Control / Vodopija Aljoša, Stork Jörg, Bartz- Beielstein Thomas, Filipič Bogdan .......................................................................................................245 From a Production Scheduling Simulation to a Digital Twin / Papa Gregor, Korošec Peter ....................249 Delavnica AS-IT-IC / AS-IT-IC Workshop ....................................................................................................253 PREDGOVOR / FOREWORD ...................................................................................................................255 PROGRAMSKI ODBORI / PROGRAMME COMMITTEES .......................................................................256 Austrian-Slovenian Intelligent Tourist Information Center: Project Progress Report / Zupančič Jernej, Gams Matjaž ........................................................................................................................................257 Tourism Related ICT Tools: a Review / Grasselli Gregor, Zupančič Jernej .............................................262 vi AS-IT-IC Databases / Zupančič Jernej, Tazl Oliver August, Mahnič Blaž, Grasselli Gregor ...................266 Content API - A Cloud-based Data Source for the AS-IT-IC Platform / Tazl Oliver August, Wotawa Franz.....................................................................................................................................................270 e-Tourist 2.0: an Adaptation of the e-Tourist for the AS-IT-IC Project / Grasselli Gregor ........................272 Planning-based Security Testing for Chatbots / Bozic Josip, Wotawa Franz ..........................................275 Soočanje z demografskimi izzivi / Facing Demographic Challenges......................................................279 PREDGOVOR / FOREWORD ...................................................................................................................281 PROGRAMSKI ODBORI / PROGRAMME COMMITTEES .......................................................................283 Koliko nas manjka? / Čepar Drago ...........................................................................................................285 Demografski trendi v svetu in Sloveniji / Gams Matjaž ............................................................................288 Komuniciranje s splošno javnostjo o zdravju in zdravilih: Razumevanje besedil z vidika starejših oseb / Kasesnik Karin ......................................................................................................................................293 Stanovanjske razmere starejših v Sloveniji in sprejemljivost različnih oblik bivanja / Kerbler Boštjan, Sendi Richard, Filipovič Hrast Maša ....................................................................................................297 Družina kot sistem in pomen družinske terapije / Kranjc Jakša Urška .....................................................300 Dolgotrajna oskrba – zahtevna demografska naloga in izjemna priložnost za krepitev sodobne solidarnosti / Ramovš Jože, Ramovš Ksenija, Rant Maja, Ramovš Marta, Mravljak Andoljšek Veronika, Ramovš Ana, Gorenc Vujović Ana, Svetelšek Ajda .............................................................303 Močna družinska medicina je najboljši način zdravstvene obravnave starajočega se prebivalstva / Rifel Janez ....................................................................................................................................................308 Razširjeni povzetek – Usposabljanje menedžmenta za obvladovanje staranja zaposlenih / Ramovš Jože, Grebenšek Tjaša ..................................................................................................................................310 Sodelovanje, programska oprema in storitve v informacijski družbi / Collaboration, Software and Services in Information Society .............................................................................................................313 PREDGOVOR / FOREWORD ...................................................................................................................315 PROGRAMSKI ODBORI / PROGRAMME COMMITTEES .......................................................................317 Self-Assessment Tool For Evaluating Sustainability Of Ict In Smes / Soini Jari, Leppäniemi Jari, Sil berg Pekka ....................................................................................................................................................319 Reference Standard Process Model For Farming To Support The Development Of Applications For Farming / Rupnik Rok ..........................................................................................................................323 Semiotics Of Graphical Signs In Bpmn / Kuhar Saša, Polančič Gregor ..................................................327 Knowledge Perception Infuenced By Notation Used For Conceptual Database Design / Kamišalić Aida, Turkanović Muhamed, Heričko Marjan, Welzer Tatjana ......................................................................331 The Use Of Standard Questionnaires For Evaluating The Usability Of Gamfication / Rajšp Alen, Kous Katja, Beranič Tina ...............................................................................................................................335 Analyzing Short Text Jokes From Online Sources With Machine Learning Approaches / Šimenko Samo, Podgorelec Vili, Karakatič Sašo ...........................................................................................................339 A Data Science Approach To The Analysis Of Food Recipes / Heričko Tjaša, Karakatič Sašo, Podgorelec Vili ......................................................................................................................................343 Introducing Blockchain Technology Into A Real-Life Insurance Use Case / Vodeb Aljaž, Tišler Aljaž, Chuchurski Martin, Orgulan Mojca, Rola Tadej, Unger Tea, Žnidar Žan, Turkanović Muhamed ........347 A Brief Overview Of Proposed Solutions To Achieve Ethereum Scalability / Podgorelec Blaž, Rek Patrik, Rola Tadej ............................................................................................................................................351 Integration Heaven Of Nanoservices / Révész Ádám, Pataki Norbert .....................................................355 Service Monitoring Agents For Devops Dashboard Tool / Török Márk, Pataki Norbert ...........................359 Incremental Parsing Of Large Legacy C/C++ Software / Fekete Anett, Cserép Máté .............................363 Visualising Compiler-Generated Special Member Functions Of C++ Types / Szalay Richárd, Porkoláb Zoltán ....................................................................................................................................................367 How Does An Integration With Vcs Affect Ssqsa? / Popović Bojan, Rakić Gordana ...............................371 Delavnica za elektronsko in mobilno zdravje ter pametna mesta / Workshop Electronic and Mobile Health and Smart Cities ..........................................................................................................................375 PREDGOVOR / FOREWORD ...................................................................................................................377 PROGRAMSKI ODBORI / PROGRAMME COMMITTEES .......................................................................379 Študija o pooperativni atrijski fibrilaciji / Čarman Dominik, Cvetkovič Tom, Avbelj Viktor, Kališnik Jurij Matija, Žibert Janez ..............................................................................................................................381 EMZ in EkoSMART-asistent / Drnovšek Mateja, Gams Matjaž ...............................................................385 vii Pregled asistentov IJS E9 / Drnovšek Mateja, Gams Matjaž, Tavčar Aleš, Grasselli Gregor .................389 Varnostna ura IJS / Drnovšek Mateja, Kolenik Tine, Gams Matjaž .........................................................394 New HRV biofeedback technique and monitoring its effects in a psychotherapeutic group of coronary patients with SAVVY mini ECG devices installed / Enova Dušan, Trobec Roman, Mesarič Katja, Slevec Katarina, Ambrožič Borut, Enova Tej .......................................................................................398 The Summoner – "Izbirčnež" / Glavač Aljaž, Zupančič Jernej, Gams Matjaž ..........................................402 Zaznavanje srčnega popuščanja z analizo srčnih tonov / Gradišek Anton, Gjoreski Martin, Budna Borut, Simjanoska Monika, Gams Matjaž, Poglajen Gregor ...........................................................................406 Elektrokardiografski senzor: uporaba med družinskimi zdravniki / Kocjančič Staša, Avbelj Viktor..........409 Pametno okolje za učinkovito ščetkanje zob / Kokol Peter, Colnarič Matjaž, Moravs Stanislav, Zorman Milan, Žlahtič Gregor, Završnik Jernej, Blažun Vošner Helena, Turčin Marko, Završnik Tadej, Jurič Simon, Slemnik Bojan, Detela Jernej ...................................................................................................411 Time Series or Relational Database for Edge and IoT / Luzar Anže, Stanovnik Sašo, Cankar Matija ....414 Spletni obrazec in REST API za naročanje receptov / Pavliha Denis, Kalan Aleš, Planinc Nataša ........418 Postopki in priporočila za izgradnjo govorne zbirke za potrebe sinteze slovenskega govora / Šef Tomaž ...................................................................................................................................................420 Telemetric ECG monitoring during physical activity in field tests / Širaiy Boris, Trobec Roman..............424 Napredni pogovorni svetovalci / Tavčar Aleš, Gams Matjaž ....................................................................427 The potential of Blockchain technology in health monitoring / Tošić Aleksandar, Burnard Michael, Vičič Jernej ....................................................................................................................................................431 Prenos projekta “Asistent IJS” na Python 3.6 / Valič Jakob .....................................................................435 Časovna sinhronizacija brezžičnih EKG senzorjev / Vilhar Andrej, Depol i Matjaž ..................................436 Integration of Oncology Information System with Proton Therapy Software and Think!EHR Platform / Vitorovič M., Kroflič Ž., Bobnar J., , Savadkouhi R., Smokvina A. .......................................................439 Training of precise movements in 3D or 2D for persons with Parkinson’s disease? / Cikajlo Imre, Matjačić Zlatko, Burger Helena, Peterlin Potisk Karmen .....................................................................441 Forecasting the physical fitness and all-cause mortality based of schoolchildren’s fitness measurements / Cigale Matej, Gradišek Anton, Mlakar Miha, Luštrek Mitja ..................................................................443 Zapestnica za pomoč starejšim / Kompara Tomaž ..................................................................................447 A Protocol for Joint Acquisition of Heterogeneous Resources / Grasselli Gregor ...................................449 Sistem za podporo odločanju zdravniku / Brodnik Andrej, Lukšič Nal, Žibert Janez, Živković Nenad, Rus Andrej ...................................................................................................................................................452 Razvoj intervencij za srčne bolnike in bolnike z motnjami gibanja / Cukjati Iztok, Bon Jure, Kališnik Jurij Matija, Žibert Janez, Pirtošek Zvezdan ................................................................................................456 Indeks avtorjev / Author index ....................................................................................................................459 viii Zbornik 21. mednarodne multikonference INFORMACIJSKA DRUŽBA – IS 2018 Zvezek A Proceedings of the 21st International Multiconference INFORMATION SOCIETY – IS 2018 Volume A Slovenska konferenca o umetni inteligenci Slovenian Conference on Artificial Intelligence Uredili / Edited by Mitja Luštrek, Rok Piltaver, Matjaž Gams http://is.ijs.si 11.–12. oktober 2018 / 11–12 October 2018 Ljubljana, Slovenia 1 2 PREDGOVOR V letu 2018 smo ponovno priča neverjetnim dosežkom umetne inteligence. Tako je bila letos poleti v Stockholmu največja svetovna konferenca na področju umetne inteligence IJCAI združena z evropsko ECAI, s čimer je imela 37 % več prispevkov kot prejšnje leto. Združeni konferenci sta skupno pritegnili preko 6.000 udeležencev. Približno polovica vseh prispevkov je bila kitajskih, pol manj je bilo evropejskih in ameriških. Velesile se zavedajo, da je področje umetne inteligence eno izmed ključnih, zato tako Putin kot Trump in Ši Džinping intenzivno povečujejo sredstva za njen razvoj, Evropa pa jih bo v prihodnjih letih nekajkrat povečala. Dnevno umetna inteligenca sprejme neverjetnih 10 bilijonov odločitev. Samo v lanskem letu je bilo dosežkov umetne inteligence toliko, da jih lahko omenimo le majhen delež. Na področju varnosti po svetu uporabljajo sistem, ki vsak dan izdela nov urnik obhodov varnostnikov po letališčih, pristaniščih in podobnih okoljih. Kjer so ti sistemi uporabljeni, je izmerjena bistveno večjo učinkovitost. V skrbi za okolje so raziskovalci pod vodstvom prof. Tambeja (med njimi je bil tudi naš doktorand dr. Kaluža) tovrstne sisteme podarili 60 rezervatom po svetu, da se bodo uspešneje upirali krivolovcem. Leta 2015 so sistemi na osnovi globokih nevronskih mrež začeli dohitevati ljudi pri prepoznavanju vidnih nalog in danes jih že prekašajo, npr. pri prepoznavanju malignih tkiv. Pri nekaterih nalogah, recimo pri ostrenju slike (zaradi dežja, megle, snega itd.), so sistemi osemkrat boljši od ljudi. Ob tem se seveda pojavlja tudi strah, vendar če na diagnozo, ali imate raka ali ne, čakate nekaj tednov, v ZDA pa to diagnozo postavi umetna inteligenca v nekaj minutah in to bolje kot katerikoli zdravnik – kaj pravite, ali bi jo uvedli tudi pri nas? V Sloveniji potrebno znanje že imamo, zatika se le pri vpeljavi. Pri nekaterih posegih, kot je recimo presaditev organov, so sistemi umetne inteligence že desetletja v uporabi in so rešila na tisoče življenj. Nekateri sistemi so tudi novejši – letos so tako vpeljali prvi inteligentni sistem, ki ugotavlja diabetes na podlagi pregleda oči, prav tako tudi prvi program za ugotavljanje abnormalnosti prsnega koša pri slikanju. Hiter razvoj je najbolj znan pri avtonomni vožnji – danes imajo povprečni avtomobili kar nekaj avtonomnih inteligentnih funkcij, modernejši (npr. Tesla) pa vozijo praktično sami in jih nadziramo samo še v nenavadnih situacijah. Nesreč avtonomnih vozil je približno stokrat manj kot tistih človeških voznikov, medijski odziv nanje pa je pogosto veliko bolj poročan in zato napihnjen. Mnoge zanimive dosežke umetne inteligence predstavljamo tudi na Slovenski konferenci o umetni inteligenci (SKUI), ki je naslednica konference Inteligentni sistemi in je sestavni del multikonference Informacijska družba že od njenega začetka leta 1997. Slovensko društvo za umetno inteligenco, ki letos praznuje že 26. obletnico, SKUI šteje za svojo konferenco. Letos je bilo sprejetih 17 prispevkov. Kot pretekla leta jih je največ z Instituta »Jožef Stefan«, nekaj pa jih je prispevala Fakulteta za računalništvo in informatiko, ki ima skupaj z Institutom vodilno vlogo pri raziskavah umetne inteligence v Sloveniji. Upamo, da bo prispevkov iz industrije in nasploh izven Instituta prihodnja leta še več, saj je ključen cilj SKUI povezovanje vseh slovenskih raziskovalcev umetne inteligence, čeprav na konferenci niso nič manj dobrodošli tudi prispevki iz tujine. Mitja Luštrek, Rok Piltaver, Matjaž Gams 3 PROGRAMSKI ODBOR / PROGRAMME COMMITTEE Mitja Luštrek Rok Piltaver Matjaž Gams Marko Bohanec Tomaž Banovec Cene Bavec Jaro Berce Marko Bonač Ivan Bratko Dušan Caf Bojan Cestnik Aleš Dobnikar Bogdan Filipič Nikola Guid Borka Jerman Blažič Tomaž Kalin Marjan Krisper Marjan Mernik Vladislav Rajkovič Ivo Rozman Niko Schlamberger Tomaž Seljak Miha Smolnikar Peter Stanovnik Damjan Strnad Peter Tancig Pavle Trdan Iztok Valenčič Vasja Vehovar Martin Žnidaršič 4 Monitoring Bumblebee Daily Activities Using Microphones Anton Gradišek Nicolas Cheron David Heise Jožef Stefan Institute Polytech Paris Sorbonne Lincoln University Ljubljana, Slovenia Paris, France Jefferson City, MO, United States anton.gradisek@ijs.si Candace Galen Janez Grad University of Missouri Faculty of Administration, Columbia, MO, United States University of Ljubljana Ljubljana, Slovenia ABSTRACT types (queen or worker) of bumblebees based on flight buzzing sound. Heise et al. [3] developed an algorithm to detect bee We present initial results of the study where we used buzzes from field recordings. In our case, the task was to detect microphones, placed in front of nest boxes, to monitor daily arrivals and departures of bumblebees from the nest boxes (both foraging activity of bumblebees. Sound recordings were analyzed of which result in buzzes recorded by the microphone), therefore using a custom-made computer algorithm which detects flight the algorithm was optimized for this task. We discuss the buzzing sounds coming from arrivals or departures of individual algorithm and show some initial results. bees. In addition, the algorithm distinguishes between arrivals and departures. We show examples of daily activities for three species 2. MATERIALS AND METHODS ( B. pascuorum, B. humilis and B. hypnorum), each was monitored 2.1 Data Collection over the course of one day. This paper presents initial results of a longer study where we plan to systematically investigate the USB stick microphones dB9PRO VR1.0 [4] were used for sound activities of bumblebees in various circumstances. recordings. Each microphone has 8 GB of flash memory, which gives it nominal storage capacity above 90 h. Sound is recorded at Keywords 48 kHz with a 192 Kbps bit rate. After each charging, a Bumblebees, foraging activity, sound analysis microphone can record for around 10 h. Microphones were placed in front of nest box entrances in order to record arrivals and departures. In the following, we demonstrate the results for three 1. INTRODUCTION different bumblebee families, each of them monitored over the Bumblebees (genus Bombus from the bee family Apidae) are an course of one day. The details of the investigated families are important group of wild pollinators. Due to different morphology listed in Table 1. In all cases, the microphones were set around 8 and lifestyle, when pollinating plants, they are often more am. For B. pascuorum, the microphone kept recording until the effective than honeybees – they are able to go foraging in rainy battery lasted while for the other two families, on the following and cold weather, and in addition, they use a special technique, day, the microphones were collected around 6 pm as the weather called buzz-pollination, to extract pollen from plants such as deteriorated. tomatoes. In addition to pollination in the wild, this makes bumblebees important players in greenhouse agriculture. Table 1. Bumblebee families studied Pollinator monitoring, as well as monitoring of wild pollinators, is No. of of high interest to agronomists, ecologists, and experts in the field Species Date Weather workers of conservation. In studies of bumblebee activity, currently the 28 14 – 28 °C, morning fog, most typical approaches are observations and capturing. B. May 10 sunny during the day, Capturing is problematic as it includes removal of individuals pascuorum 2018 storms in the evening from the environment. Sometimes, bumblebees are also studied in B. humilis 20 16 – 26 °C, morning laboratory conditions, by raising an entire colony in a lab, which 29 partially cloudy, light rain typically involves commercially available bumblebee species. One May B. after 4 pm, heavy rain after 30 can expect that the behavior in a laboratory is not identical to that 2018 hypnorum 6 pm in a natural environment. A better approach to controlled studies is introducing the bumblebees into special nest boxes outside. This allows us to monitor them in a near-natural environment. 2.2 Sound Recording Analysis In this paper, we present the first results of a study where we used The flowchart of the algorithm is shown in Figure 1. The microphones to monitor bumblebee daily foraging activities. algorithm was inspired by that of Heise et al. [3], but simplified in These activities are important to monitor as they provide a direct order to work faster as recordings of arrivals and departures in insight into pollination service. Using a microphone (recording front of a single nest box are typically cleaner than those from a sounds) is clearly advantageous from personal monitoring (such microphone located in the field. Our preliminary analysis was as in Grad et al. [1]) since it is continuous and allows us to carried out using the Audacity software while a more detailed monitor several sites simultaneously (using several microphones). analysis was done in Matlab, using in-built packages and own Bumblebee buzzing sounds have been studied before, though with code. In each recording, we manually labelled around 10 buzzes a different focus. Gradišek et al. [2] developed a machine- at the beginning in order to optimize the thresholds for the learning-based algorithm to recognize individual species and algorithm (described in the following). In addition, we manually 5 labelled the entire recording of B. pascuorum in order to evaluate 2. For each segment, we calculate the Fourier transform of the the performance of the algorithm. time-domain signal to obtain the spectra (frequency domain, spectrum amplitude as a function of frequency) Preliminary inspection showed that the microphones recorded bumblebee buzzes well, while also recording a series of noises 3. For each segment, we calculate seven independent Boolean from the environment, such as passing traffic or human speech. features that we use to decide whether the segment contains an Sometimes, these sounds can be louder than the buzzes event or not. In the following, the natural frequencies (the themselves. The task of our algorithm is therefore to detect loud frequency that a bumblebee flaps its wings during flight) are events and to check whether they are buzzes or noise. For optimized for B. pascuorum. For species with significantly positively identified buzzes, we next determine whether they different natural frequencies (see [2]), we modify the boundaries. correspond to arrival or departure of the bumblebee. The feature thresholds are set for each family as well, based on some manually inspected events (about 10 – 15 events at the beginning of each recording). a) We calculate the average amplitude of the segment (which can generally be done either in time or in frequency domain). If the amplitude is larger than a manually determined threshold value, this is a possible event (e.g. true). b) The natural frequency is f = 180 Hz. We count the number of peaks between 160 and 200 Hz (using the findpeaks function). If the number of peaks is smaller than the threshold, this is considered a buzz, otherwise we are dealing with noise. c) We calculate the ratio of average amplitudes around the proposed peak (average amplitude value on the interval 160 – 200 Hz) and below it (60 – 120 Hz). If the ratio is larger than the threshold, this can be a true buzz, otherwise it is likely to be noise. d) Similar to feature c), we check the ratio of the average amplitude around the proposed peak and above it (220 – 280 Hz). e) Similar to feature b), we look for a peak at double natural frequency (first harmonic), looking at the interval (2 * f – 20 Hz, 2 * f + 20 Hz). f, g) We follow the same procedure as for features c) and d), just Figure 1. Flowchart of the buzz detecting algorithm at the frequency of first harmonic and correspondingly higher interval boundaries. The algorithm is the following: If five or more features return “true”, we consider the segment to contain a buzz. This criterion was determined on a series of 1. The recording, typically several hours long, is cut to segments manually labelled events in order to maximize the accuracy. of 5 seconds. This length was determined to be sufficiently long to contain the entire event while short enough to contain only a Once we know that a segment contains a buzz, we can determine single event. whether it corresponds to arrival or departure. This part is carried out using signal in time domain. Figure 2 shows examples of both As the recording is cut into segments, there is a possibility that the events. They are roughly symmetric in shape, which is reasonable cutting occurs in the middle of a buzz. To identify such cases, a given the dynamics of the process. When a bumblebee arrives to special function first checks whether the peak amplitude occurs in the nest box, it is initially far from the microphone and then gets the first second of the segment. If it does, it checks the last second closer – resulting in an increasing signal amplitude. When it of the previous sample, if the peak amplitude occurs in the last lands, it stops buzzing, thus a sharp drop in signal. For departure, second there (indicating that the buzz was cut to two segments). the bumblebee starts flying (sharp jump) and then flies away from In such cases, both segments are merged into a single segment and the microphone (gradual drop in amplitude). the analysis proceeds as described in the following (with only one buzz counted). 6 Figure 2. Signal envelope in time domain for arrivals (top) and departures (bottom) To classify the event as an arrival or a departure, we do the following. First, we calculate the signal envelope and smooth it to reduce the noise. Such envelopes can be seen in Figure 2. Next, we use the findpeaks function to identify peaks and we calculate the maximum absolute difference between two consecutive peaks. We call this a “drop”. Looking at Figure 2, we see that the drop appears at the end of arrival and at the beginning of departure. By integrating the area before and after the drop over a chosen interval, we can determine the arrival or departure. As each segment has a timestamp, we are able to plot histograms of either arrivals or departures of bumblebees throughout several hours. 3. RESULTS AND DISCUSSION Figure 3 shows three histograms for bumblebee departures on a chosen day, on hourly basis. Figure 3. Histograms for number of departures (reflecting daily activity) for three bumblebee families, as described in Figure 3 only shows the number of departures. The numbers for Table 1. arrivals are very similar, as is to be expected. These three histograms provide a good insight into the daily dynamics of each family. Different species have different foraging habits, for To check the accuracy of the algorithm, we tested it on the example, B. pascuorum were mostly active around noon and in manually-labelled recording ( B. pascuorum). Out of 180 actual the afternoon while less active in the morning. On the other hand, events (counting arrivals and departures together, P), our B. hypnorum were more active after 3 pm. Light rain at 4 pm algorithm correctly detected 171 events (TP), 9 detected events made the B. humilis workers stay inside but it did not affect B. were not buzzes (FP), and 9 events were missed (FN). Based on hypnorum. Of course, as these are initial results on limited this, we can determine the algorithm sensitivity, TP / P = TP / (TP datasets, a longer data collection will be required to investigate + FN) = 0,95 and precision TP / (TP + FP) = 0,95. Clearly, this the dynamics as the family develops over the course of several estimate is based on a single long recording and may vary for months. other conditions (different species or different structure of noise). 7 4. CONCLUSION 6. REFERENCES We demonstrate that microphones can be used as a simple tool to [1] Grad, J., Gradišek, A., Gams, M., Čmrlji: pašna dejavnost in study bumblebee foraging activity, as opposed to personal zvok brenčanja, Poklukarjevi dnevi, Book of abstracts, 2016 monitoring. The algorithm we developed detects potential buzzes [2] Gradišek, A., Slapničar, G., Šorn, J., Luštrek, M., Gams, M. and classifies them as either arrivals or departures. Compared to and Grad, J. 2016: Predicting species identity of bumblebees the performance of a human manually labelling the buzzes in the through analysis of flight buzzing sounds. Bioacoustics, 26 recording, the algorithm works with 95 % sensitivity and 95 % (1), 63-76 precision, which we consider sufficient for meaningful results. In future, we plan to study several bumblebee families throughout [3] Heise, D., Miller-Struttmann, N., Galen, C., Schul, J. 2017: the year to investigate the effects of the weather, temperatures, Acoustic Detection of Bees in the Field Using CASA with family size, and other parameters on foraging activity. Focal Templates, 2017 IEEE Sensors Applications Symposium (SAS), Glassboro, NJ, pp. 1-5. 5. ACKNOWLEDGMENTS [4] http://db9pro.com/ The research is partially funded by National Geographic Grant NGS-282-18 (to C. Galen). 8 Reconstructing PPG Signal from Video Recordings Gašper Slapničar Andrejaana Andova Department of Intelligent Department of Intelligent Systems Systems Jožef Stefan Institute Jožef Stefan Institute Jamova cesta 39 Jamova cesta 39 SI-1000 Ljubljana, Slovenia SI-1000 Ljubljana, Slovenia gasper.slapnicar@ijs.si andrejaana.andova@ijs.si Erik Dovgan Mitja Luštrek Department of Intelligent Department of Intelligent Systems Systems Jožef Stefan Institute Jožef Stefan Institute Jamova cesta 39 Jamova cesta 39 SI-1000 Ljubljana, Slovenia SI-1000 Ljubljana, Slovenia erik.dovgan@ijs.si mitja.lustrek@ijs.si ABSTRACT rPPG from a camera recording is more difficult as the source Physiological signals give important insight regarding some- of light is most commonly the sun or the lighting of a room. one’s health. It would be in the interest of people to monitor This makes such an approach more sensitive to environmen- such signals without any wearable devices. We used RGB tal conditions and less accurate compared to contact sensors. camera recordings of faces to reconstruct the PPG signal, rPPG reconstruction would allow for estimation of several which can be used to monitor many physiological signals physiological parameters, such as heart rate (HR), breath- such as heart rate, breathing rate, blood pressure, etc. A ing rate, heart-rate variability and blood pressure, without deep learning method was developed to enhance existing a wearable device. state-of-the-art methods. This method uses the output of an existing method as an input into a LSTM neural network, The rest of the paper is organized as follows. Section 2 which substantially improves the reconstruction of PPG. reviews the related work. The methods for reconstructing the PPG signal are described in Section 3, while the ex- Keywords periments and results are discussed in Section 4. Finally, Section 5 concludes the paper with ideas for future work. remote PPG, signal processing, deep learning 1. INTRODUCTION 2. RELATED WORK Physiological signals, such as photoplethymogram (PPG), There are two main approaches for reconstructing rPPG, are traditionally measured using wearable devices like cuffs which are based on different underlying physiological phe- and wristbands. While such devices are rather unobtrusive, nomena. it would be preferable to omit them completely. This can be achieved with the use of contact-free devices such as RGB The first approach focuses on variations in blood volume, cameras, which can blend into the environment, allowing which is reflected in the changes of the skin color. To detect for remote physiological signal reconstruction. An example the variations of blood volume using non-contact sensors group for whom such a system would be useful are people (camera), tiny changes in RGB intensity of the skin pix- with profound cognitive impairment, who are the subjects els between two sequential video frames are analyzed. For participating in the INSENSION project1, for which our sys- example, Poh et al. [9, 10] applied independent component tem is being developed. analysis (ICA) on the RGB color signals, which were com- puted as the average of the red, green and blue intensity of This paper aims to compare and enhance existing approaches all the skin pixels over time. They then chose the most PPG- for reconstructing PPG from video data, i.e., remote PPG like resulting signal. Lewandowska et al. [5] used principal (rPPG). The PPG signal describes the changes of blood vol- component analysis (PCA) instead of ICA to reconstruct the ume in the skin tissue, which corresponds to the heart pe- PPG signal. Haan et al. [2] reconstructed the PPG signal riodically pushing the blood towards the periphery of the simply by calculating a specific linear combination of the ob- body with each beat. When skin tissue gets filled with tained RGB traces. Other approaches do not calculate the blood, it becomes slightly darker and absorbs more light. average of all skin pixels, but treat each skin pixel indepen- The light source in contact sensors (e.g., wristbands) is con- dently. For example, Wang et al. [11] tracked the variation centrated and constant (a light emitting diode – LED) and of color in each skin pixel independently and chose the most enables high-quality PPG reconstruction. Reconstructing PPG-like signal afterwards. The changes of the skin pixel values were also tracked to reconstruct the PPG signal [12]. 1http://www.insension.eu Petil et al. [7] used the basic RGB signals as inputs to a 9 neural network to reconstruct various physiological signals. SVM method produces worse results than the threshold- Another example by Wu et al. [13] amplified all the color based method, but generalizes well for various skin colors changes of the facial pixels to follow the blood flow in these and shades. Both skin classification methods incorrectly pixels. Although the presented methods seem promising, an classify some of the non-skin pixels as skin. To avoid false independent evaluation conducted by Heusch et al. [3] on a positives, we selected only the pixels that are most likely to publicly available dataset showed that they are not accurate actually be skin. We did this by calculating the mean value enough to be used in real-world scenarios. More precisely, of all the skin pixels returned by the classifier and then re- this evaluation re-implemented three state-of-the-art meth- moving the outlier pixels with respect to the mean in the ods for reconstructing PPG from RGB cameras, and the YCbCr color space. results showed that there is a very low correlation between the reconstructed and ground-truth PPG. 3.2 State-of-the-Art Methods The second approach for PPG reconstruction from video We have evaluated a set of state-of-the-art methods. These analyzes the small head movements that are induced by methods can be classified as color-based or movement-based the blood being pumped into the head. Such a study was as described in Section 2. conducted by Balakrishnan et al. [1], however, it should be noted that such movements are very subtle and might not Poh-et-al method: This is a color-based method that be detectable with a low quality camera, imposing an addi- sequences the mean value of the red, green and blue in- tional hardware requirement on this approach. tensity of all the skin pixels to create three different color traces. Since all the traces contain some information about the blood flow, it first normalizes them and then transforms 3. RECONSTRUCTING PPG WITH VISION- them with ICA using the FastICA algorithm [4, 9, 10]. This BASED METHODS method returns three signals, so we choose the one with most This section presents the developed deep-learning-based me- frequencies in the range [0.6 Hz, 4 Hz], i.e., the frequency thod for reconstructing the PPG signal from video data. range of PPG. This is done by analyzing the power spectrum This method enhances the signal reconstruction of an exist- of each output signal. ing state-of-the-art method, as none of them were satisfac- tory. We first present the state-of-the-art methods used in Haan-et-al method: This is also a color-based method the evaluation. All of these methods have a similar prepro- which, similarly to the previous method, uses the mean of cessing step, which is presented in Section 3.1. The steps the red, green and blue intensity of all the skin pixels [2]. It specific for each ofcoh these methods are presented in Sec- then creates a linear combination from the red, green and tion 3.2. Finally, in Section 3.3 we present the developed blue traces, resulting in two new traces X and Y, calcu- method that takes as input the PPG reconstructed with an lated as: X = 3R − 2G; Y = 1.5R + G − 1.5B. The X existing method and returns an enhanced reconstruction of and Y traces are then filtered and combined to reconstruct PPG. the PPG signal. In our experiments, we used the method implementation from the BOB library3. 3.1 Preprocessing of Video Data The first preprocessing step consists of the detection of the Wang-et-al method: This color-based method uses all the subject’s face as the “region of interest” (ROI). For detecting skin pixels from an individual frame to define the color space the face, we used the Haar cascades, implemented in the of frames [12]. By tracking the changes in this space, we re- OpenCV library2. More precisely, the video was segmented construct the PPG signal. To accomplish this, a covariance into individual frames and only the selected face ROI was matrix is computed. This covariance matrix changes for each cropped from each frame. frame due to the blood flowing into the skin. By calculating the eigenvectors of the original frame and the eigenvalues of The second step of video preprocessing aims at discriminat- the covariance matrix, we get a representation of the color ing between skin and non-skin pixels. For this purpose, we space for the skin pixels. The rotation between two eigen- implemented two classification methods. The first method vectors of sequential frames represents the changes of the transforms the RGB color space into the YCbCr color space, color space. This rotation is also related to different rel- which contains less redundant information. Pixel values ative PPG contributors. Therefore, by concatenating the are then classified as either skin or non-skin using thresh- rotation between the first opposing to the second and the olds. This method is fast, simple and works well on the third eigenvector, PPG-like traces are retrieved. The eigen- test dataset, however it probably does not generalize well values are also influenced by the pulsatile blood and are thus to datasets where the degree of variation of skin colors and used to normalize the PPG-like signals. As for the previous shades is higher. The second method applies one-class sup- method, we also used the method implementation from the port vector machines (SVM) to classify the skin pixels. It BOB library3. learns a decision function for novelty detection from posi- tive examples (corresponding to skin pixels), which are ob- Balakrishnan-et-al method: In contrast to the previ- tained from the forehead region of the first three frames of ously presented methods, this is a motion-based method, each video. New data is then classified as similar (skin) or since it focuses on the oscillations of the head [1]. This different (not skin) to the training set. The forehead re- method does not need to detect skin pixels, therefore, the gion is detected as the facial area of fixed dimensions above second step of data preprocessing is skipped. To recon- the eyes, which can be easily detected using OpenCV. The struct the oscillations, the Lucas-Kanade flow-tracking al- 2https://opencv.org 3https://pypi.org/project/bob.rppg.base 10 gorithm [6] is applied, which tracks the flow of the head movements in the vertical direction. The oscillation signals are then filtered using a band-pass filter with the frequency interval [0.6 Hz, 4 Hz], i.e., the frequency range of PPG. Af- terwards, PCA is applied to select the most PPG-like signal. 3.3 Deep-Learning-Based Method We developed a new method for reconstructing the PPG signal, which takes the PPG reconstructed by an existing method as the input, and outputs an improved reconstruc- tion of the PPG signal. To achieve this, it applies deep learning, which has recently shown superior performance in machine learning on many domains compared to traditional approaches. To build the deep learning model, we used a Long-Short Term Memory (LSTM) network [8]. Its architecture com- prised two LSTM layers and one fully-connected layer. The window length was set to 100 samples, i.e., five seconds. Each layer had 50 LSTM units, each taking input of length 100 and returning output of the same length, as shown in Figure 1. The output of the Wang-et-al method [12] has been selected as the input to the LSTM network. 4. EXPERIMENTS AND RESULTS Figure 1: The architecture of the network used in In order to evaluate the quality of methods described in the Deep-Learning-Based method. Section 3.2 and our method, the reconstructed PPG was compared with the ground truth obtained with a fingertip PPG sensor. 4.1 Materials and Experimental Setup The existing methods and the developed method have been evaluated on the COHFACE dataset4. This dataset consists of 160 videos from 40 different subjects with corresponding synchronized PPG collected with a fingertip device. The (a) (b) mean value of heart rate over the whole dataset is 70.25 beats per minute (BPM) with the corresponding standard Figure 2: Classified skin using the (a) threshold deviation of 11.56. method, and (b) machine learning method. A preliminary test has been done to select the best skin clas- sification method. The evaluated skin classification meth- mentioned metrics are reported, i.e., MAE, MSE and HR ods were threshold-based method and SVM-based method MAE. Note that the heart rate MAE of the baseline is 9.67 as described in Section 3.1. Examples of the masks returned BPM, while our method achieves 8.75 BPM. In addition, the by both methods are shown in Figure 2. The results show first 10 seconds of the reconstructed PPG using each of the that the threshold-based method is better than SVM-based. methods on a subset of videos are shown in Figures 3–5. However, it should be noted that the selected thresholds were fitted to the selected dataset, therefore, the method The results show that the Deep-Learning-Based method pro- might not generalize well to other data. duces better reconstruction of the PPG signal, as the error between the estimated and actual HR is the lowest. Table 1 To evaluate the developed method, a leave-one-subject-out also shows that the developed method outperforms state- experiment was conducted with the aim of testing its pre- of-the-art methods on the COHFACE dataset by a notable dictive performance and generalization capability. To this margin. end, mean absolute error (MAE) and mean squared error (MSE) were used as metrics. Additionally, to evaluate the 5. CONCLUSIONS quality of HR predictions, we computed the number of peaks in the reconstructed signals and compared it to the number We presented a new approach for the reconstruction of the of peaks in the ground truth PPG. PPG signal from video data. This approach enhances state- of-the-art methods with a deep learning model. It has been 4.2 Experimental Results evaluated on the COHFACE dataset and the results show that it improves the PPG reconstruction with respect to the Results of all the evaluated methods, as well as the devel- state-of-the-art methods. oped method, are given in Table 1. All three previously 4https://www.idiap.ch/dataset/cohface However, the reconstructed PPG signal is still noisy and it 11 Table 1: Comparison between state-of-the-art meth- Deep-Learning-Based ods and the developed method Ground truth MAE MSE MAE [BPM] Method (signals) (signals) (heart rates) Poh-et-al 0.04 0.15 42.00 Haan-et-al 0.16 0.04 20.75 Wang-et-al 0.16 0.40 11.73 Balakrishnan-et-al 0.16 0.04 39.00 Deep-Learning 0.04 0.01 8.75 Figure 5: First 10 seconds of the Deep-Learning- Based method. Wang-et-al Poh-et-al Haan-et-al Ground truth study on remote heart rate measurement. CoRR, abs/1709.00962, 2017. [4] A. Hyvarinen. Fast ICA for noisy data using Gaussian moments. In ISCAS’99. Proceedings of the 1999 IEEE International Symposium on Circuits and Systems VLSI, volume 5, pages 57–61, May 1999. [5] M. Lewandowska, J. Rumiński, T. Kocejko, and J. Nowak. Measuring pulse rate with a webcam — A non-contact method for evaluating cardiac activity. In 2011 Federated Conference on Computer Science and Information Systems (FedCSIS), pages 405–410, Sept 2011. Figure 3: First 10 seconds of the color-based meth- [6] B. D. Lucas and T. Kanade. An iterative image ods. registration technique with an application to stereo vision. In Proceedings of the 7th International Joint Conference on Artificial Intelligence – Volume 2, would thus be difficult to estimate any physiological param- IJCAI’81, pages 674–679, San Francisco, CA, USA, eters from it, which will need to be improved in future work. 1981. Morgan Kaufmann Publishers Inc. Additionally, higher quality of the recordings, especially re- [7] O. R. Patil, Y. Gao, B. Li, and Z. Jin. CamBP: A garding the lighting conditions, will be evaluated with the camera-based, non-contact blood pressure monitor. In aim of obtaining better results. Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing 6. ACKNOWLEDGMENTS and Proceedings of the 2017 ACM International This work is part of a project that has received funding from Symposium on Wearable Computers, UbiComp ’17, the European Union’s Horizon 2020 research and innovation pages 524–529, New York, NY, USA, 2017. ACM. programme under grant agreement No. 780819. [8] J. Patterson and A. Gibson. Deep Learning: A Practitioner’s Approach. O’Reilly, Sebastopol, 2017. 7. REFERENCES [9] M. Poh, D. J. McDuff, and R. W. Picard. [1] G. Balakrishnan, F. Durand, and J. Guttag. Detecting Advancements in noncontact, multiparameter pulse from head motions in video. In 2013 IEEE physiological measurements using a webcam. IEEE Conference on Computer Vision and Pattern Transactions on Biomedical Engineering, 58(1):7–11, Recognition, pages 3430–3437, June 2013. Jan 2011. [2] G. de Haan and V. Jeanne. Robust pulse rate from [10] M.-Z. Poh, D. J. McDuff, and R. W. Picard. chrominance-based rPPG. IEEE Transactions on Non-contact, automated cardiac pulse measurements Biomedical Engineering, 60(10):2878–2886, Oct 2013. using video imaging and blind source separation. Opt. Express, 18(10):10762–10774, May 2010. [3] G. Heusch, A. Anjos, and S. Marcel. A reproducible [11] W. Wang, S. Stuijk, and G. de Haan. Exploiting spatial redundancy of image sensor for motion robust rPPG. IEEE Transactions on Biomedical Engineering, Balakrishnan-et-al Ground truth 62(2):415–425, Feb 2015. [12] W. Wang, S. Stuijk, and G. de Haan. A novel algorithm for remote photoplethysmography: Spatial subspace rotation. IEEE Transactions on Biomedical Engineering, 63(9):1974–1984, Sept 2016. [13] H.-Y. Wu, M. Rubinstein, E. Shih, J. Guttag, F. Durand, and W. Freeman. Eulerian video magnification for revealing subtle changes in the world. Figure 4: First 10 seconds of the motion-based ACM Trans. Graph., 31(4):65:1–65:8, July 2012. method. 12 The Influence of Communication Structure on Performance of an Agent-based Distributed Control System Andreja Malus, Rok Vrabič, Dominik Matjaž Gams Kozjek, Peter Butala Jožef Stefan Institute, Faculty of Mechanical Engineering, Department of Intelligent Systems University of Ljubljana Jamova cesta 39, 1000 Ljubljana Aškerčeva 6, 1000 Ljubljana matjaz.gams@ijs.si andreja.malus@fs.uni-lj.si ABSTRACT based on feedback from the environment is called learning agent Industrial Internet of Things (IIoT) is a new concept denoting [2]. extensive use of ubiquitous connected devices on the In some applications, multiple software agents are used to manufacturing shop floor. While most recent research in this area collectively solve problems by interacting with each other and focuses on the monitoring capabilities of IIoT and on the resulting reaching mutual agreements through negotiations, bidding and data analysis, IIoT also presents an opportunity from the other communication mechanisms, enabling reconfigurability perspective of distributed control. The paper suggests that agent- and scalability [3]. Agents acting in parallel results in the based control of an industrial process can be realized by a multi- system’s global behaviour that may include emergent phenomena agent system in which each agent is able to learn the influence of and is often difficult to predict in advance. Design of interactions its actions on the behaviour of the system and to communicate between elements is therefore mostly performed using with other agents in its proximity. Influence of the simulations to obtain the desired behaviour of the system in a trial communication structure on performance, robustness, and and error manner. Although the ideas of agent-based systems resilience are analysed for a case of an industrial compressed air originated more than two decades ago and much research has system. The simulation results suggest that in such systems, been done on intelligence of software agents and coordination communication between the supply and the demand side mechanisms, not many examples of real-life implementations can improves resilience, while the robustness is improved through be found in the manufacturing industry [4]. learning. Despite the challenges of implementation in industrial applications, emerging computing paradigms, developing communication protocols, and decreasing cost of computing KEYWORDS power and network communications are suggesting that the Industrial Internet of Things, multi-agent system, distributed research field should be revisited in context of industrial control, machine learning, robustness applications. In [5] a control method using rationally bounded learning agents was proposed. This paper extends this work by analysing the 1. INTRODUCTION influence of different connection schemes on performance in Cyber-physical systems (CPS) represent an emerging paradigm normal and adverse conditions. The response of the system in integrating computational, networking and physical processes to adverse conditions can be evaluated from the point of view of address requirements of future industrial systems [1]. Often the robustness and resilience. Robustness measures the extent to interfaces between the physical and the virtual worlds are realized which a disturbance affects the system’s performance while using connected intelligent sensing and actuating devices. Their resilience represents the system’s ability of restoring normal use in manufacturing environment is the basis of the concept of operation. Using a simulation of an industrial compressed air Industrial Internet of Things (IIoT). Numerous connected IIoT system it is shown that (1) the communication between the agents devices enable acquisition and sharing of large amounts of data, representing the supply (compressors) and the agents representing promising time and cost savings, scalability and efficiency. the demand (consumers) improves the response of the system to However, as IIoT systems grow in size and complexity, their repeated disturbance, (2) full connectedness is not necessary as response times and computational complexity outgrow traditional additional connections beyond a certain point do not contribute to centralized control systems. Researchers work on enhancing improved system performance, and that (3) the communication flexibility, robustness, adaptability and reconfigurability of CPS structure influences resilience, but not robustness. by employing concepts of distributed and autonomous control in dynamic environments of flexible manufacturing systems. 2. DISTRIBUTED CONTROL WITH A common way to implement distributed control in RATIONALLY BOUNDED AGENTS manufacturing systems is by using autonomous computational A perfectly rational agent makes decisions under the assumptions entities called agents. Agents acquire information about their that (1) it has complete knowledge of the problem space and is environment and take actions to influence the environment. They aware of all its available actions, (2) the preferences of actions are can exhibit various levels of intelligence depending on the known and (3) it has the ability to discover the optimal policy method used to select an action based on the state of environment. regardless of the necessary computational demand [6]. Absence Reflex agents passively react to signals from environment, while of any of the three assumptions makes an agent rationally pro-active agents select their actions continuously to achieve a bounded [7]. In the engineering field, the use of the term bounded goal or utility. An agent that has the capacity to adapt its operation rationality refers to limited calculation time and computational 13 capacity. Design of artificial agents strives for optimization under learning method, resistant to overfitting and easy to use, since no time and capacity constraints. scaling and normalization of the data is required. The presented agent model assumes that in a truly distributed 4. EXPERIMENTAL CASE system none of the agents has an overview of the state of the Compressed air is widely used in industrial systems as a medium whole system and that each agent can communicate only with for energy transfer to various systems, for example power elements of the system (i.e. sensors, actuators, and other agents) equipment, spraying tools, conveyers, and power controls. It is in their proximity. Based on these limitations, the agent model safe, easy to use and maintain. However, more than 75 % of the defines what agents observe, how they learn from the life-cycle costs of compressed air system are accounted for by observations, and how they communicate with other agents. energy consumption [9] and reports estimate that only about 10– 3. AGENT MODEL 20% of the total input energy is utilized for useful work [10]. In the European Union compressed air systems are reported to A multi-agent system is described as a network 𝑁 = (𝐴, 𝐶𝐴, 𝑆, 𝐶𝑆, 𝑇, 𝐶𝑇) consume 80 TWh of electricity [9] or 10% of industrial electricity where 𝐴 represents a set of agents and 𝐶𝐴 a consumption [10] but at the same time potential economic savings set of communication channels between pairs of the agents. 𝑆 is a of more than 30 % are estimated [9]. Inefficiencies can be set of sensors, and 𝐶𝑆 a set of measurement channels between the attributed to many reasons, the most important being leakages and agents and the sensors, where one agent is connected and reads inefficient control [10]. measurement values from at least one sensor and each sensor is connected to one or more agents. Sensors and communication Few compressed air systems operate at full-load all of the time. with other agents enable the agents to gain information about the Part-load performance is therefore critical and is primarily system but each agent can only have a partial view and no agent influenced by compressor type and control strategy. The choice has an overview of the whole state of the system. Each agent is of the type of control depends largely on the type of compressor connected to exactly one actuator from the set of actuators 𝑇, the being used and the demand profile of the system. For a system connections are represented in set 𝐶𝑇. with a single processor and mostly steady demand, a simple control system may be appropriate. Simple control approach most All agents in the system have the same structure, regardless of often uses two pressure thresholds; when pressure drops below their function in the system. The components of the model are the lower threshold the compressor is turned on and if pressure shown in Figure 1. Agent 𝑎𝑖 ∈ 𝐴 has some belief 𝑏𝑖 about its exceeds the upper threshold the compressor is turned off. environment based on measurements of the sensors it has access However, a complex system with multiple compressors, varying to 𝑆𝑖 and predictions received from other agents 𝑃𝑗𝑖 as shown in demand, and many types of end-uses requires a more Eq. (1). sophisticated control strategy. 𝑏𝑖 = 𝑓(𝑆𝑖, 𝑃𝑗𝑖) (1) This developed distributed control model aims to present a robust The agent model also includes an environment model 𝑀 and scalable alternative approach for a compressed air system 𝐸 that is used to make a set of predictions 𝑃 with multiple compressors and compressed air storage tanks 𝑖 about future sensor values based on current belief 𝑏 using autonomous switches for turning the compressors on or off 𝑖 and actions available 𝑈𝑖 as shown in Eq. (2). and autonomous valves for controlling the transport paths. 𝑃𝑖 = 𝑀𝐸(𝑏𝑖, 𝑈𝑖) (2) 5. SIMULATION Based on predictions 𝑃 The simulated compressed air system, shown in Figure 3, consists 𝑖 and the sensor goal values 𝐺𝑖 the agent selects its next action 𝑢 of 2 compressors, 2 compressed-air tanks, 4 consumers, 3 smart 𝑖, as shown in Eq. (3), that minimizes the selection criterion (e.g. an error estimate) in the form of a function valves, and piping. Each compressor supplies air to one tank and 𝑓 each tank has a safety valve to prevent the tank from becoming 𝐴. over-pressurized. Pressure sensors are installed on both 𝑢𝑖 = argmin(𝑓𝐴(𝑀𝐸(𝑏𝑖, 𝑈𝑖), 𝐺𝑖)) (3) compressed-air tanks and consumers. 𝑈𝑖 The environment model M The position of the valves in the system enables flow control of E is learned by observing the control the air in the system. Smart valves and compressors’ on/off actions 𝑢𝑖 taken to influence the environment and the environment’s response to the actions, captured by the agent’s switches are controlled by software agents, that are also connected to corresponding sensors as depicted in the figure. The belief 𝑏𝑖. In absence of communication with other agents the parameters of the model, e.g. the pressures, time constants and environmental model represents the physical model of the system implicitly including other agents’ influence on the state of system. leakage rates, are chosen to correspond to values that are commonly found in real systems. Agents have set target values When connected to other agents, an agent receives predictions of for their corresponding sensors and the goal is to keep the air future states of environment. These predictions include the pressure values as close to the target values as possible all the knowledge of other agents about their influence on the time, regardless of consumption. environment implicitly captured by their environmental models. The agent has one random forest regressor for learning the The agent acquires the environmental model using random forest environmental model for each of its connected sensors. The algorithm [8]. The algorithm is a highly accurate ensemble number of estimators is set to 100. Figure 2. Co Fi mmg uu nre ic 1 a : Ag tion en sc t hear m c e hitec s for tu d r if e fe rent scenarios 14 The total time of one simulation is set to 4000 s. Each scenario was simulated 150 times. The consumers’ activity is random, each consumer is alternately set to on for 1 – 4 s and off for 30 – 40 s. The goal values of the compressor and the smart valves agents are set to 6.3 and 5.0 bar respectively and the safety valve has a set pressure of 10 bar for all scenarios. The time step ∆𝑡 is set to 1 s. The predictions are calculated for the next 1-3 s. The duration of current prediction horizon determines the duration of the corresponding reasoning cycle. The environmental model is updated every 500s. In all scenarios a disturbance was simulated. The disturbance is represented by compressor 1 failure and opening of the corresponding tank’s release valve, it is repeated 5 times and lasts for 200 s. Time of start of disturbance in simulation is noted t 1 d = 1700 s, t 2 3 4 5 d = 2200 s, t d = 2700 s, t d = 3200 s and t d = 3700 s. Figure 3. Simulated compressed air system The results of the simulations were used to evaluate the ability of The input features for the regressor for the 𝑘 − 𝑡ℎ sensor consist the system to withstand adverse conditions, called robustness, and of the last action 𝑢 its ability to recover from disturbance, called resilience. In the 𝑖 taken by the agent, the observed sensor values 𝑠 context of this paper, robustness 𝑐𝑟𝑜𝑏, shown in Eq. (7), is defined 𝑖𝑘 ∈ 𝑆𝑖, the differences ∆𝑆 of each sensor value 𝑠𝑖𝑘 compared to every other sensor value 𝑠 as the ratio between the pressure drop in the system during 𝑖𝑚; 𝑚 ≠ 𝑘, the other agents’ predictions 𝑃 disturbance ∆𝑝𝑑 and the pressure 𝑝𝑠 subtracted from the whole. 𝑗𝑖, the differences in the other agents’ predictions denoted as 𝛥𝑃𝑗𝑖 and the existence of the other agents’ predictions denoted as ∆𝑝 𝑐 𝑑 (7) 𝑃∗ as shown in Eq. (4). 𝑟𝑜𝑏 = 1 − 𝑝𝑠 𝑥 = {𝑢𝑖, 𝑆𝑖, ∆𝑆, 𝑃𝑗𝑖, ∆𝑃𝑗𝑖, 𝑃∗ } (4) The moment 𝑡𝑏𝑏 when the rise of the pressure in the system after the end of the disturbance reaches 1-1/e = 63,2% of the pressure The associated outputs y are the differences of the sensor values drop is observed. The difference between this moment and the at times 𝑡 and 𝑡 + ∆𝑡, as shown in Eq. (5). time of the end of the disturbance is defined as bounce-back time 𝑦 = {𝑆(𝑡 + ∆𝑡) − 𝑆(𝑡)} (5) ∆𝑡𝑏𝑏. Resilience 𝑐𝑟𝑒𝑠, for the context of this paper, is defined as When calculating predictions for more than one step in advance, the inverse of the bounce-back time as shown in Eq. (8). predictions of all agent's sensors for each following timestep 1 𝑐 (within the observed agent) are calculated prior to calculating 𝑟𝑒𝑠 = (8) ∆𝑡𝑏𝑏 predictions for the next timestep. The next action is selected using the mean-square-error function 6. RESULTS for evaluating the predictions against the goal values for all Simulation results for average sensor values measured by sensor sensors. The function is shown in Eq. (6), where 𝑆𝑖 is a set of SC1 for all five scenarios are shown in Figure 4. Due to learning, agent’s sensors, 𝐻 is the number of future times for which the the pressure drop in the time of disturbance is lower after every effects of the action 𝑢 ∈ 𝑈𝑖 are assessed, called the prediction repetition of the disturbance in the first four occurrences of the horizon, and 𝑝𝑖𝑘 and 𝑔𝑖𝑘, 𝑘 ∈ 𝑆𝑖, are predictions and goal values event. for the 𝑘-th sensor, respectively. |𝑆𝑖| 𝐻 1 1 𝑓𝐴(𝑢) = ∑ ( ∑ (𝑝 ) (6) |𝑆 𝑖𝑘(ℎ) − 𝑔𝑖𝑘)2 𝑖| 𝐻 𝑘=1 ℎ=𝑡+∆𝑡 In 5 scenarios, the influence of different communication connection schemes, presented in Figure 3, was tested. In scenario A the agents don’t communicate among themselves, in scenario B the compressor agents communicate with each other and the smart valves agents communicate with neighbouring smart valves agents. Communication connections in scenario C follow the flow of the air in the system. In scenario D each compressor agent communicates with the nearest two smart valves and the smart valves are also connected to their Figure 4. Comparison of average pressure on sensor 𝑺 neighbours. The last scenario E represents full communication 𝑪𝟏 for 5 scenarios scheme in which all agents communicate with all other agents in the system. 15 Figure 5 shows the pressure values 50 seconds after the end of the 7. CONCLUSION first and the fourth disturbance, effectively demonstrating the The paper argues that the current understanding of the role of effect of learning in the considered scenarios. The first IIoT, which is mostly related to monitoring and data analytics, disturbance (Figure 5a) has approximately the same effect in all should be extended to the domain of distributed control. A scenarios because the agents have not yet learned how to mitigate distributed, agent-based control model is presented. The model its effects. However, the bounce back from the fourth disturbance assumes that no agent has an overview of the whole system state, (Figure 5b) differs significantly depending on the scenario. In but rather only has a partial view of its neighbouring sensors, scenarios A and B in which the supply (compressors) and the actuators, and other agents. demand (consumers) are not connected, the bounce back is slower and more scattered than in scenarios C, D, and E. The paper builds on a previously conceived agent model [5] and explores the effects of the intra-agent communication structure for a simulated case of an industrial compressed air system. The results show that the communication structure influences resilience but not robustness. Robustness is improved through the learning mechanism in which the agents learn to predict the effects of their actions on the behaviour of the nearby system constituents. The presented agent model enables a smart controller to operate in a system without prior knowledge of the effects of its actions on the controlled variable. However, this paper shows that to achieve both robustness and resilience of the multi-agent control system, appropriate communication structure of the network must be implemented. Future work will focus on the development of a real demonstrator and transfer of the learned policies from simulation to the demonstrator. 8. REFERENCES [1] Monostori, L., Kádár B, Bauernhansl, D., Kondoh, S., Figure 5. Pressure values measured by sensor 𝑺𝑪𝟏 after Kumara, S., Reinhart, G., Sauer, O., Schuh, G., Sihn, W., disturbances, green triangles show the average value and and Ueda, K. 2016. Cyber-Physical Systems in green dots represent the outliers Manufacturing. CIRP Annals – Manufacturing Technology The results for evaluation of robustness and resilience from data 65(2):621–641. measured by sensor SC1 are shown in Figure 6. [2] Russell, S. J., and Norvig, P. 2010. Artificial intelligence: A modern approach (3rd ed.). Upper Saddle River, N.J. ; New Delhi: Prentice Hall/Pearson Education. [3] Bruzzone, A.A.G., and D’Addona, D.M. 2018. New Perspectives in Manufacturing: An Assessment for an Advanced Reconfigurable Machining System. Procedia CIRP 67: 552-557 [4] Shen, W., Hao, Q., Yoon, H.J., and Norrie, D.H. 2006. Applications of agent-based systems in intelligent manufacturing: An updated review. Advanced Engineering Informatics 20: 415 – 431. [5] Vrabič, R., Kozjek, D., Malus, A., Zaletelj, V., and Butala, P.: Distributed control with rationally bounded agents in cyber-physical production systems. CIRP Annals – Manufacturing Technology 67(1): 507 – 510, 2018. Figure 6. Robustness 𝒄𝒓𝒐𝒃 and resilience 𝒄𝒓𝒆𝒔 during [6] Rubinstein, A. 1998. Multiagent Systems: Algorithmic, disturbances Game-Theoretic, and Logical Foundations. MIT Press. As seen in Figure 6a, robustness improves over time in all [7] Simon, H.A. 1972. Theories of Bounded Rationality. scenarios. As agents learn to create better predictions of the Decision and Organization 1(1):161–176. effects of their actions, their responses improve. This is similar in all simulated scenarios, suggesting that it is not influenced by the [8] Breiman, L. 2001. Random Forests. Machine Learning, 45: communication structure. 5–32. Figure 6b shows that resilience improves over time in scenarios [9] Radgen, P. 2004. Compressed Air System Audits and C, D, and E, in which the supply and the demand are connected. Benchmarking. Results from the German Compressed Air The resilience decreases in scenarios A and B where there is no Campaign "Druckluft effizient". In: Proceedings of communication between compressors and smart valves. In ECEMEI, Third European Congress on the Economics and scenarios A and B, the agents on the demand side cannot directly Management of Energy in Industry, Rio Tinto, Portugal, detect the pressure drop which happens due to the disturbance. April 6-9. Their response is based solely on the observation of the pressures [10] Saidur, R., Rahim, N.A., Hasanuzzaman, M.: A Review of on the demand side. In time, they learn to prefer to keep the valves Compressed-Air Energy Use and Energy Savings. closed in order to maintain the pressure on the consumers during Renewable and Sustainable Energy Reviews 14(4):1135– the disturbance. However, since they cannot detect the end of the 1153, 2010. disturbance directly, they have a delayed response when conditions normalize which lowers the overall resilience. 16 Complex Decision Rules in DEX Methodology: jRule Algorithm and Performance Analysis Adem Kikaj Marko Bohanec Jožef Stefan International Postgraduate School Jožef Stefan Institute, Department of Knowledge Jožef Stefan Institute, Department of Knowledge Technologies Technologies Jamova 39, 1000 Ljubljana, Slovenia Jamova 39, 1000 Ljubljana, Slovenia marko.bohanec@ijs.si adem.kikaj@ijs.si ABSTRACT analysis of decision alternatives. DEX is supported by software DEX (Decision EXpert) is a qualitative multi-criteria decision- DEXi (http://kt.ijs.si/MarkoBohanec/dexi.html). modeling methodology. DEX models are used to evaluate and DEX models have a hierarchical structure, which represents a analyze decision alternatives. An essential component of DEX decomposition of some decision problem into smaller, less models are decision rules, represented in terms of decision tables. complex sub-problems. DEX models are developed by defining Decision tables may contain many elementary decision rules and (i) attributes, (ii) scales, (iii) hierarchically structured attributes may be difficult to be understood by the decision maker. A more (the tree of attributes), and (iv) decision rules. In DEX models, compact and comprehensible representation is obtained by attributes are variables that represent properties of decision converting elementary decision rules to complex rules. The DEX- alternatives. Attributes can be either basic or aggregated. Rule algorithm, which is currently implemented in software Aggregated attributes have subordinate attributes, while basic DEXi, has been found inefficient with large decision tables. This attributes do not. Basic attributes represent inputs and aggregate research is aimed at improving the efficiency of the DEX-Rule attributes represent outputs (results). A scale represents a set of algorithm. We propose a novel algorithm, called jRule, which values that can be assigned to an attribute. Scales are qualitative generates complex rules by specialization. According to and can take discrete values like ‘excellent’, ‘acceptable’, performance analysis, jRule is indeed more efficient than DEX- ‘inappropriate’, etc. Decision rules represent the mapping of Rule. The compactness of complex rules produced by both subordinate attributes to an aggregated attribute (see section 2 on algorithms varies and there is no clear winner. more details about decision rules in DEX). Categories and Subject Descriptors In a DEX model, an aggregated attribute may involve many H.4.2 [Types of Systems]: Decision Support subordinate attributes (e.g., more than five) in which case the decision table will contain many elementary decision rules and F.2.0 [General] may be difficult to be understood. In order to obtain a more comprehensible representation, the DEXi software implements General Terms DEX-Rule, an algorithm that converts elementary decision rules Algorithms, Performance, Experimentation to more compact complex rules. DEX-Rule has been found inefficient in decision tables with many subordinate attributes and many elementary decision rules that map to a single decision Keywords value. DEX methodology, decision rules, complex decision rules, algorithm analysis This research is aimed at improving the efficiency of the DEX- Rule algorithm. We propose a novel algorithm, called jRule, 1. INTRODUCTION which finds complex rules by specialization, i.e., by narrowing down too general rules that are constructed initially. The jRule Decision-making is a difficult and complex process. During this algorithm performed better regarding the running time. The process, a decision maker (DM) faces several decision results generated by both algorithms are guaranteed to cover the alternatives. To choose a particular alternative from the set of whole decision table. possible alternatives, a decision-analysis approach [3, 4] can help to satisfy the aims or goals of a decision maker. Decision analysis This paper is structured as follows: Section 2 formulates the [3, 4] is the discipline used to help a decision maker to deal with Decision Rules in DEX, Section 3 presents the DEX-Rule uncertainty, complexity, risk, and trade-offs of the decision. The algorithm, Section 4 presents the jRule algorithm, Section 5 idea of decision analysis is to develop a decision model, which presents the comparison of the two algorithms regarding the can help decision makers to evaluate alternatives and to choose algorithm complexity and the number and form of complex the best action. decision rules that they generate. Section 6 summarizes and concludes the paper. The decision maker in a decision problem have to deal with multiple and possibly conflicting criteria. Multiple Criteria Decision Analysis (MCDA) or Multiple Criteria Decision Making 2. DECISION RULES IN DEX (MCDM) [3] provides methods for structuring, planning and In DEX models, attributes can be either basic or aggregated. solving such decision problems. DEX methodology is one of the Aggregated attributes are attributes which depend on their MCDM methods. DEX is a qualitative multi-criteria decision- descendants, known as subordinate attributes. Decision rules in making methodology [1, 2, 5] aimed at the assessment and DEX define the bottom-up mapping of the scale values of 17 subordinate attributes to the values of the aggregated attribute. An example of such mapping, represented in terms of a decision table, is shown in Table 1. The example is taken from a well- known model for evaluating cars based on attributes such as buying price, maintaining price, safety, and comfort [1]. The example occurs at the top level (root) of the model and maps the subordinate attributes PRICE and TECH.CHAR (technical characteristics) to the overall evaluation of a CAR. The value scale of the involved attributes are ordered values as follows:  PRICE = {high, medium, low},  TECH.CHAR = {bad, acc, good, exc}, and  CAR = {unacc, acc, good, exc}. Each row in Table 1 defines the value of the aggregated attribute Figure 1. Elementary decision rules represented in a matrix. CAR for each combination of subordinate attributes’ values. Therefore, the decision table maps all the combination of PRICE In order to represent the decision table in a more compact and and TECH.CHAR scale values into the value of CAR. possibly comprehensible way, DEX uses complex decision rules. A complex decision rule consists of the condition and decision Table 1. Decision table with elementary decision rules of DEX value part. In contrast with elementary rules, each clause in the model known as CAR Evaluation Model [1]. condition part can represent an interval. The decision value is PRICE TECH.CHAR CAR always a single value. Thus, a complex rule generally takes the 1 high bad unacc form: 2 high acc unacc if subAttr 1  [ low_value 1 , high_value 1] 3 high good unacc and subAttr 2  [ low_value 2 , high_value 2] 4 high exc unacc … 5 medium bad unacc and subAttrn  [ low_valuen , high_valuen] 6 medium acc acc then aggAttr = value 7 medium good good 8 medium exc exc For comprehensibility, DEXi software traditionally represents 9 low bad unacc intervals as follows: 10 low acc good  ‘ *’: the asterisk include all possible scale values of a 11 low good exc specific subordinate attribute; 12 low exc exc  ‘ >=w’: stands for better than or equal to value;  ‘<= w’: stands for worse than or equal to value; A decision rule consists of the condition and decision part:  ‘ w if subAttr 1 :w 2’: interval between value w 1 and value w 2 , 1 = value 1 including the two values. and subAttr 2 = value 2 … Figure 2 shows several complex decision rules on the matrix from and subAttrn = valuen Figure 1. It is important to notice that each complex decision rule then aggAttr = value (or interval of values) covers an area that corresponds to one or more elementary decision rules. In this way, the number of complex rules that The condition part is the Cartesian product of the scale values of completely cover the matrix is generally lower than the number of the subordinate attributes of an aggregated attribute ( subAttr 1, elementary rules, and the resulting representation is more subAttr 2, …, subAttrn). The decision-maker defines the value of compact. each decision rule, which might be a single value or an interval of values of the aggregated attribute. Such decision rules are also called elementary decision rules, since each rule defines the value for exactly one combination of subordinate attributes’ values. In this way, the first row in Table 1 represents the following elementary rule: if PRICE = high and TECH.CHAR = bad then CAR = unacc An alternative representation of the decision rules can be by an n- dimensional matrix, depending on the number of subordinate attributes. Figure 1 shows such a representation of Table 1. Here, each cell of the matrix represents one elementary decision rule from the decision table. Figure 2. Complex decision rules represented in a matrix through different dotted rectangles for each decision value. 18 3. DEX-RULE ALGORITHM DEX-Rule is an algorithm currently implemented in DEXi [1] that converts elementary decision rules into more compact complex decision rules. The DEX-Rule generates complex decision rules by finding areas limited by bounds, which may cover more than one elementary decision rule. An area is represented by two bounds: a low and a high bound. Both are vectors of scale values of the subordinate attributes. The input to the DEX-Rule algorithm is a decision table, represented in a form of a decision matrix, such as in Figure 1. All the rules are marked as uncovered. The low and high bound ( l and Figure 3. Two cases of generating complex decision rules with h) are vectors (coordinates) that define an area of decision rules DEX-Rule algorithm. with the target value t. Initially, l = h, which means that they define a single elementary decision rule. Later, with recursive 4. JRULE ALGORITHM invocation of the algorithm, these boundaries are gradually The aim of this research was to improve the efficiency of the extended to cover larger areas with the target value t. On the DEX-Rule algorithm. We propose a novel algorithm, called jRule. output, DEX-Rule generates a set of decision rules, such as in the While the main idea behind DEX-Rule is to find areas by example shown in Table 4. DEX-Rule proceeds by considering all generalization (extending the area bounds), the main idea of the target decision values, t, in succession. For each t, DEX-Rule jRule is to reverse this method and use specialization. jRule proceeds by generalization, as shown in Algorithm 1. proceeds by finding largest areas covering yet uncovered rules for Algorithm 1. Pseudo-code of the DEX-Rule Algorithm. t and gradually reducing them. Inputs: Algorithm 2. Pseudo-code of the jRule Algorithm. l := low bound. Inputs: h := high bound. t := target decision value. t := target decision value. ger := elementary decision rules for target value t, m := last elementary decision rule from decision table lexicographically sorted by subordinate attribute values. (representing the highest current bound). Outputs: Outputs: p := complex decision rules p := complex decision rules begin begin l := lowest subordinate attributes’ values from ger cover := ValidateBounds( l, h, t) for i = | ger| to 0 if cover then if ! ger[ i].isCoveredBy( p) then for i = 0 to | h| do lb = l if h[ i] < m[ i] then hb = ger[ i] DEXRule( l, Increase( h), t, m) while !ValidateBounds( lb, hb, t) do end if lb = Increase( lb) // reduce the area by increasing the lb end for end while for i = 0 to | l| do p.add( lb + hb) if l[ i] > 0 then end if DEXRule(Decrease( l), h, t, m) end for end if end end for p.add( l + h) The pseudo-code of the jRule algorithm is shown in Algorithm 2. end if First, the algorithm finds l, the lowest bounds for each end subordinate attribute of elementary rules for the target value t. Then, it locates the last (i.e., highest) currently uncovered For each decision value t and each elementary decision rule that elementary rule. This gives the high bound of the area. If the area has not been covered so far (represented by l and h, l = h), DEX- with bounds lb and hb is valid, meaning that covers only rules for Rule tries to extend the boundaries l and h in different directions. t, a new complex rule is generated. Otherwise, this area is reduced When the area cannot be extended any more, a complex decision by increasing the low bound lb. This process is repeated until all rule is created. More precisely, a complex decision rule is elementary rules for t have been covered by complex rules. Notice generated in two cases: that, unlike DEX-Rule, areas in jRule are gradually reduced by  when the algorithm reaches the highest or lowest scale increasing only the low bound lb. Figure 4 illustrates this process value for the specific subordinate attribute, see Figure for elementary rules shown in Table 1 and the target value t = 3.a, or unacc. In this case, ger is composed of rules 1, 2, 3, 4, 5 and 9 (in  when an extension would cover an elementary decision this order) from Table 1. The low bound is lb = . rules with a different target value, see Figure 3.b. jRule makes two iterations, first finding the high bound from rule The process continues until the matrix has been completely 9 ( hb = ) and then from rule 4 ( hb = ). In covered by complex rules. both cases, the areas cover t and no reduction is necessary. 19 The two algorithms, in general, produce different complex decision rules. For example, Tables 5 and 6 show the respective complex rules for the CAR evaluation model. The rules are very similar, there is only a small difference in rule 7. In some other cases (Table 4), the differences between the algorithms are more pronounced: jRule produces more compact representations for Models 1 and 2, but less compact for Model 3. More research is needed to establish which algorithm is better and under which circumstances. Table 6. Complex decision rules generated by jRule for CAR aggregated attribute of CAR Evaluation model. # PRICE TECH.CHAR CAR 1 high * unacc Figure 4. jRule Algorithm identifying the lowest and highest 2 * bound for elementary decision rules with decision value t = bad unacc unacc. 3 medium acc acc 4 medium good good 5. PERFORMANCE ANALYSIS 5 low acc good 6 >=medium exc exc The comparison between the DEX-Rule and jRule algorithms is 7 low good exc made with respect to (i) time complexity, (ii) the running time and (iii) the number of complex rules that these two algorithms 6. CONCLUSIONS generate. In this work, we proposed a novel algorithm jRule for converting Regarding the time complexity, the DEX-Rule algorithm is O(mn) elementary decision rules to complex decision rules in the DEX because of its recursive nature, where m is the number of methodology. In contrast with the current DEX-Rule algorithm, subordinate attributes and n is the number of the elementary which employs generalization, jRule uses the principle of decision rules. On the other hand, the time complexity of the specialization. jRule algorithm is O(n2m). Regarding the time complexity and running time, jRule algorithm The experimental comparison of the algorithms is based in perform better than DEX-Rule in all experiments performed for different DEX models for different aggregated attributes. Both different DEX models. On the other hand, none of the algorithm algorithms are implemented in JDEXi was clearly better with respect to the number of generated (http://kt.ijs.si/MarkoBohanec/jdexi.html) and DEX.NET2 complex decision rules. As part of this work, both algorithms (http://kt.ijs.si/MarkoBohanec/dexinet.html). The algorithms were were implemented in two open source libraries, JDEXi V4 and compiled with the same compiler and run on the same computing DEXi.NET2. environment. Table 4 shows running times of the algorithms on three selected DEX models. Generally, jRule is more efficient, 7. ACKNOWLEDGMENTS and a major difference occurs with Model 2, which is a large The Public Scholarship, Development, Disability and decision table having five subordinate attributes and 1728 Maintenance Fund of the Republic of Slovenia (Contract no. elementary decision rules. 11011-44/2017-14) financially supported this research. Table 4. Difference between two algorithms based on running time and number of generated complex rules. 8. REFERENCES # Running time [s] # of complex decision rules [1] Bohanec, M. (2015). DEXi: Program for Multi-Attribute Models DEX-Rule jRule DEX-Rule jRule Decision Making User's Manual. Ljubljana, Slovenia: Institut Jozef Stefan. 1 0.75 0.200 30 18 2 1280.00 0.395 121 64 [2] Bohanec, M., Žnidaršič, M., Rajkovič, V., Bratko, I., & 3 1.94 0.980 11 26 Zupan, B. (2013). DEX methodology: three decades of qualitative multi-attribute modeling. Informatica, 37(1). Table 5. Complex decision rules generated by DEX-Rule for [3] Bouyssou, D., Marchant, T., Pirlot, M., Tsoukias, A., & CAR aggregated attribute of CAR Evaluation model. Vincke, P. (2006). Evaluation and decision models with multiple criteria: Stepping stones for the analyst (Vol. 86). # PRICE TECH.CHAR CAR Springer Science & Business Media. 1 high * unacc [4] Greco, S., Figueira, J., & Ehrgott, M. (2016). Multiple 2 * bad unacc criteria decision analysis. New York: Springer. 3 medium acc acc 4 medium good good [5] Trdin, N., & Bohanec, M. (2018). Extending the multi- 5 low acc good criteria decision making method DEX with numeric 6 >=medium exc exc attributes, value distributions and relational models. Central 7 low >=good exc European Journal of Operations Research, 26(1), 1-41. 20 Sensitivity Analysis of Computational Models that Dissolve the Fermi Paradox Jurij Nastran Beno Šircelj "Jožef Stefan" Institute "Jožef Stefan" Institute Jamova cesta 39 Jamova cesta 39 Ljubljana, Slovenia Ljubljana, Slovenia jurij.nastran@gmail.com beno.sircelj@gmail.com Drago Bokal Matjaž Gams University of Maribor "Jožef Stefan" Institute Koroska cesta 46 Jamova cesta 39 Maribor, Slovenia Ljubljana, Slovenia drago.bokal@um.si matjaz.gams@ijs.si ABSTRACT Why is there no evidence of other civilizations in our galaxy Given the estimated number of stars and planets in our despite the sheer number of planets we now know of? This galaxy, the probability of existence of intelligent civilizations is the question physicist Enrico Fermi first asked in 1950 seems high. The first to indicate this was the Drake equation and is known as the Fermi paradox. Fermi was particularly with the assumed default parameters. Yet, the actual obser- effective when dealing with estimates of ill-defined physical vations have yet to reflect those expectations. This discrep- phenomena. However, he did not coin the first universally ancy corresponds to the so-called Fermi Paradox. Although accepted equation for an estimate of the number of civiliza- many key factors about the likelihood of alien civilizations tions. still remain largely unknown, new methods of estimating the probability are being proposed. Some of them use prob- 1.1 Drake equation ability distributions and the Monte Carlo methods. In this Probably the best known equation for an estimate of the paper we recalculate one of those – the Sandberg interpre- number of detectable civilizations in the Milky Way was tation of the Drake equation, analyze the difference between composed by Frank Drake [3], phrased as a product of seven the methods, their strengths and weaknesses. In the conclu- factors: sion, we find that the probability distribution better reflects our ignorance about the properties of alien environments N = R∗fpneflfifcL than the dot-product method. The parameters are: R In our opinion, there are several ways to further improve ∗ which is the rate of star formation per year, f the computational model based on the Drake equation us- p is the fraction of stars with planets, ne is the number of Earth-like (or otherwise habitable) planets per a ing the AI methods, thus eliminating the problem with too star that has planets, f small probabilities and introducing 3D views. l is the fraction of habitable planets with actual life, f What makes these analysis relevant, is not only the number i is the fraction of life-bearing planets that develop intelligence, f of estimated civilizations in our galaxy and a probability c is the fraction of intelligent civiliza- tions that are detectable and L is the average longevity of that we encounter them in the near future. More impor- such civilizations. finally N is the number of detectable civ- tant, these models enable estimation of the life-span of the ilizations. human civilization. Unfortunately, there is a considerably The Drake equation is used to directly estimate the number high probability that it will be quite short. of civilizations or as an analysis tool for various components in our galaxy. Most importantly, it can also be used to esti- Keywords: Drake equation, Fermi paradox, Monte Carlo, mate the lifespan of our civilization (L). The Drake equation Probability distribution, Extraterrestrial intelligence can provide exact numbers given proper parameters (i.e. fac- tors), but the problem is that several factors in the equation 1. INTRODUCTION are not well determined either by observations or with lab- oratory models. By assuming different values for them, say There are billions of stars in the observable universe. Hun- f dreds of millions are broadly estimated to be in our galaxy l – probability of life on a habitable planet, results vary a lot [1]. alone. If there is at least a modest chance of intelligent life emerging on a given planet, surely there should be at least some number of alien life in the relative vicinity, yet we see 1.2 Point estimates none. We are apparently the only one in our part of the uni- When trying to estimate the number of civilizations in the verse even though we were able to expand our search quite galaxy, a point estimate is often used for each of the seven successfully in recent decades [5]. parameters of the Drake equation. This provides an exact 21 numerical value. If we take estimates based on the distribu- magnitude given other galaxies. Our uncertainty about this tions in Sandberg’s paper [6], we get: R∗ = 10, fp = 0.3, ne parameter is from 2 to 16 solar masses, LU[R∗] = 0.9. = 0.3 , fl = 0.5, fi = 0.03, fc = 0.1, L = 106 which gives us N – Fraction of stars with planets fp is also pretty well known around 270. Drake with colleagues originally estimated that and is about 1 with LU[fp] = 1. L = N and probably between 1000 and 100.000.000. Cur- – The estimates for ne, which is number of habitable plants, rent best estimates[4] differ from several hundreds to several range from <10−12 in rare earth arguments to >1 when tak- millions for the civilization’s lifespan, and from being alone ing non terrestrials like icy moons into account. Sandberg to several millions civilizations in our galaxy. proposed LU[ne]=12. Post-2000 literature estimates cover Based on actual observations, it seems quite likely that the smaller orders of magnitude so they postulated earth-like optimists overestimate their factors: If an advanced civi- planet as rocky planet within habitable zone and assumed lization appeared somewhere in the galaxy before, moving LU[ne]≈2. with the speed of one percent of light speed, it would reach – The parameters with the most uncertainty are fl and fi. all parts of our galaxy in less than 2o million years. This fl (probability of life) is modeled as a physical transition is a tiny fraction of the lifespan of our galaxy, which is as that occurs at some rate per unit time per unit volume of a old as our universe, i.e. around 13.5 billion years, with the suitable prebiotic substrate. The probability on a habitable perimeter around 100 - 200.000 light years. planet with volume V, time period t and abiogenesis rate λ is fl = 1 − e−λV t. They take log-uniform distributions of 2. SETTING THE PARAMETERS AND t with LU[t]=3 and range from 107 to 1010, V with LU[V ] ≥ 20 range from 10−35 to 1015. They use log-normal dis- COMPUTATIONAL MODELS tribution for λ with a mean of 1 and σ of several orders of Sandberg and colleagues [6] suggested that using point esti- magnitude. In the paper they do not specify the exact num- mates to solve the Drake equation is too wild a guess, pro- ber used for σ, just that its very big. When we recreated viding only one number. In their paper they suggest an ap- their experiment we tested several values for sigma and 200 proach that models each parameter by it’s distribution, thus gave the most similar results. the computation results in a probability distribution. They – Based on the literature, the parameter fc, which is the frac- used a Monte Carlo method for calculating the distribution tion of planets which develop civilization, is between 0.001 of the final result. During each iteration, they sampled from and 1. distributions to obtain point estimates to use in Drake equa- – The final factor L, is longevity of a space-communicating tion, which results can be used to generate final distribution civilization in years and is in the range from 100 to 1010, of N. which is the current estimate of the age of the universe. 2.1 A toy model All parameter(factor) distributions are listed in Table 1. To show how using distributions for calculations leads to re- With this set of parameters we managed to obtain a dis- sults that differ from those obtained by simply multiplying tribution for N, displayed in Figure 1, quite similar to the point estimates, Sandberg and colleagues introduced a sim- one by Sandberg and colleagues. The two axes in Figure 1 plified toy model. In their toy model to demonstrate the represent N and its corresponding frequency with blue color. differences, there are 9 parameters (f1,f2,. . . ), which if mul- A vertical black line marks N equal to 1. The blue graph is tiplied together determine probability of ETI (extraterres- therefore PDF (probability density function). It is scaled so trial intelligence) on a single star. Each of those parameters that the highest value is 1 to fit on the same graph as the can obtain values from an interval [0, 0.2], with an average red line which is CDF (cumulative density function). From of 0.1. In their case-example, the point estimate of each Figure 1 it can be observed that the probability of us be- factor is set to 0.1, the same as the average from the inter- ing alone in our galaxy is about one half, e.g. since the red val. For 100 billion stars (as in our galaxy) the computation graph reaches 0.5 around N equal to 1. gives 100 intelligent civilizations. But if instead of using a point estimate we sample from a uniform distribution with an average of 0.1, we get the results indicating that there is 21.45% chance that we are alone in the galaxy. 2.2 Recomputing the Sandberg interpretation of Drake equation Sandberg et al. applied probability distribution as a way of recalculating the Drake equation, but instead of a uni- form distribution presented in Subsection 2.1, they used the probability distribution for each factor obtained from scien- tific literature – a range and the type of the distribution. They defined a parameter ”log-uncertainty” of a parameter X (LU[X]= log( max(X) )) as an estimate of the number of min(X) orders of magnitude of the current uncertainty of parameter Figure 1: Recomputation of the Sandberg approach: X. Consequently, the factors were defined in the following probability density function and cumulative density way: function of N computed with Monte Carlo method. – Star formation rate R∗ is fairly well constrained by as- tronomical data and ranges over a maximum of 5 orders of 22 Parameter Distribution R∗ log-uniform from 1 to 100 fp log-uniform from 0.1 to 1 ne log-uniform from 0.1 to 1 fl log-normal rate, described above fi log-uniform from 0.001 to 1 fc log-uniform from 0.01 to 1 L log-uniform from 100 to 10,000,000,000 Table 1: Summary of the current knowledge about the parameters of the Drake equation. Figure 2: Graphs for Toy model with 3 parameters and 9 parameters. 3. ANALYSIS OF THE SANDBERG’S COM- PUTATIONAL MODEL We tested this phenomenon on multiple number of parame- In this section, we present our initial analysis of the Sand- ters ranging from 1 to 9 and the result can be seen in Figure berg et al. computational approach. We tested various is- 3. We can see that by increasing the number of parameters sues only on the toy model as it is simpler and better reflects the PDF widens, thus increasing the possibility of us being the issues with the computational model. We hypothesized alone. that if one parameter is assigned a number close to zero, the whole product will be a very low number no matter what the other parameters are. For example, the blue graph in Figure 1 might give a misleading impression that the major- ity of the possibilities is on the right side of the black line indicating 1. However, due to the logarithmic x scale, it is about the same possible that there is only 1 civilization in our galaxy compared to 2 or more, as the red graph shows. The probabilities on the left of the black line indicating 1 are therefore significantly higher. The closer to 0, the higher, meaning lots of Ns are strangely close to 0. From the AI field we are familiar with this problem from the analysis of the naive Bayer theorem: if one of the factors is 0 or close to 0, it should better be modified. We tried to analyze this hy- pothesis by calculating the same results by using a different number of parameters and by using ranges with small offset to move away from zero. Namely, if any of the parameters Figure 3: Probability of there being no civilizations is zero, then there is no intelligent civilization in that galaxy. depending on the number of parameters (1-9) 3.1 Effect of small values with multiple pa- 3.2 Modifying small values with epsilon rameters We tried to minimize the effect of parameter/factor values being too close to zero, which seemingly spoils the calcula- Tu study the effect of small values of parameters/factors tion, by introducing an offset . Instead of the range [0, 0.2] in the product, we varied the number of parameters. To we now try the same experiments with range [0 + , 0.2 − ]. normalize the result, we adjusted the number of stars so that The offset on the left side ensures that the minimum value the average value of N according to the distribution obtained is increased and the offset on the right side is there to keep by the Monte Carlo analysis is still 100 as it is in the original the same average. As we can see in Figure 4 for [0.02, 0.18], case of 9 parameters and 1011 stars. So the number of stars the computed N changes significantly. In Figure 5, the same is 102+Np where N p is the number of parameters. All the effect is demonstrated for values from 0.00 to 0.040 by a blue parameters are in the range [0, 0.2]. The graph showing 3 line. By increasing the , the curve becomes narrower thus parameters and 9 parameters can be seen in Figure 2. The chances of us being alone decrease. solid lines are PDFs and the dotted lines are CDFs. The PDF is scaled like before so that the maximum value is 1. In both cases the average value is still 100. With green dot 3.3 Effects of log-uniform distributions we mark the probability that there is no civilizations in the The toy model had all its parameters distributed uniformly. toy galaxy. Note that the size of our toy galaxy also varies We recreated the same experiments, but with log uniform with the number of parameters to enable comparison. The distribution of parameters, which is the same type of dis- dot is on the CDF at value of one civilization in the galaxy. tribution as in the Sandberg paper. In Figure 5 one can It can be seen that the value at 9 parameters is higher than see that the effect of adding epsilon in the log distribution the value at 3 parameters. Therefore, the more parameters, modifies the graph even more. Figure 6 represent another the more probability of being alone. Or in other words: analysis with the toy problem and various epsilons, using the the more uniformly distributed parameters one introduces log uniform distribution. This is another indication that the in a product, the more likely small values of N if parameter small values of parameters introduced by probability distri- values are in [0,1]. bution by Sandberg strongly influence the final result. 23 Figure 6: How affects distributions in log-space. Figure 4: The effect of on the probability distribu- tion 5. CONCLUSION AND DISCUSSION We first showed how the number of ETIs was estimated us- ing the original Drake equation, and then with the advanced approach by Sandberg and colleagues. Our re-computation of the Sandberg method yielded nearly the same results. When reanalyzing the Sandberg approach, one issue emerged - multiplying with small values. We analyzed this phenom- ena and found out that with increasing number of parame- ters the probability gets closer to zero. We also introduced some corrections to the ranges of parameters. While it is unlikely that the computing mechanisms will solve the Fermi problem on its own, they can provide better understanding of the current and future observations. The computational models and universe observations progress, increasing our knowledge and narrowing the unknowns in Figure 5: Effect of is enhanced with log-uniform the estimation of the number of civilizations and the lifespan distributions. of our civilization. However, until we meet another civiliza- tion or none for a long time or, as the third and most tragic option - our civilization decays, there is still a long way to go. 4. DISCUSSION: AI-BASED MODIFICATIONS 6. ACKNOWLEDGMENTS The proposed and to a certain extend already tested AI- We would like to thank Filip Talimdzioski for his advice and based modifications are to be fully described in the submis- cooperation. sion of an SCI paper. Some of the ideas will be presented live at the paper presentation. Here, a couple of hints are 7. REFERENCES presented here. [1] J. Achenbach and P. Essick. Life Beyond Earth. The first observation when reanalyzing the Sandberg ap- National Geographic, 197(1):24–51, 2000. proach is that the it is significantly more informative than [2] B. Cestnik et al. Estimating probabilities: a crucial point estimates. Unlike providing just one number based task in machine learning. In ECAI, volume 90, pages on assumptions, it shows the whole probability distribution, 147–149, 1990. i.e. all possible combinations of values of parameters. On the second thought, the computational model has a weak- [3] F. Drake. The Drake Equation: Estimating the ness - when multiplying with zero or very small values. In Prevalence of Extraterrestrial Life Through the Ages. the Drake equation, several factors are multiplied together, Cambridge University Press, 2015. but originally, none of them was very close to 0. However, [4] F. Drake and D. Sobel. Is anyone out there?: The when dealing with probability distributions, numbers close scientific search for extraterrestrial intelligence. to 0 can appear and as a result N becomes very small. We Delacorte Press, 1992. presented this effect in the Toy problem with uniform distri- [5] J. Gribbin. Are Humans Alone in the Milky Way? butions, and later with log uniform distributions where the Scientific American, September 2018. effect intensified. [6] A. Sandberg, E. Drexler, and T. Ord. Dissolving the In AI, there have been similar problems when multiplying Fermi paradox. arXiv preprint arXiv:1806.02404, 2018. with small values and solutions. For example, several anal- yses of the Naive Bayes were performed for the case when one of the factors was zero. One of the first analysis was by Cestnik [2]. By modifying the Cestnik or Laplace approach for the Sand- berg method, we managed to introduce important modifica- tions. 24 Context-Aware Stress Detection in the AWARE Framework∗ Marija Trajanoska Marko Katrašnik Junoš Lukan Faculty of Electrical Engineering and Jožef Stefan Institute Jožef Stefan Institute Information Technologies Jamova cesta 39 Jamova cesta 39 Ss. Cyril and Methodius University 1000 Ljubljana, Slovenia 1000 Ljubljana, Slovenia 1000 Skopje, Macedonia marko.katrasnik@gmail.com junos.lukan@ijs.si marijatrajanoska@gmail.com Martin Gjoreski Hristijan Gjoreski Mitja Luštrek Jožef Stefan Institute Faculty of Electrical Engineering and Jožef Stefan Institute Jamova cesta 39 Information Technologies Jamova cesta 39 1000 Ljubljana, Slovenia Ss. Cyril and Methodius University 1000 Ljubljana, Slovenia martin.gjoreski@ijs.si 1000 Skopje, Macedonia mitja.lustrek@ijs.si hristijang@feit.ukim.edu.mk ABSTRACT to be extended by plugins. There are already several plugins Physiological signals are good predictors of stress, which can available such as a Google activity recognition plugin, which be thought of as part of a user’s context. In this work, an captures the users’ mode of transportation, and a Fitbit option to combine the user’s stress level with other contextual plugin, which enables collecting data such as heart rate and factors is presented. This is done in the form of two AWARE sleep duration from a Fitbit device. plugins – Android applications that can be incorporated In this work, a state-of-the-art stress detection method is im- into a smartphone monitoring setup. In the first part, the plemented as an AWARE plugin. To make this possible, the stress detection method is described, which consists of a lab method was adapted to real-time operation, being previously stress detector, an activity classifier, and a context-aware only used offline. The plugin classifies the user’s physiological stress model. In the second part, two plugins are described. data as representing a stressful or a non-stressful condition, One streams the data from the Empatica E4 wristband after receiving the data from the Empatica wristband via and the other one uses this physiological data to predict another plugin. Both plugins are planned to be released stress. Finally, some possibilities to improve this work are publicly, so that other researchers will be presented with a presented. ready-made solution for the first time. In Section 2, we first present the stress-detection method and Keywords what data is needed for it. In Section 3 our implementation in AWARE, plugin, stress detection, Empatica E4, physiology the AWARE framework is presented: a plugin for the stress detection model itself (Section 3.2) and a plugin for data 1. INTRODUCTION collection (Section 3.1). Finally, some possible improvements are outlined in Section 4. Mental stress is most often researched because of its negative health consequences when it is chronic. The ability to detect stress from physiological signals collected with a wearable 2. CONTEXT-AWARE STRESS DETECTION device is thus valuable for research in situations when stress METHOD occurs, as well as to trigger stress-relief interventions. In addition, stress affects one’s short-term psychological state The stress-detection AWARE plugin is based on a real-life and behaviour, which makes it a part of the user’s context stress detection method as described by Gjoreski et al. [7], as understood in ambient intelligence. Detecting stress is and the more general context-based reasoning framework therefore also valuable for adapting intelligent services to the introduced by Gjoreski et al. [5]. It consists of three separate user (e.g., a mobile application may postpone non-essential machine learning components: a laboratory stress detector, notifications when the user is stressed). In this paper we an activity recognition classifier, and a context-based (real- present a stress-detection plugin (and its prerequisite – the life) stress detector. Each is presented in its own subsection. plugin for the Empatica sensing wristband) for the AWARE framework [3]. This makes stress detection easily accessible 2.1 Lab Stress Detection to researchers and other interested parties. The lab stress-detection model was trained using data ob- AWARE is an Android framework, used to capture the phone tained in a laboratory experimental setup during a standard- sensors’ data to infer context. Its modular nature enables it ized stress-inducing experiment [7]. The main stressor in ∗The work presented in this paper was carried out as a part of this experiment was solving a mental arithmetic task un- a project funded by the Slovenian Research Agency (ARRS, der time and evaluation pressure. The laboratory data was project ref. N2-0081). then labelled taking into account both the difficulty of an 25 equation-solving session (easy, medium or hard) and short The context-based stress detection classifier was trained us- STAI-Y anxiety questionnaires [8] filled out by the partici- ing the data obtained as part of the real-life experimental pants. According to this information, the data was classified setup described in Gjoreski at al. [7] The data duration into three degrees of stress: no stress, low and high. Addi- totalled to 1327 h and involved 5 participants who wore Em- tionally, baseline no-stress data was recorded on a separate patica E4. The labelling process involved a combination of day when the participants were relaxed. a stress log and Ecological Momentary Assessment (EMA) prompts implemented on a smartphone. For the stress log, For the creation of the laboratory stress-detection classifier, the participants logged the start, duration, and intensity (on the machine-learning pipeline involves segmentation, signal a scale from 1 to 5) of everyday stressful situations. The filtering, feature extraction, model learning, and evaluation EMA prompts were additionally displayed randomly 4 to of the models. 6 times throughout the day, with at least 2 hours between Segmentation refers to the partitioning of the data into consecutive prompts. windows for the purposes of feature extraction. According The labelled data was then windowed using non-overlapping to the windowing experiments, which provide a performance windows lasting 10 min, since aggregation experiments showed comparison between models for varying data window sizes, that most algorithms perform better for smaller aggregation the optimal data window size was 5 min with 2.5 min overlap. windows (10 min to 17.5 min) as compared to larger ones. The signals obtained from the Empatica sensors are: blood The context-based stress detector’s input is four-fold: volume pulse (BVP), interbeat intervals (IBI), heart rate (HR), electrodermal activity (EDA) and skin temperature • context features, (TEMP). After filtering the signals to reduce noise, numerical • features extracted from the output of the activity- features are extracted from each data window using statis- recognition model, tical functions, regression analysis, and frequency and time • features extracted from the output of the lab stress- analysis, depending on the type of signal. A total of 70 detection model, and features are extracted from these signals. • a subset of the lab stress detector features. The best performing classifier on this dataset proved to be The whole stress-detection method, including lab stress de- the WEKA implementation of the support vector machine tection and activity recognition, is illustrated in Figure 1. algorithm. Its final output is a stress level prediction of 0 The context features refer to the hour of the day (1 to 24) (no stress), 1 (low stress) or 2 (high stress), which is then and the type of day (a weekday or weekend). used as input to the context-based stress detector. The output from the activity recognition model gives an estimate of the reliability of the lab model’s prediction. Aside 2.2 Activity Recognition from features extracted from the activity level predictions themselves, the activity level is also taken into account as It is important for a stress-detection system to be aware a modifier to the lab stress predictions prior to performing of the user’s physical activity, since physical activity elicits feature extraction on them. The lab model’s prediction is physiological arousal similar to psychological stress. For this discarded if it is made in an unsteady environment, which is reason, we used the 3-axis accelerometer provided by the defined as the occurrence of an average activity level above Empatica wristband, which has proven to be successful in 4 (high) in one of the (5-minute long) instances within the recognising activities, according to Gjoreski et al. [6]. The last 30 min. Additionally, the stress prediction is decreased activity recognition model was trained on 60 minutes of real- (its class is changed to a lower one, or left unmodified if its life Empatica data from one person, with nearly 10 minutes already zero) if the subject exhibited an average activity of labelled data per class. level between 2 and 4 (moderate) within the last 20 min. The machine-learning pipeline for the acceleration data is There are a total of 15 features extracted from the modified similar to the one used in the lab stress detector. Here, data lab stress prediction. A subset of lab stress detector features segmentation involves an overlapping sliding-window tech- is also used as input to the context-based stress detector. nique, which divides the continuous stream into 4 s windows with a 2 s overlap. The best performing algorithm for making a binary (“stress” or “no stress”) prediction using the outlined windowing pa- Feature extraction produces 52 features: seven represent rameters on the labelled real-life data was a Decision Tree [7], body posture, while the remaining represent body motion. so this algorithm was used to build the final context-based The extracted feature vectors are fed into a machine-learning real-life model. Using event-based windowing, this algorithm algorithm to build an activity-recognition classifier. achieved an F -score of 0.9 using leave-one-subject-out cross- The best performing algorithm on multiple acceleration validation. datasets was Random Forest [6], so this is the final algo- rithm used to build the activity-recognition model. The final output of the activity recognition model is a numeric activity 3. AWARE IMPLEMENTATION level on a scale from 1 to 5, where each number corresponds AWARE is a mobile instrumentation toolkit which had the to an everyday activity as follows: 1 = lying, 2 = sitting, initial purpose of inferring users’ context [4]. Extensibility, 4 = walking or standing, 5 = cycling or running. Finally, however, was a primary requirement when developing the the activity recognition classifier’s output is input into the framework. Specifically, extending the context by using context-based stress detection model. external sensors was explicitly envisioned, as was using the gathered data and machine-learning techniques for “creat[ing] 2.3 Context-Based Stress Detection new higher-level context” [4, p. 4]. 26 feature subset HR, IBI, heart rate, BVP, heart rate variability, EDA, skin conductance, “stress” or skin temperature TEMP stress Feature Context- Lab “no stress” Lab stress level extraction based Database feature Database detection and stress extraction smoothing detection activity acceleration acceleration level (x,y,z) Activity features Activity feature recognition extraction Figure 1: An overview of the context-based stress detection model. Features are extracted from physiological signals from Empatica and input to a lab stress detector. Its predictions are used in the context-based stress detector, which in addition takes activity features from acceleration data into account and considers the time and the type of day. In the next two sections, two such extensions are presented. application. The first one gathers physiological data from Empatica, sav- The collected physiological data is also written to a database ing it in the standard AWARE format. The second imple- (the lower part of Figure 2) by using content providers. Be- ments the stress detection method presented in Section 2 as cause of different sampling rates, each Empatica sensor has an AWARE plugin. its own content provider. A separate SQL table for each sensor contains columns for id, timestamp, device id and 3.1 Empatica Data Streaming Plugin a value from the sensor. Columns are defined in this way, so that the database tables maintain the standard AWARE The AWARE framework already offers plugins for acquiring format. Again, each content provider has its own “content data from Fitbit and Android Wear wristbands, but it does URI”, a unique address used by other plugins to identify not have one for the more research-oriented Empatica E4. providers for specific sensors. Our goal was to create an AWARE plugin that enables users When broadcasts are sent and the data is written to a to easily connect the Empatica E4 wristband to an Android database, it is up to other plugins to use it. Even though smartphone. the main purpose of this plugin is to provide the data that Figure 2 shows the overview of the processes implemented in will be handled by other plugins, it also offers a basic user this plugin. The physiological data is first transmitted over interface. It has its own activity (a user interface that most a Bluetooth connection. It is then available to other plugins Android apps have) from which users can export and clear via broadcasts and written to a database for later use. the data in the database. The data from Empatica is transmitted over a low-energy Contrary to most other Empatica data acquisition applica- Bluetooth connection, so that the impact on the smartphone’s tions, our plugin is meant to run in the background. There- battery is minimal. This enables Empatica E4 to stream fore, it was anticipated that the connection is lost without up to 24 hours on a single charge [1]. When a sensor read- users noticing. In an effort to solve this problem, our plugin ing is successfully transmitted from the wristband to the uses a notification to inform the user about the current state smartphone, specific functions (didReceiveAcceleration, of connection. didReceiveBVP etc.) are called automatically. In our case, these functions were expanded to include code to 3.2 Stress Detection Plugin send broadcasts and to save the reading to the database. In The goal of the stress detection plugin is to provide real-time this we follow the logic of other AWARE sensors or plugins, in stress predictions based on the context-based stress detection which received data is accessed via broadcasts to display and method described in Section 2. handle in real time, via content providers (through database) for more complex analysis (the middle part of Figure 2). In the original stress-detection study [7], the sensor data was recorded in the Empatica E4 wristband’s internal memory Broadcasts are inter-app messages that are sent when a and later transferred to a computer for further processing. specific event happens, in our case triggered by the data The novelty in our AWARE plugin is that the data is streamed transmission. These messages can be read with observers to an Android device via Bluetooth in real time and stored (broadcast receivers) from any app or plugin that is installed in the phone’s database. This allows for real-time processing on the phone. Since broadcasting happens in real time, other and classification. plugins can use the data from Empatica without including code for communication with the wristband (an example of The machine-learning pipeline for the stress detection plu- such a plugin is described in the following section). The gin mirrors the original pipeline used to train and test the data can also be displayed in real time in the native AWARE context-based stress detector. The three models (lab stress 27 Temperature Context Accelerometer broadcasts card BVP Empatica plugin GSR OTHER IBI PLUGINS SQLite Content database providers STRESS Bluetooth DETECTION Figure 2: An overview of the Empatica data streaming plugin. The data is first received via a Bluetooth connection and can then be broadcast to other plugins or written to a database. detector, activity recognition classifier and context-based To understand causes of stress and the situation where a stress detector) are independent and saved locally in the physiological stress response arises, it is helpful to know plugin’s assets. The models are triggered periodically using as much as possible about a user’s context. The plugins the optimal time intervals discussed in Section 2. described in this work will simplify combining physiological data with other contextual data the AWARE framework As discussed in Section 3.1, the Empatica data streaming already provides. Additionally, stress predictions can be plugin writes the raw data from the E4 in SQL tables in real used as context themselves and inform other interactions time. The stress detection plugin then has access to this data with users, such as offering them prompts at certain stress through the former’s content providers. The plugin reads the levels by using the AWARE Scheduler [2] and using stress last 5 min of raw Empatica data every 2.5 min and provides predictions as broadcast triggers. this data to the models for processing. The context-based model gets its context features using the phone’s current date and time. 5. REFERENCES The features from the lab stress detector, the activity recog- [1] E4 wristband for developers. http://developer. nition classifier and the context-based stress detector are, empatica.com/, 2018. Accessed: 2018-08-29. both broadcasted and saved in the phone’s database. The [2] D. Ferreira. AWARE: Scheduler. http://www. same is true for each lab stress prediction, activity level awareframework.com/scheduler/, 2015. Accessed: prediction and context-based prediction. The stored data is 2018-08-29. further accessible through content providers for any other [3] D. Ferreira and V. Kostakos. AWARE: Open-source application to use, as is the case with other AWARE plugins. context instrumentation framework for everyone. In this way, both the Empatica data and the stress prediction http://www.awareframework.com/what-is-aware/, method are easily available for other researchers to use. 2018. Accessed: 2018-08-29. [4] D. Ferreira, V. Kostakos, and A. K. Dey. AWARE: 4. FURTHER WORK AND CONCLUSIONS Mobile context instrumentation framework. Frontiers in ICT, 2(6):1–9, 2015. The plugins described in the previous section offer a ready- [5] H. Gjoreski, B. Kaluža, M. Gams, R. Mili, and made solution which researchers could use to add a stress M. Luštrek. Context-based ensemble method for human level to the user’s context. There are some limitations in energy expenditure estimation. Applied Soft Computing, their current implementation which we aim to amend. 37:960970, 2015. Currently, the standardization (normalization) of some of [6] M. Gjoreski, H. Gjoreski, M. Luštrek, and M. Gams. the features is arbitrary. It is done by subtracting a “typical” How accurately can your wrist device recognize daily value of a given signal and divided by a “typical” standard activities and detect falls? Sensors, 16(6), 2016. deviation. To account for inter-individual physiological dif- [7] M. Gjoreski, M. Luštrek, M. Gams, and H. Gjoreski. ferences, means and variability could be calculated on a Monitoring stress with a wrist device using context. person-specific basis. This would, at the very least, require Journal of Biomedical Informatics, 73:159–170, 2017. keeping track of a user ID and then calculating signal mean [8] C. D. Spielberger and S. J. Sydeman. State-Trait and variability over a longer time-period when a new user Anxiety Inventory and State-Trait Anger Expression would start using the application. If baseline values would be Inventory. Lawrence Erlbaum Associates, Inc, 1994. needed, this would also require the user to indicate when they are not under stress and calculate their specific physiological values in that time-window. This type of user interaction has not been accounted for in the current implementation. An evaluation of this method is also planned. The models described in Section 2 have been evaluated as outlined in related work, but they have been used in different experi- mental scenarios. Online real-life use of the method would merit its own evaluation. 28 BRISCOLA: Being Resourceful In Stacking Cards - Opponent, Lament Away! Vito Janko Nejc Mlakar Jani Bizjak Jožef Stefan Institute Faculty of Computer and Jožef Stefan Institute Jamova cesta 39 Information Science Jamova cesta 39 Ljubljana, Slovenia Večna pot 113 Ljubljana, Slovenia vito.janko@ijs.si Ljubljana, Slovenia jani.bizjak@ijs.si nejcmlakar37@gmail.com ABSTRACT start with three cards. Every turn both players play a card This paper describes a robot system that can play the pop- and at the end of the turn draw a card from the deck. After ular Mediterranean card game called Briscola. It elaborates both players play their cards, the second player wins if his on the three main components needed for an operational card shares the color with the Briscola card or shares the platform. First, it describes several artificial intelligence color with the first card and has greater strength, which is agents that can play the game using a combination of prob- based on the card number. The winning player gets points abilities, heuristics and the min-max algorithm. Second, it corresponding to the played cards value. After 20 rounds, describes the computer vision component for card detection whoever gets more than 60 points wins (a 60-60 score results using both classical and deep learning approaches and fi- in a draw). nally, it proposes a scheme for a robotic arm that can move the cards on the table. Categories and Subject Descriptors H.4 [Information Systems Applications]: Miscellaneous; D.2.8 [Software Engineering]: Metrics—complexity mea- sures, performance measures Keywords Intelligent Society, Computer Vision, Game Theory, Robot Arm Figure 1: The 4 aces of Briscola. 2. PLAYING THE GAME 1. INTRODUCTION First task was to develop an artificial agent that would be Briscula is one of Italy’s most popular card games, played able to play the game in a virtual environment. Due to the all around the Mediterranean (Italy, Spain, France, Greece, lack of external opponents, we developed several progres- Slovenia, Croatia). Despite that, the possibility of making sively stronger AI agents and matched them against each an artificial intelligence player for the game is poorly re- other to determine their strengths. Final version was also searched, with no papers on the topic found. We propose a matched against five human opponents to further evaluate system that can not only play the game on a computer, but its performance. can actually play against human players in the real world using a robotic arm. To do so, we developed three sepa- Mr. Random rate modules. First, we used a combination of heuristics, The first agent plays cards completely at random. While card probabilities and min-max algorithm to plan agent’s this strategy seems inadequate, it both provides a basic base- moves. Second, we used computer vision (CV) algorithms line, and demonstrates an interesting property of the game: to determine which cards the AI agent has and which cards the game’s variance is so high, that even this agent wins are being played by the (human) opponent. Finally, we pro- more than 5% of the games against the best agent (and sig- posed a robotic arm that is capable of picking up a card nificantly more against others), simply by having superior from a predetermined spot using suction at the arm’s end. cards. High degree of chance, explains why progressively better agents have diminishing returns in their win rate. 1.1 Briscula - Rules Overview The game is played with a special deck, containing 40 cards Mr. Greedy divided equally into 4 colors - Spade, Coppe, Bastoni, Denari This agent tries to maximizes the score after the current [Figure 1]. One card is selected at random at the beginning round. It opens the round with the lowest card, while taking of the game and is placed face up under the deck. Its color with the strongest card, if possible, when second. While is called ”Briscola”, giving the game its name. In this paper this provides the biggest boost in performance, its liberal we consider the 2-player variant of the game. Both players spending of strongest cards does not lead to optimal play. 29 Mr. Heuristics time. The heuristic used at the end nodes was simply the Agent implements author’s expert knowledge using if-else number of points accumulated in those rounds. rules. The wide set of rules includes holding strong cards until a valuable card is played, trying to start each round This variant performed best of all described and matches second if possible (as seeing your opponent play lets you expert human play. The contribution of each min-max use know how to best respond) and being careful in which situ- case was individually assessed by replacing it with heuristics ations to open the round with a valuable card. for that part of the game, and it was determined that all three parts contribute to the game-play improvement. Mr. Probable The use of heuristics can be amplified by predicting the like- 3. RECOGNIZING CARDS lihood of cards the opponent might be holding. Predicting In order to be able to play the game in real-life, it is essen- that the opponent has a strong card of one color, might lead tial to detect cards on the table and cards that are picked the agent to open the round with another. This was done by up from the deck. In this image recognition problem we as- weighting the influence of each conflicting if-else rule, by the sume that the card’s images are constant and that they are probability that it applies. The play was then determined placed on a mostly uniform background - table. The prob- by the ”strongest” rule. lem gets complicated due to the fact, that the card’s images can be very similar, they frequently overlap in practical play The prediction of card probabilities however, is not triv- and they can be sometimes covered by the opposing player’s ial. By counting the cards already played, we can deter- hands. Here we present several attempted approaches, rang- mine cards left in the deck and calculate the base probabil- ing from the simplest to the beyond state-of-the art deep ity that any of them is in the opponent’s hand, given his learning. hand size. This probability can be further modified by two factors. First, we can exclude some types of cards from their Removing background hand, given their past plays. This step assumes that the op- Since the cards are on the table and thus the surface color ponent has an elementary knowledge of the game, and will does not change much, we first tried to remove the back- play the obviously good play, given opportunity. For each ground color from the image and detect cards left on the card in opponent’s hand, we track when was it drawn, and table. A predetermined threshold, based on the RGB val- what plays were made since then. This allows us to pre- ues of pixels was used. This approach proved unreliable, as dict more precisely what kind of card it is. Second, cards of the subtle changes in lightning (lights in the room, clouds high strength or Briscula color tend to get ”stuck” in player’s over the sun) could change the color scheme enough for the hand, as players wait for a good opportunity to play them. threshold to fail to remove enough of the background. This means that their likelihood of being in a hand is greater than the base probability would suggest, especially in the Comparing differences later game. Their probability was weighted with an empir- ically determined weight, that moved from 1 to 1.5 as the Similarly, since most of the image is static (table, deck of game progressed. cards) and the only changes are the two cards being placed on the table, we looked at the history of images and tracked changes between them. Ideally, when a new card is placed on Mr. Calculator the table it should be the only changed part of the image and To avoid the if-else behavior, an agent can try to calculate could easily be detected. Once exact card position is known, different game branches and then decide for one that most any template matching technique could be used to identify likely leads to the desired outcome. There are two popular the card. The same approach could solve the overlapping frameworks for this task: variants of the min-max algorithm cards problem, as they could be identified one by one, as and the Monte-Carlo tree search. We decided to try the they are played. In reality however, this approach did not former and leave the latter for future work. work either. Due to the camera noise, most of the image was constantly changing. Second and bigger problem was The base version of the min-max algorithm [8] works with that when a card was placed on the table, hands and their perfect information and thus had to be adapted for this prob- shadows went over half of the table, changing pixel values abilistic case. Instead of using the probabilistic variant of in the image, complicating the use of this approach. min-max, that would have a huge branching factor, we tried to transform the problem into a perfect information one. Three cases were considered. 1.) In the last three rounds, Edge detection all cards are drawn and thus we have the case with per- Another attempt was to detect the edges of the objects on fect information. 2.) When only a few cards remain in the the table using Canny Edge detector [2]. To detect a card deck, we can do an exhaustive search of all possible orders of from the edges we used Hough Line Transform [3] in order to cards in the deck and all subsets of cards that can be in the detect straight lines that could later be combined into square opponent’s hand, and do a simple min-max search for each shapes to form cards. This improved the results significantly, possibility, averaging the results. 3.) In remainder of the however due to the camera noise and wood pattern of the game we sampled 100 different hands the opponent could table, the edges were often miss-detected. have each round, with regards to the probability described in the previous subsection. For each of the possibilities the SIFT min-max search is performed and the results were averaged. SIFT [5] is a scale-and-rotation-invariant image-recognition In all cases, the search depth was set to three rounds, as re- algorithm, which means that the object in the image can liability of our information on the opponent decreases with be rotated or scaled and the method should still be able to 30 15 fps, which is more than enough for real time detection. Detection accuracy was high when there were only 1 or 2 non-overlapping cards on the table [Figure 3], however it had problems with overlapped card. After investigating, we found out that since the network is trained with bounding boxes that are always aligned with the x and y axis, if the card is tilted at an angle, only half of it will be in the bound- ing box. Therefore the network is unable to learn to tightly detect a card and when they overlap the overall error is smaller if it just combines the two cards into one bounding box. Figure 2: Detecting Spade 4 card. Matching fea- tures from template image to camera view. detect it. SIFT finds so called interesting points (features) in the image, these are usually shapes of edges, stores all features from the sample images and then compares them with the features found in the new images, as seen on Figure 2. By comparing features position, the algorithm can also determine the image position, rotation and scale. This algorithm proved to work much better than previously described methods. It’s detection rate was high, detecting even partially obscured (overlapped) cards.However, it has two drawbacks. For each frame in the video (image) it has to Figure 3: Detecting cards with YOLO is fast and compare image (table) with 40 template cards. Depending reliable if objects do not overlap. on the resolution this can be very slow, ranging from 1 to 15 seconds. Sadly the accuracy is correlated with the size To solve this problem we started working on a modified ar- of the image, which means that for good predictions high chitecture that in addition to bounding box also predicts resolution images need to be used, which slows detection the angle at which the bounding box is rotated. The initial significantly. Second problem was distinguishing between results on generated data (photos of cards stitched on top lower ”Denari” cards. of different backgrounds) show promising results, where for most of the single cards in the image the network correctly Deep learning predicts the rotation of the bounding box. The network In the last decade deep learning has become the dominant works a bit worse where there are two overlapping cards but ML approach for multiple domains, with different deep learn- still manages to recognize a large percentage of images. We ing architectures achieving almost human level precision for believe that with some more time, larger set of training im- problems regarding computer vision. ages, tweaks and optimization of the architecture we could achieve close to 100% accuracy for the detection using this The standard approach is using several layers of convolution, new architecture. which is similar to what SIFT does, and then combining several fully connected layers in order to classify the features. 3.1 Robotic Arm and Cameras This works fine for image classification, but is unable to The last step in bringing the agent to the real world is the detect objects on the image. A naive approach would be presence of sensors and actuators. This component is com- to have a small sliding window that goes through the whole posed of two cameras and a simple robotic arm. The robotic image and classifies every part of it. This would be accurate, arm has 4 Degrees of Freedom created by 4 servo motors, but extremely slow. Several approaches have been developed that are controlled by Wemos D1 mini board. The board in order to tackle such problems: YOLO [6], Faster R-CNN acts as a web client, receiving the commands from the main [7], SDD [4]. Mentioned papers all go through the image only server and executing movement (controlling servo motors) once (working with 30-60 fps), but still achieve comparable actions. At the end of the arm there is a suction pump for results to slower window CNN approach. lifting and dropping the cards. All movements are predeter- mined and described with sets of motor’s rotation degrees. To test how well deep-learning approach works on our prob- Two movement patterns exist: drawing a card and placing it lem we implemented the YOLO architecture. Architecture on one of the three predetermined spots and picking the card consists of roughly 120 layers of convolution, pooling, regu- from one of the three spots and dropping it at the center of larization and fully connected layers. The training started the table. with pre-trained weights, obtained from VOC 2017 object detection. We manually labeled around 1000 card images, The system also has two cameras, first one to overlook the using the VOC format, and then trained the network for 3 table – tasked with detecting the cards played by the oppo- days on NVIDIA’s GeForce GTX 1080 graphic card. The nent. The second one is behind the arm, turned from the trained network performed relatively fast, achieving around floor up. Before dropping a card on the table, arm is rotated 31 1 2 3 4 5 6 7 8 9 10 avg. 80 90 80 90 85 70 60 60 25 50 69 Table 2: Win percentage (%) of Mr. Calculator against 10 human opponents of roughly increasing strength. To test the CV component we recorded several human games from the same angle as the final system uses. We then manu- ally compared the cards predicted by the CV with the actual ones. The best two approaches were SIFT and YOLO algo- rithms. The first worked flawlessly in all cases, except differ- entiating some of the Denari cards. The second could flaw- lessly recognize all cards, when they were not overlapped. Overlapped cards had roughly 50% accuracy. In the end we decided to use the SIFT algorithm for our first system pro- Figure 4: Server controls the robot arm. There are totype, since second player usually overlaps the first card. two cameras to oversee the table and picked up cards from the deck. 5. CONCLUSION R G H P C In this work we described three different components (from R - 26 13 14 6 different computer science fields) of a system that is able to G 74 - 21 18 15 play the Briscola card game against the human opponent H 87 79 - 42 34 in a real-life setting. For each component we individually P 86 82 58 - 40 tried different approaches, creating a strong AI agent and a C 94 85 66 60 - serviceable CV and robotic component. While the current version should be able to reliably play the game, all com- Table 1: Win percentage (%) of row agent against ponents still have room for improvement - we plan to test the column agent. Agents: Mr. Random (R), Mr. the Monte-Carlo search tree and improve the deep learning Greedy (G), Mr. Heuristic (H), Mr. Probable (P), architecture. We hope that we will be able to successfully Mr. Calculator(C) present a live demonstration at the paper’s presentation. 6. REFERENCES so the card is over the second camera and can be identified. [1] Robot prototype. https://dis.ijs.si/wp- The system is schematically presented in Figure 4 and its content/uploads/2018/10/briscola/briscola AI.mp4, first prototype is recorded and can be seen on the web [1]. 2018. [2] J. Canny. A computational approach to edge detection. The main logic for controlling the arm and taking actions IEEE Transactions on pattern analysis and machine is on the server, coupled with CV model and AI in order to intelligence, (6):679–698, 1986. take appropriate actions. [3] P. V. Hough. Method and means for recognizing complex patterns, Dec. 18 1962. US Patent 3,069,654. [4] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, 4. RESULTS C.-Y. Fu, and A. C. Berg. Ssd: Single shot multibox We started by assessing the comparative strength of the dif- detector. In European conference on computer vision, ferent AI variants. Each played 1000 games against each pages 21–37. Springer, 2016. other. Results are shown in Table 1 and show that the [5] D. G. Lowe. Object recognition from local agent’s skill increases with their increasing complexity. It scale-invariant features. In Computer vision, 1999. The also shows high degree of variance in the games, as even ran- proceedings of the seventh IEEE international dom agent got surprisingly many wins and the best agent - conference on, volume 2, pages 1150–1157. Ieee, 1999. Mr. Calculator is achieving only 66% win rate against sim- [6] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. ple ones. In repeated experiments we noted, that win rate You only look once: Unified, real-time object detection. fluctuates ±2% between runs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016. Next we compared the play strength against 10 human oppo- [7] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: nents of different skill levels. Each played 10 games against Towards real-time object detection with region proposal Mr. Calculator. Results are listed in Table 2 and show networks. In Advances in neural information processing an average 69% win rate of the AI against the human op- systems, pages 91–99, 2015. ponents. Volunteers that played, commented that the skill [8] S. J. Russell and P. Norvig. Artificial intelligence: a level of the agent is quite high, with some room for im- modern approach. Malaysia; Pearson Education provement in regards to increasing the agent’s risk aversion. Limited 2016. While the sample size is too small for definitive conclusions, ” we can assume that the agent is at least on par with average human players of the game. 32 Emotion Recognition Using Audio Speech Signal Maj Smerkol Mitja Luštrek Jožef Stefan Institute Jožef Stefan Institute Jadranska cesta 39 Jadranska cesta 39 Ljubljana, Slovenia Ljubljana, Slovenia maj.smerkol@ijs.si mitja.lustrek@ijs.si ABSTRACT about human emotions. We can choose a discreet repre- Emotion recognition is an important part of affective-aware sentation of emotions or a contiuous representation in some applications. Specifically, using audio speech signal has the space of emotions. In first case, we define different categories advantage of being compatible with applications using a nat- that represent different emotional states. The most widely ural language interface. There are multiple valid representa- known categorization is Paul Ekman’s basic emotions. Ek- tions of emotions. We propose a new representation aimed man studied facial expressions of emotions across different at using differently labeled databases jointly. We include a cultures and came to the conclusion that there are six basic short overview of some of the available databases and meth- emotions that are expressed equally across cultures. Those ods for feature extraction and selection. Both classification are sadness, happiness, anger, fear, disgust and surprise. of emotions and regression in 2D emotional space are dis- cussed. We concentrate on using neural networks for both On the basis of Ekman’s work others proposed different tasks. Regression provides good results but is hard to inter- models. Some of them have a different set of categories. pret while classification is more robust. Others use a continuous representation in two, three or four dimensional spaces. There is Plutchik’s wheel of emotions Categories and Subject Descriptors that represents emotions as four pairs of exclusive cate- gories, that are treated as four axes along which emotions are I.5 [Pattern recognition]: Neural nets; I.5.2 [Design method- spread. Emotions are represented as points in this space. J. ology]: Classifier design and evaluation Russel proposed a different model, a two dimensional space. Dimensions are arousal, which represents how active one Keywords feels, and valence, representing pleasurableness of the emo- Emotion recognition, Neural networks, Affective computing tion. 1. INTRODUCTION For our purpose, we prefer classification robustness over pre- cision. We don’t need very fine-grained information to better Nowadays applications such as personal digital assistants are the user experience of the application. Therefore in order to becoming more popular. Some also utilize natural language use as much training data as possible, we propose a four- interfaces. Next step in this direction seems to be affective class representation of emotions. The main idea of this rep- computing - applications that can detect human emotions. resentation is to be able to easily transform labels in other Such applications can enrich the user experience by respond- representations into a common one. Classes correspond to ing according to the user’s current mood and perhaps even quadrants in space of arousal and valence, and to groups of detect when the user is not happy with the application’s Ekman’s basic emotions: Happy: positive arousal and posi- functioning. However, in order to implement such appli- tive valence, includes basic emotion happiness. Calm: nega- cations we need to first be able to understand the user’s tive arousal and positive valence, there are no basic emotions emotions. This, in conjunction with other knowledge (such in this quadrant. Instead we include neutral. Sad: negative as user’s daily routines and other contextual information) arousal and negative valence, includes basic emotions sad makes it possible to detect certain mental health problems, and bored. Upset: positive arousal and negative valence, such as depression or bipolar disorder, shown by Osmani et includes basic emotions disgust, anger and fear. al. [8]. Therefore, we can jointly use databases that are labeled in Models we are developing will be used in an emotionally- space of arousal and valence (Recola, Semaine), as well as aware virtual assistant application. Our priority is to deliver those labeled discretely (EmoDB, Ravdess). information that can be acted upon in order to better the user experience. Application’s target population are people from Italy, Spain and Denmark. 1.2 Learning from Features or Raw Audio Traditionally in machine learning we first extract features 1.1 Representations of Emotions from audio. This can be done using specialized software, When talking about emotions in the context of affective com- such as OpenSMILE [3], or libraries, such as LibROSA [6]. puting, we first need to consider how to represent human emotions. In psychology, there exist many different theories With deep learning, it is possible to learn from raw audio sig- 33 nal. This recent approach is interesting, as in the raw audio session labeled for fear). Each available dimension is labeled signal there is encoded certain information that is missing by at least 2 annotators. Differences among different anno- in extracted features. Deep learning has two problems: (1) tators are quite noticeable which is to be expected in such larger databases are needed for training, and (2) training a setting. is very computationally expensive, both regarding compu- tational power and large amounts of memory needed. Problematic aspects of this database are: • Very unbalanced due to the chosen labeling method- ology. Counting each label sample, there are almost 2. DATABASES 4x as many examples of low arousal and high valence There are many public audio databases available for use in than examples of high arousal and low valence. affective computing. Most of them are targeted towards speech recognition or a subset of emotions, specific for a • For some dimensions label values span a very small given problem (such as detecting frustration in call centers). interval, which may cause problems with regression We describe the few of them that we have used. 1 along those dimensions. • Differences between annotators are often quite big. We chose those based on the way they were labeled, language Some files have inter annotator correlations below 0.2. and audio format used. Regarding labels we preferred labels While this is not unexpected - emotion expression and in space of arousal and valence or basic emotions in order perception are inexact - it is problematic for training to be able to do both regression in some emotional space or and testing. classification of emotions. We decided to only use European languages, since it has been shown that model trained on • Expressed emotions are very mild and often notice- language from similar cultural background to target popu- ably acted. There are examples in which we can hear lation gives slightly better results [2]. Audio simply needs to the actor, supposedly gloomy and depressive, express be of high enough quality. Human speech ranges up to 5kHz amusement by laughing. While extreme emotions are so we need at least 10kHz sampling rate. To be on the safe problematic so are very mild emotions - ML algorithms side and not lose any non verbal information we decided to often overfit to find other characteristics in the data. only use audio recorded at 16kHz or higher. 2.3 Recola 2.1 EmoDB Recola Database is a French multimodal datset of emotional EmoDB [1] (Berlin Database of Emotional Speech) is an speech. It includes audio, video, biosignals, labels (annota- older database. It contains 535 utterances spoken by 10 tions) and metadata. It is similar to Semaine in that it is also different actors. Each actor expressed each of the Ekman’s labeled continuously. It is only labeled along arousal and 6 basic emotions (and a neutral version) at least once for valence dimensions, but labels are of higher quality. Each each of the ten different texts. Each file is labeled. Texts recording is labeled by 6 different annotators, 3 male and 3 themselves are emotionally neutral. Utterances are quite female. short, recordings are between a couple of seconds long up to half a minute. Problematic aspects of this database are: • Each file is exactly 5 minutes long, but some of the labels are missing a few samples. We have cut the Problematic aspects of this database are: audio files to match the label lengths. • Utterances are very short. Often when classifying au- dio, recordings are cut into segments from 1 second • There are only 23 recordings. Since each is 5 minutes up. If we do that with EmoDB, there are simply not long it is still quite large. enough instances to use deep learning techniques, in • It is quite unbalanced. Counted by each label sam- same cases there are even not enough for traditional ple, there are more than 8x as many examples of high ML. arousal and high valence than examples of low arousal • Expressed emotions are extreme to the point of over- and low valence. acting. This means that classifiers trained on this set may produce weak generalization, as most speech is 2.4 Ravdess closer to neutral as considered in this database. The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) contains video and audio of 24 ac- 2.2 Semaine tors. Emotions expressed include the 6 Ekman’s basic emo- tions, neutral and calm. Each utterance corresponds to one The SEMAINE [7] database is a multi-modal database that emotion. There are 1440 files in the speech section and 1012 includes audio, video and transcripts of English texts. The in the song section. database is labeled on a continuous scale along many di- mensions. Not all sessions (couple of minutes long record- Problematic aspects of the speech section of this database ings) are labeled in all dimensions. Most are labeled along are: the arousal and valence dimensions, as well as intensity and • All utterances contain one of the two texts: (1) ”Kids power. Fewer are labeled along basic emotions (e.g. only 2 are talking by the door” or (2) ”Dogs are sitting by the 1 door”. This may represent a problem as all files are in Some of reportedly high quality databases such as the Hu- maine database and Vera am Mittag (eng. Vera at noon, a way very similar. On the other hand, this is good as a database of German emotional speech taken from reality it helps prevent overfit as the algorithm can’t learn to TV and talkshows) are not available anymore. differentiate utterances based on text contained. 34 • Utterances are very short, similar to EmoDB. 4. EXPERIMENTS We have tested regression in the space of arousal and valence 3. FEATURES and classification of basic emotions in order to compare two very different approaches and decide which is preferable for Among features used for classification of audio of human our use-case. speech are (1) simple features such as loudness, signal en- ergy and pitch, (2) Mel spectrum: similar to frequency spec- trum, transformed to Mel scale which corresponds roughly 4.1 Regression in Arousal and Valence Space to human perception of pitch, (3) Mel frequency cepstrum While deep learning on raw audio signal is slower and more coefficients: inverse Fourier transform of log-scaled Mel spec- computationally expensive, it may produce better results as trum, (4) Jitter, shimmer: frequency noise instability and raw signal contains more information. We have replicated amplitude instability, (5) Formants: most present harmonic the experiment done by Trigeorgis et al. [9]. Due to hard- frequencies, (6) Spectral features: describe the shape of the ware constraints we had to introduce certain modifications: frequency spectrum and (7) Chroma features: describe tonal (1) we had to use 3 second segments instead of 6 second properties, such as melody. segments and (2) we used a mono-directional LSTM layers instead of bi-directional as in the paper. Network topology is otherwise same. 3.1 Tools Some of the commonly used tools for audio feature extrac- Training and testing was done on the RECOLA database. tion are OpenSMILE and LibROSA. Data was split into train and test sets by actors - 80% of ac- tors in the train set and 20% in the test set. Our results were OpenSMILE is a standalone program with a very step learn- very similar to those reported in the paper. Measurements ing curve. Writing custom configuration files which is needed shown in Table 1 are Concordance correlation coefficients for extracting custom features as opposed to using one of the (CCC)2 between predictions and ground truth, obtained as predefined features sets is quite complicated. Most users use averaged labels. Predictions are scaled to have the same predefined configurations, which can also be found online. standard deviation as the ground truth and time-shifted in order to remove any delays that a human annotator may LibROSA library is an easy to use alternative that works produce. Thus we can confirm that deep learning from raw with Python and offers similar functionality. It also offers audio data is feasible. some utility functions for reading and storing audio files, filters etc. Arousal CCC Valence CCC Raw audio 0.641 0.250 3.2 Feature Selections and Analysis Features 0.574 0.187 Feature selection is an important step in the ML pipeline as Trigeorgis et al. [9] 0.684 0.249 having fewer features is beneficial for reducing training time as well as reducing the possibility of overfitting. Table 1: Valence and arousal regression results We have performed feature selection using each of described The network is made of two distinct functional units. First databases, using features calculated by OpenSMILE (using are the convolutional layers that learn to perform feature a slightly modified ComParE13 LLD configuration) and sep- extraction. It has been shown [9] that certain neurons are arately using features calculated using the LibROSA library. highly correlated to some of the known good features. The second part is made of two LSTM layers. These learn to 1. Remove features with variance below 0.2, as they hold regress arousal and valence from extracted features. little information. The same experiment was repeated using only the second 2. Sort by correlation with labels and remove those with part of the neural network, trained using extracted features absolute correlation below 0.1 as they mostly contribute (feature set ComParE lld). Results were somewhat worse, noise or bias towards groups with certain vocal quali- which indicates that the convolutional part of the full neural ties. network learns to extract a better set of features than we get using simple feature selection (as described above). Unfor- 3. Greedy feature selection: take the feature with the tunately predictions in the space of arousal and valence are highest correlation, add it to the feature set and test hard to interpret and there is no direct way to convert them on a surrogate model (logistic regression or random to basic emotions. forest classification). We use surrogates to reduce the computation time. Keep feature if it improves perfor- 4.2 Classification of Emotions from Features mance of the surrogate. We have used EmoDB for initial experiments. All reported results are averaged over leave-one-person-out cross valida- Using this method the number of features was reduced from tion. Simple fully connected feedforward neural networks 132 to 60 (feature set ComParE lld extracted using OpenS- tend to overfit. This can be reduced with hyperparameter MILE), and from 167 to 110 (custom feature set extracted adjustment (learning rate and algorithm, mini-batch size, using LibROSA). We achieved the same performance on the early stopping etc). models while reducing the training time compared to no fea- 2Concordance correlation coefficient is a measure of agree- ture selection. ment, often used to measure inter-rater reliability. 35 This was tested on the database split into 3 sets, train, test A new categorization of emotions was proposed with the aim and evaluation. Using 3 sets show that overfit is still there, of using multiple databases jointly. Preliminary experiments but the difference in performance was small between train show that we can use it for machine learning on multiple set and test set probably due early stopping based on test databases. Whether models trained in such way will perform set loss. Performance on evaluation set is still much lower. better is yet to be seen. We used all features from the ComParE lld feature set. In- In conclusion, emotion recognition using audio signal is a put layer therefore has 130 units, fist hidden layer 70, second complex and difficult task. Some of our experiments come hidden layer 30 and output layer 7 (6 Ekman’s basic emo- close to state of the art, but still not very good. We believe tions + neutral). For the experiment, MSE was used as loss we can improve our work further in the future. function, and Adam as optimizer. Without using regulariza- tion, we achieve very poor performance. As the model starts 6. REFERENCES to overfit we stop training it, which is before it achieves good [1] Felix Burkhardt, Astrid Paeschke, Miriam Rolfes, performance. Without regularization accuracy is therefore Walter F Sendlmeier, and Benjamin Weiss. A very low on all sets. Even with strong regularization, using database of german emotional speech. In Ninth both added Gaussian noise (std = 1/2std(f eatures)) to in- European Conference on Speech Communication and put layer and dropout (p = 0.5), large differences on train Technology, 2005. set and evaluation set can be seen. [2] Moataz El Ayadi, Mohamed S Kamel, and Fakhri Karray. Survey on speech emotion recognition: We compare out results to state of the art as achieved by Features, classification schemes, and databases. Yenigalla et al. [10] in 2018 and Gjoreski et al. [4] from Pattern Recognition, 44(3):572–587, 2011. 2014. Yenigalla et al. achieved high performance using [3] Florian Eyben, Martin Wöllmer, and Björn Schuller. convolutional neural networks, trained using extracted fea- Opensmile: the munich versatile and fast open-source tures and phonemes. IEMOCAP dataset was used. Gjoreski audio feature extractor. In Proceedings of the 18th used Auto-WEKA, a machine learning tool that automati- ACM international conference on Multimedia, pages cally chooses best classical-ML algorithm. They trained and 1459–1462. ACM, 2010. tested using EmoDB. [4] Martin Gjoreski, Hristijan Gjoreski, and Andrea Accuracy (test) Accuracy (eval) Kulakov. Machine learning approach for emotion recognition in speech. Informatica, 38(4), 2014. No regularization 0.54 0.48 [5] Yann LeCun, John S Denker, and Sara A Solla. Noise, dropout 0.82 0.65 Optimal brain damage. In Advances in neural Gjoreski et al. / 0.77 information processing systems, pages 598–605, 1990. Yenigalla et al. / 0.73 [6] Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Table 2: Classification results for test set and eval- Nieto. librosa: Audio and music signal analysis in uation set, compared to state of the art [10]. python. In Proceedings of the 14th python in science conference, pages 18–25, 2015. We have also performed some preliminary experiments using [7] Gary McKeown, Michel Valstar, Roddy Cowie, Maja Optimal Brain Damage (OBD) algorithm[5] to prune the Pantic, and Marc Schroder. The semaine database: network. Results are not yet conclusive but seem promising. Annotated multimodal records of emotionally colored We did not achieve better performance, but did achieve same conversations between a person and a limited agent. performance while pruning up to 60% of all units. IEEE Trans. Affect. Comput., 3(1):5–17, January 5. CONCLUSION 2012. [8] Venet Osmani, Agnes Gruenerbl, Gernot Bahle, We have experimented with regression in space of arousal Christian Haring, Paul Lukowicz, and Oscar Mayora. and valence. Results confirm that a combined convolutional Smartphones in mental health: detecting depressive and recursive neural network can effectively learn on raw and manic episodes. arXiv preprint arXiv:1510.01665, audio signal. Since the authors who propose this approach 2015. state that the convolutional part of the network learns to [9] George Trigeorgis, Fabien Ringeval, Raymond perform feature extraction we tested only the recursive part Brueckner, Erik Marchi, Mihalis A Nicolaou, Björn of the neural network, trained on pre-extracted features. Re- Schuller, and Stefanos Zafeiriou. Adieu features? sults were somewhat worse, which can be interpreted as the end-to-end speech emotion recognition using a deep convolutional part of the network learns to extract better convolutional recurrent network. In Acoustics, Speech features. Aditional experiments, such as classification using and Signal Processing (ICASSP), 2016 IEEE a similar neural network are needed in the future. International Conference on, pages 5200–5204. IEEE, 2016. We have also tried using a fully connected artificial neu- ral network (FNN) to classify emotional speech. FNN is [10] Promod Yenigalla, Abhay Kumar, Suraj Tripathi, extremely prone to overfit. Even using very aggressive regu- Chirag Singh, Sibsambhu Kar, and Jithendra Vepa. larization techniques show some overfit. It seems that either Speech emotion recognition using spectrogram & (1) FNNs need a larger amount of labeled training data or phoneme embedding. Proc. Interspeech 2018, pages (2) are not well suited for this problem. Related future work 3688–3692, 2018. is performing experiments using OBD to prevent overfit. 36 Improvement of AI through Deep Understanding Jani Bizjak Matjaž Gams Department of Intelligent Systems Department of Intelligent Systems Jamova cesta 39 Jamova cesta 39 1000 Ljubljana 1000 Ljubljana jani.bizjak@ijs.si matjaz.gams@ijs.si ABSTRACT legs of the animals were collection of pixels statistically grouped Humans have concerned with semantics behind words, objects and around the body, the color was a number and classification was a facts for thousands of years. Yet, when a computer uses a model calculation pointing towards one possible class. The class itself was and learns to recognize it from a video, does it really understand a number and was linked to a term “cat” by humans. what it is or does it only map a number of specially colored pixels In the past and nowadays alike, semantics are often treated just as and shapes to a term “object”? In addition, humans use semantics another feature, another numerical input to the computer system. to improve performance; therefore, it seems reasonable to assume The computer uses it to improve performance, but in a similar way that computers would as well. This paper overviews latest state of as any other numerical feature, without additional semantics, the art papers that learn and use semantics in order to improve the meaning or procedures attached to it. It can and usually does results of established methods in different areas. Two branches of increase classification accuracy, but the model does not understand AI: natural language processing and computer vision seem to be what it means. For computers to “understand” does not necessarily especially active in this area. mean to be very similar to the way humans use semantics, in particular regarding the way the understanding is coded in the Categories and Subject Descriptors computer model, which is likely to be different in humans. Rather, I.2.m [Artificial Intelligence]: Semantic analysis understanding in computers should be functionally somehow General Terms similar, i.e., enabling solving tasks in somehow similar way. Algorithms, Theory. In this paper we provide a brief overview of the latest state-of-the- art papers that learn or use semantics to further improve their Keywords models. Review paper, semantics, ML 2. SEMANTICS IN NATURAL LANGUAGE 1. INTRODUCTION PROCESSING Machine learning and artificial intelligence are as old as modern 2.1 Approximating Word Ranking and computer science. However, there is an essential difference. Even though we have thought the computer for example to recognize the Negative Sampling for Word Embedding [8] images of a cat, be able to predict the future of a stock or play a Word embedding is a technique to present each word by a dense game of chess, in reality we just thought computers a good enough vector, aiming to capture the word semantics in a low rank latent mathematical model that approximates the real world application. space, e.g. each word is translated into a vector of 0 and 1s in such The computer methods seem to be qualitatively much different than a way, that words that are semantically closer differ in less bits than the way humans use intelligence and learn. semantically different words. It is widely adopted in Natural Language Processing (NLP) tasks. Variants can also be used in In early 2010, deep learning started gaining popularity. Google was almost any domain where semantics play a role, such as computer the first one to successfully use deep neural networks (DNN) when vision. One of the latest approaches in implementing word trying to classify cats in photos on the internet. They used embedding is Continuous Bag-of-Words (CBOW) [9]. CBOW Convolutional Neural Networks (CNN) architecture and were predicts a target word given a set of contextual words, where the incredibly successful. No method before achieved such target word is labeled as positive and the others are classified as classification accuracy for image recognition, not yet similar to negative, e.g. if we have a sentence and want to predict word 𝑤𝑖 humans, but splitting the difference in two. A couple of years later, (positive) we take a look at the neighboring words 𝑤𝑖−2, 𝑤𝑖−1, the DNNs achieved the human level accuracy and from that 𝑤𝑖+1, 𝑤𝑖+2 (negative). However, the method treats all words threshold on, each year further improvement in their performance equally based on frequency in text instead of favoring the positive is made. ones. Because deep networks work like a black box, we do not know Authors of this paper [9] develop a new approach to word exactly why something is classified as is. People started to wonder, embedding, based on CBOW that favors positively ranked words. has Google made a first step to superintelligence? Has their method They do so by selecting negative words that tend to decrease overall actually learned what a cat is, learned the meaning, semantics and performance. The method works in two steps: it first increases the everything that goes with it? With further experiments the question score of the positive words, and in the next step decreases the score deepened. For example, the network “knew” that a cat has four legs, of the negative words. With this approach they improve the overall a tail and so on. It knew how to distinguish it from a dog, who also performance compared to other BOW approaches. has a tail and four legs. In the end however, it tuned out the method This approach works relatively well for larger texts but still was practically as “shallow” as everything that came before. The struggles with short texts. Authors in [7] propose additional steps 37 that can be taken in order for the approach to also work with short network doesn’t predict or use semantics as an output or input for texts. The words are usually represented as vectors, and based on predictions, semantics proves to be essential in training phase. the Hamming distance of the embedded vectors, one can notice that In order to learn to recognize different pieces of clothing, the semantically closer words are also grouped closer (have shorter authors proposed a twofold deep learning architecture [Figure 3]. Hamming distance) as seen in Figure 1. The first part is standard CNN architecture while the second is only the feature extraction part. Both networks are then forced to produce features for different (pairs) of images. If the images are semantically similar, the output features should be as close as possible. Figure 1. Word clustering based on their semantical meaning. In the left image the words are randomly located around 0,0, while on the right side image we can see that clusters form. 2.2 Task-Guided and Semantic-Aware Ranking for Academic Author-Paper Correlation Inference [6] Figure 3. Semantically similar images should have similar weights. In this paper [6], the authors consider author-paper correlation inference in big scholar data, such as Google Scholar, Microsoft In this paper the contextual knowledge of clothing images is Academic or aMiner. In other words, they would like to provide an manually defined as finding neighboring images with similar author relevant and related publications based on the author’s appearances or poses. This is because such images also have similar previous papers and citations. high-level features. The authors decipher the pose and appearance from the image using OpenPose [2] tool by extracting it from the To solve the problem, the authors propose a model by joint content convolutional layers. To then find the two closest images they use semantic encoding and heterogeneous relations augmented Euclidian distance between the extracted features. ranking, and design the corresponding learning algorithm. In the first step they use Gated Recurrent Neural Networks (GRU) in The semantics in this paper are manually defined as a pose and order to obtain latent features for authors and semantic embedding appearance of the model in the image. The method in the end is still for each paper. To further improve the results, authors also include incapable of understand semantical meaning of the clothing, but citations and transitive citations (multiple papers deep) of author can use it as an additional feature in order to help with training and and his/hers papers using a heterogeneous network (HetNet). The in the end increases overall accuracy of the model. architecture is presented in Figure 2. 3.2 Deep Joint Semantic-Embedding Hashing (DSHE) [3] The days when searching for similar images meant comparing histograms, colors or metadata attached to the image are long gone. Nowadays the labels for the images are automatically created, for example if the image contains a cat it will be labeled as a cat and so on for every object it contains. This works well for reasonable numbers of pictures, but because of the sheer number of images on the internet it would take too long to compare labels for each image that exists. To solve this problem, a special hash is used that transforms the text label into a vector of 1s and 0s. Ideally the visually (contextually) similar images should have this vector very close to each other when using the Hamming distance. This means Figure 2. Framework is composed of GRU for direct author that a vector of a dog should be closer to the vector of a cat paper relation and HetNet for indirect author paper compared to a vector of a dinosaur, which should still be closer to correlations. the pair then to a vector of a truck. The approach is similar to the The paper learns semantic representation of each paper and then vectors gained from word embedding described in the previous compares it with related papers allowing authors to better find sections. related work based on their paper history. Authors in this paper [3] present a new approach to hashing. They 3. SEMANTICS IN COMPUTER VISION use twofold deep architecture [Figure 4], one part of which is tasked 3.1 Semantic Locality – Aware Deformable with feature extraction (CNN), while the other embeds labels to vectors. The features extracted are then joined in common semantic Network for Clothing Segmentation [1] space, where dependencies between image and labels are learned. This paper [1] tries to solve the problem of clothing segmentation By doing so the features extracted from the image are forced to be and identification from photos of people wearing them. While the similar for images with similar semantics. 38 3.4 Adversarial Attribute-Image Person Re- identification [6] In previous sections we described a different approaches in extracting and storing semantical meaning from different types of images. In this section, one possible use case is presented where semantics are used to find or identify a person in an image. In the previous section [3.2] it was presented how the computer Figure 4. Features extracted from labels and images are joined method finds similar images using semantical labels and hash in second half of the architecture where semantic dependencies codes. Those approaches work fine; however, they have one major are learned. drawback - they require an input image from which they can calculate those features and then search for similar image. The semantics used in this paper are hidden and calculated inside However, when humans want to look up for someone or describe it the network. The network actually learns deeper connections to someone, they usually describe that person’s features [Figure 6], between labels and the images. The network now not only knows for example: Caucasian, male, 1.8m tall, blue eyes, wearing a hat how a cat looks like but also knows that a cat is closer to a dog than and a blue backpack. If a human were given these instructions it it is to a car. would be easily for them to find a person in a set of images, but for 3.3 Semantic Structure-based Unsupervised computer methods, on the other hand, this present a major problem. Deep Hashing [4] Hashing is becoming increasingly popular for approximate nearest neighbor searching in massive databases due to its storage and search efficiency. Related work shows promising results when learning from labels, however it is significantly more difficult to do the same in an unsupervised setting. The paper [5] shows that a lot of semantic information can be extracted from features obtain from CNN. Authors first analyze statistical properties from the obtained features. With this information they are able to construct a semantic structure that explicitly captures the semantic relationship across different data points. In the following step they calculate semantical distance between two points using cosine distance. The experiments show Figure 6. Person identification using high level descriptors. that semantically closer features have lower distance - as expected. The framework described in the paper learns semantically In the last step they use special loss function that calculates inner discriminative structure of low-level person images, and generates product between significantly similar or dissimilar points and use it a correspondingly aligned image-analogous concept for high-level to train hash codes using deep learning. Network schematics can be attribute toward image concept. This averts direct estimation of the seen in Figure 5. attributes in a person image and solves the problem of imperfect prediction and low semantic discriminability. The framework [Figure 7] is trained using adversarial learning approach. This means that part of the network is trying to generate concepts while the other part is trying to distinguish if they are good or not. Both parts of the network are trained simultaneously, learning to be better at their task and thus competing against each other while at the same time improving. The network again consists of two parts. The first part is tasked with concept extraction from the images, while the other branch is tasked with concept generation from the labels. Both branches are then joined in semantic classification. Figure 5. Network architecture. Network first discovers statistical properties from CNN features. Then it learns their hash functions using special loss. This paper [4] takes the next step and removes the need of semantically labeling the images. It forces the architecture to learn semantic relations without telling it what the main context / feature of detected object is. 39 images, words and sentences and use it to better model the real world? In summary, one of the central questions can be presented as follows: Can deeper understanding through automated semantic extraction increase the AI performance independently of domain or task? 5. REFERENCES [1] Ji, W., Li, X., Zhuang, Y., Bourahla, O.E.F., Ji, Y., Li, S. and Cui, J., 2018. Semantic Locality-Aware Deformable Network for Clothing Segmentation. In IJCAI (pp. 764-770). [2] Cao, Z., Simon, T., Wei, S.E. and Sheikh, Y., 2016. Realtime multi- person 2d pose estimation using part affinity fields. arXiv preprint arXiv:1611.08050. Figure 7. Network architecture has two parts, concept extraction from image and concept generation based from the [3] Li, N., Li, C., Deng, C., Liu, X. and Gao, X., 2018. Deep Joint description. Semantic-Embedding Hashing. In IJCAI (pp. 2397-2403). [4] Yang, E., Deng, C., Liu, T., Liu, W. and Tao, D., 2018. Semantic Structure-based Unsupervised Deep Hashing. In IJCAI (pp. 1064- 4. CONCLUSION 1070). In this paper we presented a short overview of latest state of the art [5] Girshick, R., Donahue, J., Darrell, T. and Malik, J., 2014. Rich methods and approaches that use semantics in several different feature hierarchies for accurate object detection and semantic ways. We first looked at semantic extraction in NLP tasks, more segmentation. In Proceedings of the IEEE conference on computer precisely with word embedding, where semantically closer words vision and pattern recognition (pp. 580-587). also have smaller Hamming distance. In later sections we looked at [6] Zhang, C., Yu, L., Zhang, X. and Chawla, N.V., 2018. Task-Guided semantics being used primarily as an additional attribute for object and Semantic-Aware Ranking for Academic Author-Paper recognition in images. Correlation Inference. In IJCAI (pp. 3641-3647). The architectures used are similar across all papers. They consist of [7] Wang, P., Xu, J., Xu, B., Liu, C., Zhang, H., Wang, F. and Hao, H., 2015. Semantic clustering and convolutional neural network for two part deep networks, one part is usually tasked with extracting short text categorization. In Proceedings of the 53rd Annual Meeting features from the image, while the other part is tasked with of the Association for Computational Linguistics and the 7th extracting semantical meaning, either from labels or learning it on International Joint Conference on Natural Language Processing its own. In the next steps both parts are merged, which forces the (Volume 2: Short Papers) (Vol. 2, pp. 352-357). network to incorporate semantical knowledge into the features [8] Guo, G., Ouyang, S.C., Yuan, F. and Wang, X., 2018. extracted from the images. Approximating word ranking and negative sampling for word One can see that semantics are more often than not used as an embedding. IJCAI. additional parameter, i.e. feature, which helps the established [9] Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S. and Dean, J., model achieve better accuracy. The question here is if true 2013. Distributed representations of words and phrases and their understanding of semantics behind could further increase the compositionality. In Advances in neural information processing model’s performance and bring us one step closer to true systems (pp. 3111-3119). intelligence or even superintelligence. With enough training examples, can an architecture learn the deeper meaning behind the 40 Assessment and Prediction of Auxiliary Carabid Species in Agricultural Fields Marko Debeljak Vladimir Sašo Džeroski Veronique Tosser Aneta Trajanov Jožef Stefan Institute Kuzmanovski Jožef Stefan Institute ARVALIS Jožef Stefan Institute and Jožef Stefan Jožef Stefan Institute and Jožef Stefan and Jožef Stefan 91720 Boigneville, International and Jožef Stefan Int. International France International Postgraduate School Postgraduate School Postgraduate School Postgraduate School +33 1 64 99 23 15 Jamova 39, Ljubljana, Jamova 39, Ljubljana, Jamova 39, Ljubljana, Jamova 39, Ljubljana, Slovenia v.tosser@arvalis.fr Slovenia Slovenia Slovenia +386 1 477 3124 +386 1 477 3143 +386 1 477 3127 +386 1 477 3662 marko.debeljak@ijs.si vladimir.kuzmanovski saso.dzeroski@ijs.si aneta.trajanov@ijs.si @ijs.si ABSTRACT and knowledge about biological pest control from empirical data, Biological pest control depends on the abundance and richness of an extensive data pre-processing is needed [3]. beneficial species. Development of efficient pest management The goals of this study was to assess and predict the abundance plans requires new knowledge about complex interactions of predatory species that could perform biological pest control in between the elements of agricultural ecosystems and their natural agricultural fields. In particular, we are exploring the taxonomic and management environment. Empirical ecological data and functional biodiversity of ground beetles of the family represent a big obstacle in the acquisition of this specific Carabidae, which prey on slugs, that are the most damaging pests knowledge as they are most often, incomplete, inconsistent and to cereal crops. imbalanced. In addition, they require a lot of pre-processing for their use in analyses and modelling. In this study, we are dealing In this paper, we first present the comprehensive data pre- with carabid beneficial species that could provide biological pest processing that was carried out in order to obtain high quality control in agricultural fields. In particular, our goal is to describe datasets. These were later used for assessment of the abundance taxonomical and functional diversity of carabid species to assess and taxonomic and functional diversity of carabid species. In the the potential performance of biological pest control in the studied part that follows, we present the predictive models developed by area and to develop predictive models for the most abundant data mining. The paper ends with conclusions and directions for carabid species and their predator functional group. The results further work. show high potential of carabids to provide biological pest control 2. MATERIALS AND METHODS in the studied area, but the predictive models achieved relatively low predictive performance. They could be improved by an 2.1 Data additional set of attributes describing specific habitat Abundance and biodiversity assessment is based on data from requirements of carabid species. field surveys. For the development of the predictive models, we included also data describing the environmental conditions and Keywords the applied agricultural practices. Therefore, we used data from Carabidae, beneficial species, data pre-processing, taxonomic several different datasets and data providers. and functional diversity, data mining, predictive models Specimen data from field surveys were provided by ARVALIS, 1. INTRODUCTION Institut du vegetal, France. Field surveys of carabid species were Sustainable agriculture enhances biological pest control in order conducted in Boigneville (central France) in years 2009 to 2011 to reduce the use of pesticides and to foster natural biodiversity and 2013 to 2015. Carabid species were collected in pitfall traps and improve the quality of the environment. (Figure 1) using a standardized sampling scheme. There were 21 pitfall traps in the period from 2009 to 2011 and 15 for the period In this context, the control of pests provided by the natural enemy 2013-2015. They were placed in four fields and their locations populations reduces the dependency on plant protection products. were permanent throughout the study period. Specimens were Predatory (beneficial) species such as Syrphidae (hoverflies) , taken from pitfall traps on a weekly basis between April and July which control aphids, and Carabidae (ground beetles) , which and September and November. The caught specimens were feed on slugs, are the main natural enemies of these crop pests determined to the species level and the number of caught [1]. To improve the regulation of pests by predatory species, we specimens per species was counted. The total number of pitfall need knowledge about the effects of landscape, soil and crop samples included in our research was 2873. management on these beneficial species. This is a demanding research challenge because of two main reasons. The first reason To describe the ecological functional traits of carabid species, we is the complexity of interdependencies among elements of used an additional and extensive database “Functional traits of agricultural ecosystems and their interactions with the Carabidae species” [4], compiled, maintained and provided by environment (e.g., climate, soil humidity). The second reason are ARVALIS. It includes information on species’ size, diet (larvae the empirical ecological data that are many times incomplete, and adults), humidity preferences, wintering habitat, reproduction inconsistent, containing out-of-range values, are collected at period and ability to fly. The database contains traits information different temporal and spatial scales, dispersed in different for 171 carabid species. databases, noisy and imbalanced [2]. To obtain new information 41 In addition, several other attributes describing the taxonomic and functional diversity of carabid species were calculated from the available data. The richness of carabid species was described by the Shannon’s and Simpson’s diversity indices [2]. To measure the distribution of abundances between species, evenness was calculated [3]. Since the interpretability of individual diversity is hard we combined indices of species richness into Hill numbers (N0, N1, N2), which are very suitable for ecological interpretation [3]. In particular Hill numbers (H0-number of all species, H1- numer of abundant species and H2-number of very abundant species) are calculated from the three most important and known Figure 1: Pitfall trap. measures of diversity: S-number of species, H’-Shannon’s index and λ-Simpson’s index [2]. A more detailed explanation of Data about the landscape structure and crop properties were diversity indices is given in the conference paper explaining the obtained for an area within a 500 m radius around each pitfall trap diversity of syrphid species that provide biological control of (Figure 2). In the delineated area, the absolute and relative surface aphid pest species [5]. of different crops and natural vegetation types were measured, as well as the length of linear corridors (tree lines, grass strips, grass To describe the habitat diversity in a radius of 500 m around the pathways, hedges, roads). Landscape data were obtained from sampling point, we applied Shannon’s and Simpson’s indices and digital maps using GIS software tools. In addition, crop evenness indices for landscape diversity as well. The former two development stages were estimated for each crop in the studied describe the habitat richness and diversity and the latter area and crops were grouped into several categories according to emphasizes the evenness of the landscape categories [2]. Despite the habitat preferences of carabid species. the fact that the used indices were initially developed for description of species diversity, they can be used for description Soil data were obtained from the ARVALIS soil database, which of landscape diversity as well. In our case, we used the types of contains information on chemical, physical and biological soil landscape instead of species and instead of abundance, we used properties (e.g., soil texture, available water holding capacity, the land cover area (m2) of a particular landscape type. bulk density, etc.). The presence and activities of carabid species depend on their Climate data were obtained from the French national development stage (e.g., larva, pupa, adult) that is primarily meteorological station located in Boigneville. Data about driven by daily temperatures that are expressed in degree-days. maximum, minimum and average temperature, and cumulative Degree-days are the most common phenological indicator in rainfall have been collected at daily bases for the period from entomological research. Degree-days provide information about 1.1.2009 to 31.12.2019. the cumulative value of average daily temperature for a time period when the average daily temperature is above the selected minimum threshold. We used the most recommended simple logistic equation for calculation of the degree-days with the minimum temperature threshold of 50C. Using degree-days, we can compare the abundance and diversity dynamics of carabid species between years and locations. This could not be done by using calendar dates because the climatic and environmental conditions of the selected dates are different in different years. The pre-processing of environmental, agricultural, taxonomic and functional data resulted in a dataset containing groups of attributes describing the taxonomic and functional structure of carabid species, soil properties, climatic conditions, landscape and crop properties. The total number of obtained/calculated Figure 2: An area within a radius of 500 m around a sampling attributes used was 95 (Table 1). point for which data about landscape, soil and crop properties was obtained. 2.2 Data pre-processing Table 1: Groups and number of attributes in the final dataset. The collected data were very heterogeneous and as such their Group of attributes Number of attributes harmonization, normalization and aggregation for the purpose of Field description 13 the analysis and modelling was required. To overcome these Species description 7 problems, we used a lot of modelling and ecological knowledge Soil description 7 background. In addition, we followed the standard data pre- Landscape description 48 processing procedure to ensure high quality of the input data, such Climatic conditions 7 as data cleaning, outlier detection, missing value treatment, etc. Temporal component 4 The data describing the abundance of carabid species were highly Functional aspect of species 9 imbalanced. Therefore, we used the Inverse Hyperbolic Sine transformation of the abundance data that were later used for development of the data mining predictive models. 42 2.3 Data mining Table 3: Richness indices, evenness and Hill numbers for the carabid species caught at the experimental sites in different To discover the interactions between the attributes describing years. geographical, environmental and management parameters and abundance of the most abundant carabids, we choose data mining Year Richness metrics Hill numbers methods for induction of decision trees. They are ideally suited Shannon Simpson Evenness N0 N1 N2 for discovery of relations between attributes in complex 2009 1.521 0.335 0.556 58 4.6 3.0 ecological data, because they are interpretable and can provide meaningful explanations of the relationships and causalities 2010 1.542 0.354 0.497 68 4.7 2.8 among attributes. In our case, the dependent variables are the 2011 1.905 0.285 0.439 42 6.7 3.5 abundance of the selected carabid species and the abundance of the predator functional category, which comprises of all predatory 2013 1.550 0.327 0.555 54 4.7 3.1 carabid species that have been caught in the fields by pitfall traps. 2014 1.729 0.316 0.467 60 5.6 3.2 To develop the predictive models, we applied the M5 algorithm 2015 1.733 0.322 0.452 57 5.7 3.1 to induce regression trees using the WEKA data mining software. All 1.680 0.326 0.473 101 5.4 3.1 To evaluate the induced data mining models, we used 10-fold cross-validation as the most common and standard way of The functional characteristics of the three most abundant species estimating the performance of a model on unseen cases [6]. ( Poecilus cupreus, Pterostichus melanarius, Anchomenus 3. RESULTS dorsalis) show that they are all predators throughout all of their life stages (larvae and adults). They avoid dry habitat conditions, For the purpose of biological pest control, the abundance of they are wintering as adults and they reproduce in spring time. predatory and parasitic individuals is as important as diversity. Analysis of the abundance of all caught carabid species in the According to the selection of the independent attributes, we studied area shows that at yearly basis three species significantly constructed two types of models to predict the abundance of prevail over the others (Table 2) with Poecilus cupreus being the carabid species. The first group of models included the attributes most abundant one (Figure 3). Three species appear as the most describing the current structure of the carabid species in the abundant in all sampled years and in total as well (Table 2). sampled fields, such as Hill numbers, evenness, Shannon’s and Simpson’s indexes . This type of prediction models gave us insight Table 2: Relative abundance of carabid species for all years into the interspecies interactions and because of that, we named (2009-2011 and 2013-2015). them “ecological models”. However, to use these models for predictions in reality is very demanding because they require very specific data describing the carabid community, which are very Carabid species 2009-2011 and 2013-2015 (%) hard to obtain. To overcome this practical problem with data, we Poecilus cupreus 53 created a second type of predictive models, where the attributes Pterostichus melanarius 13 describing the community structure were not included as independent attributes. The data required to populate this type of Anchomenus dorsalis 7 models can be easily obtained. This makes the application of the All other carabid species 27 models in reality easier and therefore we named them (98 carabid species) “management models”. Table 4: Validation performances of the regression trees for predicting the abundances of the most dominant carabid species and the two relevant carabid functional groups for biological pest control. Predictive models Correlation Mean absolute coefficient error Ecolog. Manage. Ecolog. Manage. model model model model Poecilus cupreus 0.694 0.628 1.42 1.55 Figure 3: Poecilus cupreus, the most abundant carabid species Pterostichus 0.614 0.542 1.20 1.21 in all sampling years (2009-2011, 2013-2015). melanarius The richness metrics of carabid species are presented in Table 3. Anchomenus 0.318 0.287 1.10 1.15 The values of the Hill number N2 (N2 – number of very abundant dorsalis species), which varies from 2.8 to 3.5, and the evenness (values Predator species – 0.253 0.211 1.48 1.64 from 0.439 to 0.556) are consistent with the results about the rank larvae abundance of carabid species presented in Table 1. A highly Predator species – 0.374 0.328 1.24 1.42 uneven distribution of the abundance is indicated when the adults evenness index has values far from 0 (i.e., equal distribution of the abundances of all species gives values of evenness index close We obtained predictive models (regression trees) for the three to 0). most abundant carabid species (Table 1) and for two functional groups of predator carabid species, where we made a distinction between larval and adult life cycle development stages. The 43 predictive performances obtained using 10-fold cross validation 4. CONCLUSIONS for all induced predictive models are given in Table 4. The total number of instances was 2873. The abundance is predicted in a This study has confirmed the complexity of using empirical weekly time step. ecological data. The data pre-processing was the most demanding and time consuming step in the analysis of the taxonomic and functional diversity of carabid species for the construction of predictive models using data mining methodologies. The results about the taxonomic and functional diversity of carabid species show a great potential for biological pest control in the studied area, as the identified most abundant species of ground beetles are predators of slugs. In addition, the high abundance of the predatory carabids provides an additional quantitative support to the biological control of slugs. However, the predictive performances of the model are not that promising. Despite the large amount of data and the long list of attributes, we were not able to produce trustable predictive models. The main reason could be having a non-optimal selection of the attributes that ARVALIS used for describing the sampling Figure 4: Ecological predictive model for abundance of locations and habitats. The structures of the predictive models Poecilus cupreus. indicate that both carabid community and habitat properties influence the abundance of the predicted species. In addition, the attributes describing the temporal variation of environmental conditions appear in the models as well (e.g., degree days, soil humidity). So the models contain all major ecological components that direct the abundance of carabid species. We can conclude that both the abundance and the diversity (taxonomic and functional) of carabid species in the studied area has high potential to provide efficient pest regulation. Based on the structure of the predictive models, simple guidelines for crop management for enhancement of the biological control of slugs can be proposed (e.g., enlarge woody area, introduce organic or integrated crop production). In addition, we suggest to include additional attributes in the monitoring schema describing habitat conditions that are specific for ground dwelling carabid species. Such additional data would enable us to employ several other data mining methodologies in order to provide significant contribution to the development of efficient biological pest control strategies. 5. ACKNOWLEDGMENTS Figure 5: Management predictive model for abundance of Poecilus cupreus. This research is supported by the applied project “Structured output prediction with application in sustainable agricultural The structure of the ecological model for Poecilus cupreus production” financed by ARRS and co-financed by ARVALIS. (Figure 4) shows the sensitivity of this carabid species on the presence and abundance of other carabid species and on the 6. REFERENCES habitat type. Its highest weekly abundance appears in conditions [1] Chaplinin-Kramer, R., Valpine, P., Mills, N.J., Kremen C., of low number of other carabid species and presence of forest 2013.Detecting Pest Control Services across Spatial and Temporal habitat. Under such conditions, this species gets very abundant, Scales. Agriculture, Eco. and Environment 181 (2013) 206–212. surpasses other species and it becomes the most abundant one. In [2] Legendre P., Legendre L. 2012. Numerical ecology. Elsevier, the case of absence of attributes describing the carabid Amsterdam, Netherlands. [3] Ludwig, J.A., Reynolds, J.F. 1988. Statistical Ecology. New York, community (management model), the abundance of Poecilus Chichester, Brisbane, Toronto, Singapore, John Wiley & Sons. cupreus depends mostly on the quality of the habitat they are [4] ARVALIS 2015. Functional traits of Carabidae species. Technical occupying (Figure 5). In particular, the presence of a woody document and internal dataset. Boigneville, France. habitat nearby stimulates its abundance. Both models gave [5] Debeljak, M., Kuzmanovski, V., Tosser, V., Trajanov, A. 2017. consistent and complementary interpretations, which show that Knowledge discovery from complex ecological data: exploring Poecilus cupreus can be potentially efficient predator of slugs in Syrphidae species in agricultural landscapes. In: Luštrek,M et al. well preserved agricultural environments and in conditions of low (Eds) Proceedings of the 20th International Multiconference diversity of the carabid community. This is consistent with Information Society, volume A. Ljubljana: Institut Jožef Stefan, pp. 55-58. observations where high abundance of Poecilus cupreus in fields [6] Witten, I.H., Frank, E. 2011. Data Mining: Practical Machine is detected in early spring time, particularly if semi-natural Learning Tools and Techniques - 3rd edition. Morgan Kaufmann. habitats, like woodland, are in the vicinity of the fields. 44 Taxonomies for Knowledge Representation of Sustainable Food Systems in Europe Aneta Trajanov Tanja Dergan Marko Debeljak Jožef Stefan Institute and Jožef Stefan Jožef Stefan Institute Jožef Stefan Institute and Jožef Stefan International Postgraduate School Jamova cesta 39 International Postgraduate School Jamova cesta 39 1000 Ljubljana, Slovenia Jamova cesta 39 1000 Ljubljana, Slovenia tanja.dergan@ijs.si 1000 Ljubljana, Slovenia aneta.trajanov@ijs.si marko.debeljak@ijs.si ABSTRACT Instead there exists a set of interrelated concepts and indicators Sustainability is becoming a core concept in every area (scientific, that describe the sustainability from different aspects. The social, environmental and economic) of human life. Sustainability sustainability is defined through three main pillars/aspects: acknowledges that human civilization takes resources to sustain economic, social and environmental pillar/aspect (Figure 1) [7]. our modern way of life and strives towards balancing between our The economic pillar of sustainability represents the economic competing needs – our need to continue developing functions of the food systems, which should provide prosperity technologically and economically, and the need to protect the (wealth) to the (farming) community and thus refers to the environment in which we live. However, sustainability is a very economic viability of the food system. The social pillar represents complex concept that incorporates social, environmental and several social functions, both at the level of the community, as economic aspects and interactions between them and can be well as at the level of society (e.g., awareness and legislation described by a number of different sustainability indicators. protection of the health of people from pollution, or access to Therefore, assessing the sustainability of a system is a demanding basic resources without compromising the quality of life). The task and requires gathering and structuring of knowledge from environmental pillar represents environmental functions that are experts, literature surveys and other sources. In this paper, we connected to the management and conservation of natural present the use of taxonomies to represent the complex concept of resources (water, air, soil, energy and biodiversity) and fluxes sustainability of European food systems. Structuring the within and between these resources [13]. knowledge on sustainable food systems in Europe is a first step in assessing their level of sustainability. The goal of this study is to use the developed taxonomies as basis for the development of a complex DSS system for assessment of the sustainability of legume food systems across the whole quality chain. Keywords Taxonomies, knowledge representation, sustainability, food systems. 1. INTRODUCTION The world’s population is increasing at a speeding rate and with that the production and consumption of food as well. All this comes at an enormous environmental cost. Each year, more than 10 million hectares of arable land are lost to degradation, plant- protection products pollute the rivers and aquifers and one third of all greenhouse gas emissions are due to agriculture [4]. Therefore, a shift to more sustainable agri-food systems is needed in order to address these problems. A formal definition of sustainable food systems given by the Food and Agriculture Organization (FAO) Figure 1. The three pillars of sustainability and their states the following: “A sustainable food system is a food system intersections describing partial (bearability, viability, that ensures food security and nutrition for all in such a way that equitability) and overall sustainability. the economic, social and environmental bases to generate food security and nutrition of future generations are not compromised.” In order to assess the sustainability of a food system, one needs to [2]. understand and take into account all these different aspects of sustainability, which is a demanding task. This paper describes the Assessment of the sustainability of a food system is not an easy first step towards modelling the transition towards sustainable task, as there is not a simple and measurable indicator to assess it. food systems, which is done within the H2020 project TRUE 45 (TRansition paths towards sUstainable legume based systems in Sustainability aspect Europe) [12]. In order to set the foundations for the development of a Decision Support System (DSS) for sustainability assessment of legume systems, we carried out an extensive literature survey in order to capture as extensive knowledge as possible on sustainable Sustainability Sustainability level Sustainability indicators Sustainability assessment food systems and all the concepts and indicators connected to that. The knowledge and concepts were organized in a Figure 2. Top level of the taxonomy, decomposing the hierarchical structure using taxonomies. This kind on knowledge sustainability aspect into four sublevels: sustainability, has not been represented in an organized, systematic and formal sustainability level, sustainability indicators and sustainability way so far. Using these taxonomies, we wrote a glossary of terms, assessment. which will serve as a knowledge library when constructing the DSS system. 3.1 Sustainability The general sustainability as described in the Introduction section, 2. MATERIALS AND METHODS consists of three sustainability pillars: environmental, social and economic [5, 11]. True sustainability requires a balance between In order to produce a working protocol for harmonization of data the environmental, social and economic aspects describing it. and knowledge to develop the future DSS within the TRUE Besides these, the intersections between them ( bearability, project, we had to derive definitions of sustainability terms and viability and eqitability) are also an important partial aspect of the concepts, and review as much sustainability indicators as possible, sustainability as a whole (Figure 3) [5]. which are non-deterministic and ambiguous. For that purpose, we reviewed more than 24 papers and 7 books dealing with different Environmental aspects of sustainability of (legume) food systems. Pil ars Economic The obtained knowledge from the extensive literature survey was Social structured using taxonomies. Taxonomies, like ontologies, Environmental provide ordered/structured representation of concepts and terms in Bearable Sustainability Social a form of a hierarchy. They are semantic classification schemes Environmental and represent a knowledge map [6]. They are classification Intersections Viable schemes, because they group related things together, so that if you Economic search one thing within a category, it is easy to find other related Economic Equitable things in that category. They are semantic because they provide a Social vocabulary to describe the knowledge in them. Finally, if the taxonomy is complete, it should provide an immediate grasp of Figure 3. Decomposition of the “Sustainability” part of the the overall structure of the knowledge domain it covers. taxonomy. Many of the taxonomies have hierarchical tree structures. The tree Bearability is the intersection between the environmental and structure is the most intuitive representation, because it provides a social sustainability pillars. It represents a system that is both visual representation of the relationships between categories and environmentally and socially sustainable, but lacks an economic sub-categories, enabling navigation between categories. However, sustainability [11]. Viability is the intersection between the they can be represented in other forms, such as: environmental and economic sustainability pillars, and  Lists e quitability is the intersection between the economic and social sustainability pillars.  Trees  Hierarchies 3.2 Sustainability Level  The next part of the taxonomy represents the concepts connected Polyhierarchies to the sustainability levels, with respect to the different spatial and  Matrices temporal scales of sustainability. The sustainability levels can be  observed and defined through different aspects (Figure 4): Facets  System maps.  Normative The taxonomies presented in this paper are represented in a tree  Spatial structure and discussed in the Results section.  Temporal 3. TAXONOMIES FOR SUSTAINABLE  Systems FOOD SYSTEMS: RESULTS AND Normative level refers to the building blocks (aspects) of DISCUSSION sustainability (environment, economic, social), which we The taxonomy describing the knowledge on sustainability of food described earlier, as well as their interactions (equitability, systems in Europe starts by the general sustainability aspect of viability, bearability) [3]. European food systems (Figure 2). It incorporates sustainability in The spatial level of sustainability refers to the spatial specifics of its general form, sustainability level, sustainability indicators and sustainability. Sustainability of a food system can be sustainability assessment. considered/assessed on a local, regional, national or international level [3]. 46 The temporal aspects of sustainability refers to the time horizon of Sustainability indicators should satisfy certain criteria, which the sustainability assessment. represent specific objectives relating to a state of the system. The criteria should consider the environmental, economic and social Finally, the boundary of the (food) system under consideration characteristics of the system. They must provide specific should be defined, the hierarchy of aggregation levels and their conditions for the development of sustainability indicators that interactions for descriptive, assessment and management purposes will have analytical soundness and will be measurable and in relation to sustainable development, which gives us the system suitable for application at different scales (e.g., farm, district, level of sustainability [3]. country, etc.). 3.3 Sustainability Indicators The actual indicators are variables of any type that can be induced Indicators are quantified information, which explain how things from the sustainability criteria and can provide information about are changing over time. The sustainability indicators measure the the potential or realized effects of human activities on the sustainable development and its progress. They have to reflect the sustainability of the food system. These are variables that can be definition of sustainability and be able to connect partial used to assess both the socio-economic and environmental conditions to policies for sustainable development and monitoring conditions of the food system, to monitor trends and conditions its progress [9]. Indicators are used to compare the actual state of over time, to provide early warning signal of change and a solid the system with reference values for sustainability (sustainability basis for decision making processes, consistent with sustainable assessment), or with the state of the system in the past and in the development principles at all levels [1, 10]. The indicators can be future (sustainability monitoring) [13]. The part of the taxonomy also used to reduce the complexity of the system description and addressing the Sustainability indicators is given in Figure 5. integrate information about processes, trends or states into a more readily understandable form at local, regional and global levels. Environmental Pil ars Economic Social Environmental Bearable Normative Social Environmental Intersections Viable Economic Economic Equitable Social Local Sustainability level Regional Spatial National International Short term Static Long term Temporal Short term Dynamic Long term Descriptive Systems Assessment Management Figure 4. Decomposition of the “Sustainability level” part of the taxonomy. Dimensions Criteria Measures Local Level Regional National/International Trand Time Indicators State Driving Type Function Response Sustainability indicators Single Structure Composit Thresholds Absolute Target values Reference value Comparison Relative Trend Scientific quality Selection System relevance Data management Figure 5. Decomposition of the “Sustainability indicators” part of the taxonomy. 47 The indicators can have different levels [3, 8, 10]: The obtained knowledge on sustainability of European food  systems, represented using taxonomies will represent the basis for Local – measure the state of the system regarding the development of a complex Decision Support System for sustainability finding transition pathways towards sustainable legume-based  Regional – compare the system’s performance from an food systems. economic, social and environmental aspect  5. ACKNOWLEDGMENTS National/international level – inform policy makers This research is supported by the TRUE project, funded by the about the current state and trends in sector performance EU Horizon2020 Research and Innovation Programme, Grant and facilitate public participation in sustainability Agreement number 727973. discussions. The type of indicators refers to their functional category according 6. REFERENCES to their purpose of use. According to the type, the indicators can [1] Dale, V.H., Efroymson, R.A., Kline, K.L., Langholtz, M.H., describe [1]: Leiby, P.N., Oladosu, G.A., Davis, M.R., Downing, M.E.,  Time-related issues: Hilliard, M.R. 2013. Indicators for Assessing Socioeconomic Sustainability of Bioenergy Systems: A Short List of Practical o Trend indicators – describing temporal Measures. Ecological Indicators, 26: 87–102. dynamic aspects of sustainability over time [2] FAO, 1984. Potential population supporting capacities of lands o in the developing world. FAO, (Food and Agriculture State indicators – describing the condition of Organization of the United Nations), Rome. the assessed system at a certain time point [3] Hayati, D., Ranjbar, Z., Karami, E. 2010. Measuring  Function-related issues: Agricultural Sustainability. In: Biodiversity, Biofuels, Agroforestry and Conservation Agriculture, Eds. Eric o Driving indicators – measure the changes of Lichtfouse, 5:73–100. Dordrecht: Springer Netherlands. the assessed food system due to management practices and other farming inputs [4] HLPE, 2014. Food losses and waste in the context of sustainable food systems. A report by the High Level Panel of o Response indicators – measure the response Experts on Food Security and Nutrition of the Committee on of a system to the induced management World Food Security, Rome 2014. changes or inputs [5] Kwami, H.I, Che-Ani, A.I., Tawil, N.M., Tahir, M.M., Basri,  H. 2014. Approach to Campus Sustainability at Universiti Structure-related issues: Kebangsaan Malaysia (UKM): A Review. Eds M.A. Othuman o Single indicators – characterizing single parts Mydin and A.I. Che Ani. E3S Web of Conferences 3: 01011. of the food system [6] Lambe, P. 2007. Organising knowledge: Taxonomies, Knowledge and Organisational Effectiveness. Chandos o Composite indicators – aggregate Publishing, Oxford, England. environmental, social and economic indicators [7] into a unique measure describing complex Moldan, B., Janouskova, S., Hak, T., 2012. How to understand and measure environmental sustainability: indicators and functions and processes of the system. targets. Ecological Indicators, 17: 4–13. The reference values of the sustainability indicators refer to the [8] Muñoz, Lucio. 2016. Linking Sustainable Development desired level of sustainability for each indicator. They are Indicators by Means of Present/Absent Sustainability Theory established on scientific or empirical basis and can be presented and Indices: The Case of Agenda 21, GDS, IIG, Spain. as absolute (fixed) values, as threshold values, or as relative [9] Pollesch, N., Dale, V.H. 2015. Applications of Aggregation reference values according to a selected baseline [13]. Theory to Sustainability Assessment. Ecological Economics, 114: 117–27. Finally, the selection of indicators should be made according to their scientific quality, system relevance and data management [10] Samuel, V.B., Agamuthu, P., Hashim, M.A. 2013. Indicators principles. for Assessment of Sustainable Production: A Case Study of the Petrochemical Industry in Malaysia. Ecological Indicators, 24: 392–402. 4. CONCLUSIONS [11] Slocum, S.L. 2015. The Viable, Equitable and Bearable in Structuring and organizing domain knowledge in a structured and Tanzania. Tourism Management Perspectives, 16: 92–99. formalized way, using taxonomies, is a crucial first step towards understanding complex problems and concepts. It also represents [12] TRUE (TRansition paths towards sUstainable legume based a crucial pre-processing step, which sets the basis for data mining systems in Europe), 2018. H2020 project, https://www.true- project.eu (11 September 2018). analyses or development of Decision Support Systems. [13] Van Cauwenbergh, N., Biala, K., Bielders, C., Brouckaert, V., Within the TRUE project, the taxonomies for knowledge Franchois, L., Cidad, V.G., Hermy, M., Mathijs, E., Muys, B., representation of sustainable food systems in Europe were Reijnders, J., Sauvenier, X., Valckx, J., Vanclooster, M., der “translated” into a glossary of terms, which will be evaluated by a Veken, B.V., Wauters, E., Peeters, A. 2007. SAFE – a group of domain experts. In that way, they will validate and hierarchical framework for assessing the sustainability of confirm the structure of the taxonomy. agricultural systems. Agric Ecosyst Environ, 120: 229–2 48 Uporaba povezave kalkulacijskega simulacijskega modela z analizo tveganja pri podpori odločanja v kmetijstvu Tanja Dergan Aneta Trajanov Marko Debeljak Institut Jožef Stefan Institut Jožef Stefan Institut Jožef Stefan Jamova cesta 39, 1000 Ljubljana Jamova cesta 39, 1000 Ljubljana Jamova cesta 39, 1000 Ljubljana Slovenija Slovenija Slovenija tanja.dergan@ijs.si aneta.trajanov@ijs.si marko.debeljak@ijs.si POVZETEK kmetijskih pridelovalnih zemljišč za samooskrbo pridelave hrane [2]. Učinkoviti sistemi ekonomskega upravljanja in obvladovanja Kmetijska gospodarstva se pri načrtovanju pridelave neprestano tveganja bodo omogočili učinkovitejše upravljanje s tem odločajo o načinu proizvodnje, izbiri poljščin in količini naravnim virom ob hkratnem zagotavljanju kvalitetnega pridelave. Pri tem se soočajo z vprašanjem kako ob čim manjšem življenjskega standarda prebivalstva, ki mu kmetijstvo predstavlja ekonomskem vložku ter obvladovanju pridelovalnih in okoljskih osnovno ekonomsko dejavnost. Od tega je odvisna tudi tveganj zagotoviti optimalen ekonomski rezultat. Raziskava sposobnost ohranjanja kmetij in agrarne krajine. obravnava sistem za podporo odločanja, ki temelji na integraciji Kmetijska gospodarstva se morajo neprestano prilagajati simulacijskega modela ekonomske kalkulacije in sistema analize spremembam lokalnih in globalnih družbeno ekonomskih ekonomskega tveganja na primeru pridelave ječmena, fižola in dejavnikov. Okoljska in gospodarska razvojna politika od koruze v premenah hmelja. Rezultati simulacijskih modelov in pridelovalcev zahteva povečevaje pridelave in izboljševanje njene analiz tveganja so pokazali, da je v premenah hmelja najboljše kakovosti ob hkratnem upoštevanju vse bolj zahtevnih kriterijev ekonomske kriterije dosegla pridelava fižola. Zaključki naše trajnostnega kmetijstva. raziskave govorijo v prid povečanju pridelave stročnic, ki predstavljajo trajnostno možnost za povečanje samooskrbe z Pridelava hmelja predstavlja eno od zelo potencialnih kmetijskih rastlinskimi beljakovinami in zmanjšanju obremenitve okolja z dejavnosti glede možnosti izpolnjevanja kriterijev in zahtev negativnimi vplivi kmetijske pridelave. sodobne družbe glede trajnostne pridelave hrane [5]. Hmelj je trajnica in kot večletna monokultura negativno vpliva na kvaliteto Ključne besede tal. Povečuje zbitost tal in zmanjšuje količine aktivnega humusa v Sistem za podporo odločanja, kmetijska pridelava, ekonomske tleh [9]. Da bi odpravili tovrstne negative vplive na tla in da bi kalkulacije, ekonomsko tveganje, hmeljarstvo, trajnostno nasade zavarovali pred povzročitelji bolezni in škodljivci, se na kmetijstvo površinah za pridelavo hmelja, hmelj za krajše obdobje nadomesti z drugimi poljščinami, kar imenujemo premena [3]. 1. UVOD Osrednji cilj raziskave je izgradnja sistema za podporo odločanja, V kmetijstvu ekonomsko upravičenost skupnih stroškov pridelave ki temelji na integraciji simulacijskega modela ekonomske ugotavljamo s pomočjo ekonomskih kalkulacij, ki so eno od kalkulacije in sistema analize ekonomskega tveganja na primeru osnovnih orodji za planiranje in podporo odločanja v kmetijskem pridelave ječmena, fižola in koruze v premenah hmelja. S menedžmentu. Na podlagi kalkulacij so ocenjeni skupni stroški pomočjo izbranih kriterijev tveganja ocenimo posamezne pridelave in izračunani indikatorji ekonomske uspešnosti, kar alternative, ter izberemo tisto, ki je glede na specifiko konkretnega predstavlja osnovo za nadaljnje načrtovanje kmetijske pridelave primera premene hmelja najustreznejša. Izračuni, ocenjeni na [7]. osnovi uporabljene metodologije tveganja in predpostavljenih ekonomskih parametrov pridelave na modelni kmetiji, Tveganje je pomemben poslovni vidik v kmetijstvu. Visoka predstavljajo pomembno podporno orodje za nosilčeve nadaljnje proizvodnja in cenovna tveganja so postala stalnica kmetijske odločitve. proizvodnje in so v večji meri odraz nepredvidljivega obnašanja naravnih dejavnikov (vreme, škodljivci, bolezni, itd.). Uspešnost Preostali del prispevka je strukturiran kot sledi. Opis podatkov in obvladovanja teh tveganj vpliva na ekonomsko uspešnost zasnova so predstavljeni v poglavju 2. Sledi opis obravnavanih kmetijskih gospodarstev, saj napačne odločitve hitro vodijo v metod v poglavju 3. Rezultate razprave predstavimo v poglavju 4 zmanjšanje dohodka. Kmetijski pridelovalci so pri tem soočeni s in zaključimo v poglavju 5. kompleksnostjo odločevalskega problema saj so primorani sprejemati številne odločitve tako na vsakdanjem nivoju, kot tudi 2. PODATKI na nivoju dolgoročnejših investicij [6]. Raziskava analizira podatke pridobljene iz referenčne kmetije Doseganje boljšega in predvsem stabilnejšega rezultata kmetijske izbrane v Žalcu v spodnji Savinjski dolini. Kmetija je poljedelsko- pridelave je zato zelo odvisno od obvladovanja tveganja v procesu živinorejska, integrirano usmerjena, s 25 ha pridelovalne površine odločanja. V načrtovanju kmetijske proizvodnje zato spremljanje hmelja. V premenah hmelja trenutno pridelujejo izmenično in ocenjevanje tveganja predstavlja zelo pomemben segment v koruzo, fižol in ječmen (Slika 1). V prihodnosti želijo kmetijo procesu spremljanja in načrtovanje kmetijske pridelave [4]. preusmeriti zgolj v poljedelsko dejavnost. Podatke, ki opisujejo trenutno stanje na kmetiji smo pridobili z osebnim intervjujem Zaradi velike odvisnosti od uvoza hrane in krme se v Sloveniji lastnika kmetije in sodijo v obračunsko leto 2017. premalo zavedamo kako pomembno je varovanje in ohranjanje 49 Lastna cena (LC) predstavlja višino skupnih stroškov pridelave za proizvodnjo enote izbranega pridelka (proizvodnja enota pridelka je definirana kot vrednost pridelave za 1 kg pridelka). Izračunan je kot koeficient med skupnimi stroški pridelave (SS) in količino pridelka (Y) in je ekvivalent prelomni ceni proizvoda [11]. KE = SP/SS Koeficient ekonomičnosti (KE) predstavlja razmerje med skupnimi prihodki (SP) in skupnimi stroški pridelave (SS). Z njim ugotavljamo gospodarnost poslovanja. Če je koeficient ekonomičnosti večji od ena pomeni, da je poslovanje ekonomično in obratno [11]. Glavni namen kalkulacij je spremljanje skupnih stroškov pridelave. S tem pridobljene informacije predstavljajo kmetu Slika 1: Premena hmelja z visokim fižolom osnovno informacijsko podporo za odločanje v načrtovanju proizvodnje, načrtovanju nadaljnjih investicij in ostalih aktivnosti na kmetiji. 3. METODE 3.2 Analiza tveganja v kmetijstvu Za reševanje problemov ekonomske upravičenosti pridelave in Deregulacija trgov, spremembe glede potreb po hrani in druge tveganja smo uporabili izviren pristop integracije dveh sicer družbene zahteve (npr. trajnost), ter vplivi podnebnih klimatskih pogosto posamično uporabljenih metod, ki jih uporabljamo v sprememb vodijo v vedno večja cenovna, pridelovalna in načrtovanju pridelave na nivoju kmetije. Z njuno uporabo lahko z dohodkovna nihanja in posledično s tem tudi v povečevanje različnih zornih kotov ovrednotimo razvoj kmetijskih tveganja v kmetijstvu [1]. Tveganje na kmetijskih gospodarstvih gospodarstev in pokažemo možnost povečanja dodane vrednosti v ne smemo zanemariti, saj poskušajo nosilci odločanja v kmetijstvu kmetijstvu. tveganja obvladati in predvsem tudi zmanjševati [6]. 3.1 Kalkulacijski modeli Med kmetijskimi gospodarstvi obstajajo pomembe razlike v Ekonomske kalkulacije v kmetijstvu zahtevajo uporabo izpostavljenosti, zaznavanju in upravljanju tveganja. Učinkovito kvalitetnih vhodnih podatkov [10], ki smo jih v našem obvladovanje tveganja je eden izmed ključnih dejavnikov konkretnem primeru zagotovili. uspešnega poslovanja. Nosilci kmetijskih gospodarstev lahko pri upravljanju s tveganjem izbirajo med različnimi ukrepi in orodji. Kalkulacije so samostojni simulacijski modeli, ki na podlagi Kaj bo posameznik izbral, pa je odvisno od njegove naklonjenosti opredeljenih (izbranih) vhodnih atributov omogočajo oceno tveganju in okoliščinam v katerih kmetuje [12] . porabe vnosov v kmetijsko proizvodnjo (npr. semena, gnojila, krmila, škropiva, gorivo, najem strojev…) in s tem oceno skupnih Za analizo tveganja je potrebno zapisati tako imenovano tabelo stroškov pridelave kmetijskih pridelkov oz. proizvodov [7]. odločanja (Tabela 1), ki se uporablja pri vseh kriterijih tveganja. Poraba vnosov je odvisna od intenzivnosti pridelave, pridelovalne V tabeli prikažemo vse alternative (A) kot vrstice (kjer i= 1, površine, oddaljenosti od kmetije, nagiba terena itd. Za razliko od 2,…m) in stanje (S) kot stolpce (kjer j=1, 2,…n). Rij nam t.i. kalkulacij pokritja, modelne kalkulacije pri posameznem ponazarja ekonomski donos za izbrano alternativo Ai, če pride do pridelku neposredno vključujejo vse skupne stroške pridelave, ki stanja Sj in p nam ponazarja porazdelitev verjetnosti, ki velja za S so povezani s proizvodnjo in ne samo spremenljive stroške, kar (niz verjetnosti pj, ki opisuje verjetnost, da se bo stanje Sj omogoča tudi neposredno primerjavo skupnih stroškov pridelave zgodilo). s skupnim prihodkom ter izračunom različnih ekonomskih Tabela 1: Tabela odločanja kazalcev. Za potrebo naše raziskave smo uporabili v nadaljevanju na kratko opisane kazalce: A S SS=FS+ VS S1 S2 … Sn Skupni stroški pridelave (SS) predstavljajo seštevek fiksnih p1 p2 … pn stroškov (FS) (zavarovanja, obresti kreditov, plače delavcev) in A1 R11 R12 … R1n variabilnih stroškov (VS) (stroški ki so odvisni od obsega proizvodnje, npr. stroški amortizacije…) [11]. A2 R21 R22 … R2n … FR=VP-SS … … … Finančni rezultat (FR) predstavlja razliko med skupnim pridelkom Am Rm1 Rm2 … Rmn proizvodnje (VP) in skupnimi stroški pridelave (SS) [11]. VP = Y X Cy X PR +Y1 X Cy1 V naši raziskavi smo kot modelno poljščino obravnavali pridelavo Vrednost pridelave (VP) predstavlja zmnožek količine pridelka hmelja, kjer se v premenah na isti pridelovalni površini (Y) in njegove cene (C pridelujejo tudi druge poljščine. Pridelavo poljščin smo analizirali y), pomnoženega s površino pridelka (PR), k temu pa prištejemo še količino stranskega pridelka (Y1), (npr: s pomočjo petih kriterijev tveganja, ki so v naslednjih pri koruzi je stranski pridelek slama), pomnoženega s ceno podpoglavjih na kratko opisani. Pri tem je potrebno poudariti da stranskega pridelka (C uporaba različnih kriterijev, lahko pripelje do izbere različnih y1) [11]. alternativ. LC = SS/Y 50 3.2.1 Waldov kriterij (MaxMin) 3.2.5 Laplaceov kriterij Waldov kriterij ali MaxMin kriterij je kriterij pesimizma, kjer Ta kriterij je realističen, ter izhaja iz principa maksimalnega odločevalec upošteva le najmanjše vrednosti alternativ in izbere neznanja. Po Laplaceovem kriteriju predpostavljamo, da so vsi alternativo ki ima izmed najnižjih najvišje rezultate. MaxMin rezultati enako verjetni, kjer vrednosti med seboj seštejemo ter kriterij vpliva na odločevalčevo zavest, kateri si prizadeva določimo alternative z najvišjo dano vrednostjo. Odločevalec zagotoviti, da v primeru negativnega izida, obstaja vsaj minimalno lahko izračuna iz vsake vrstice tabele odločanja (Tabela 1) izplačilo[8]. povprečno vsoto, kot rezultat pa izberemo najvišjo povprečno vrednost vrstice [8]. Pravilo Laplaceove odločitve: 3.2.2 MaxMax kriterij 1. Dodajte p Kriterij imenujemo tudi optimističen kriterij in je ravno nasprotje j = P (Sj) = 1 / n vsaki Sj v S, za j = 1, 2, ..., n. MaxMin metode. Je optimističen in agresiven pristop k odločitvi 2. Za vsako Ai (vrstica matrike izplačil) izračuna pričakovano pod pogojem negotovosti. Z MaxMax kriterijem bo odločevalec vrednost (E): E (Ai) = Σj pj (Rij) = pj Σj Rij. vedno računal na najboljši izid pri vsaki alternativi. MaxMax pravilo je primerno za ekstremne optimiste, kateri pričakujejo 3. Kot optimalno odločitev izberemo najboljšo vrednost najbolj udoben položaj [8]. alternative E(Ai), ki najboljše predstavlja dobiček in minimalno 3.2.3 Hurwizcev kriterij (H) predstavlja stroške. Hurwizc pristop poskuša vzpostaviti ravnovesje (sredino) med MaxMax in MaxMin kriteriji. Namesto ob predpostavki popolnega 4. REZULTATI IN RAZPRAVA optimizma ali pesimizma, Hurwicz uporabi parameter (α), ki je na Raziskava je zasnovana v dveh sklopih. V prvem sklopu smo intervalu med 0 in 1 in jo odločevalec izbere subjektivno [8]. razvili modelno matematično orodje za analizo ekonomske upravičenosti posamezne pridelave. Pridobljene rezultate smo Če je vrednost α bližje 1, pomeni absolutni optimizem in velja nato v drugem sklopu prenesli in uporabili pri analizi tveganja v MaxMax kriterij (maksimalna vrednost maksimalne vrednosti kmetijstvu. Z integracijo kalkulacije in tveganja smo odločevalcu prihodka na letni ravni) omogočili dodatno natančnejšo vrednotenje alternativ ter s tem Če je vrednost α bližje 0, pomeni daje absolutni pesimizem in zanesljivejše odločanje. velja Waldov MaxMin kriterij (maksimalna vrednost minimalne vrednosti prihodka na letni ravni) 4.1 Rezultati kalkulacij S simulacijskim kalkulacijskim modelom smo za vsako od treh Vrednost α nam hkrati poda koeficient pesimizma 1-α, ki odraža obravnavanih poljščin (fižol, ječmen in koruzo) ocenili njihove odločitev odločevalca na tveganje. Hurwiczovo tehtno povprečje glavne ekonomske kazalce, predstavljene v poglavju 3.1 (Skupne se sedaj lahko izračuna za vsako alternativo Ai [8]. stroške pridelave (SS), skupne prihodke (VP), lastno ceno (LC), H(A koeficient ekonomičnosti (KE) in končni finančni rezultat (FR)), i) = α (vrstica alternative z maximum vrednostjo) + (1- α ki temeljijo na podatkih iz obravnavane kmetije. Izračuni so )(vrstica alternative z minimum vrednostjo) je pozitiva (dobiček, pokazali, da so največji skupni stroški pridelave nastali pri prihodek) pridelavi fižola, medtem ko so najnižji skupni stroški pridelave H(Ai) = α (vrstica alternative z minimum vrednostjo) + (1- bili pri ječmenu (Tabela 2). Analiza ekonomičnosti pridelave je α)(vrstica alternative z maximum vrednostjo) je negativna pokazala, da je najvišjo vrednost pri glavnih ekonomskih (stroški, prihodek) kazalnikih dosegala pridelava fižola, najslabše rezultate pa je kljub boljši lastni ceni od koruze prejela pridelava ječmena 3.2.4 Savageov kriterij (MinMax) (Tabela 2). Z drugim imenom poznan tudi kot MinMax kriterij obžalovanja. Tabela 2: Rezultati analize ekonomičnosti Je pesimistični pristop, ki proučuje obžalovanje, izgubo [8]. Ta kriterij se povsem osredotoča na izogibanje najhujših možnih Skupni Finančni Lastna Vrednost KE posledic, ki lahko nastanejo pri odločanju [8]. stroški rezultat cena proizvodnje pridelave (eur/ha) (eur/kg) (eur/ha) Izguba priložnosti (OL) se definira kot razlika izplačil med (eur/ha) najboljšim možnim izidom pod pogojem Sj in dejanskim Fižol rezultatom iz izbire A 4378 3032 0,89 7411 1,7 i, če se pojavi Sj. To pomeni, da če izbrana alternativa poda najvišjo možno vrednost, potem izguba ni Koruza 2026 1303 0,10 3330 1,6 prisotna in je definirana kot vrednost nič. Ječmen 1505 457 0,28 1962 1,3 Savageov kriterij je definiran kot: OLij = (stolpec stanja j maximum izplačil) - Rij, je pozitivna (upoštevamo dobiček, prihodek) Koeficient ekonomičnosti (KE) je vseh treh primerih večji od ena (Tabela 2), kar pomeni da je prodajna vrednost večja od skupnih OLij = Rij - (stolpec stanja j minimum izplačil) je negativna ( upoštevamo stroškov pridelave (poglavje 3.1). Pridelava fižola je tako kljub vrednost stroškov) najvišjim skupnim stroškom pridelave, iz ekonomskega vidika Rij ponazarja vsoto vrstic i in stolpcev j v tabeli odločanja najustreznejša poljščina za pridelavo v premeni hmelja (Slika 2). (poglavje 3.2). Najboljši možen rezultat je 0 – kar pomeni, da ni obžalovanja. Višja kot je vrednost rezultata, večje je obžalovanje za odločitev. 51 5. ZAKLJUČEK Cilj raziskave je bil izgradnja sistema za podporo odločanja, ki temelji na kalkulacijskem modelu za oceno kmetijske pridelave ječmena, fižola in koruze v premenah hmelja, ter s pomočjo pomembnejših kriterijev tveganja opredeli katera izmed izbranih alternativ je za izbrano kmetijo ekonomsko najugodnejša. Predstavljeno metodo lahko zlahka prenesemo v proces izbire tudi drugih poljščin in v druge pridelovalne procese. Želeli smo oblikovati trajnostne smernice za razvoj kmetije in z izračuni ocenjenimi na osnovi uporabljene metodologije razviti podporno Slika 2: Finančni rezultat kalkulacij orodje za pridelovalčeve nadalje odločitve, ki bodo poleg ekonomskih kazalnikov upošteval tudi tveganje, ki je v kmetijstvu 4.2 Rezultati tveganja močno prisotno. Zavedati se moramo da je znižanje tveganja V okviru ekonomske analize smo kot vhodne podatke za nadaljnjo možno le do določene mere in da se bo tveganje v prihodnje, analizo kriterijev tveganja uporabili finančne rezultate verjetno samo še povečalo. Ključno merilo tveganja je pridobljenih v prvem delu raziskave (poglavje 4.1). variabilnost obsega proizvodnje, ki je tesno povezana z tipom Tabela 3 prikazuje tabelo odločanja (poglavje 3.2), ki prikazuje proizvodnje. Tako je npr., variabilnost rastlinske pridelave bolj povezana z vremenskimi vplivi kot živinorejska. vrednosti prihodka na letni ravni in sicer glede na različne Ob tem velja izpostaviti, da tveganje samo po sebi še ne pomeni škode. Ta scenarije potencialne prodaje pridelka (p), (100%, 80% in 50% možnostjo prodaje). nastane takrat, ko se predvideno tveganje tudi zgodi, kar povzroči negativne ekonomske posledice. Tabela 3: Tabela odločanja vrednosti prihodka različnih poljščin (alternativ) 6. ZAHVALA na letni ravni glede na različne scenarije Raziskavo podpira TRUE projekt, ki ga financira program EU za prodaje raziskovanje in inovacije Horizon2020, sporazum o dodelitvi Možni scenariji sredstev št. 727973. Možni Scenarij 1 (p1 = Scenarij 2 ( p2 = Scenarij 3 ( p3 = scenariji 1.00 ): 100% 0.80 ): 80% 0.50 ): 50 % prodaja (eur/ha) prodaja (eur/ha) prodaja (eur/ha) 7. LITERATURA A1 - fižol 3032 2426 1516 [1] Almin H. 2010. Uncertainty Aversion and Risk Management in Agriculture. Agriculture and agricultural Science Procedia A2 - koruza 1304 1043 652 1. 152-156 str. A3 - ječmen 457 366 228 [2] Bavec F. 2001. Pridelovanje nekaterih poljščin v ekološkem kolobarju. Ekološko kmetovanje. 158 – 231 str. Če primerjamo rezultate analize tveganja pridelave obravnavanih poljščin [3] Džuban T. s sod. 2008. Tehnološka navodila za intergrirano (Tabele 4) vidimo, da Waldov kriterij predlaga pridelavo poljščin. Ministerstvo za kmetijstvo, gozdarstvo in alternativo A1 - fižol z dobičkom 1516 EUR/ha. Najvišji donos za pridelovalca kažejo izračunane vrednosti za MaxMax kriterija, ki prehrano, Ljubljana. Str: 50. predlaga alternativo A1 - fižol, v vrednosti 3032 EUR/ha. Prav [4] Girdžiute L. 2012. Risk ain agriculture and opportunities of tako prikaže najboljši donos alternative A1 - fižol Hurwicz in their integrated evaluation. Proc. Soc.Beh.Sci: str. 783-790. Laplacov kriterij. Poudariti je potrebno da smo pri Hurwicz [5] Hacin J. 1982. Podorine v hmeljarstvu. Hmeljar.4. Str: 3-4. kriteriju uporabili koeficientom optimizma, ki je v našem primeru znašal (k=0,7), ter koeficientom pesimizma, ki smo ga izračunali [6] Hardaker J.B., Huirne R.B. M., Anderson J.R., Lien G. 2007. kot: 1- koeficient optimizma (k=0,3). Po Savage kriteriju pa se Coping with Risk in Agriculture (2nd ed.). Oxfordshire: odločimo za alternativo A2 - pridelava koruze, s čim bi dosegli CABI Publishing 1728 EUR/ha. [7] Pažek K., 2003, Finančna analiza ocenjevanja investicij Tabela 4: Skupni rezultati analiz tveganj dopolnilnih dejavnosti na ekološki kmetiji: Mag. Maribor Kriterij tveganja Odločitev [8] Pažek, K., Rozman Č. 2008. Decision Making under Conditions of Uncertainty in Agriculture: Case study of oil Waldov kriterij pesimizma - MAXMIN A1 – fižol crops. Original scientific paper Kriterij optimizma - MAXMAX A1 – fižol [9] Plazovnik R. M. 2008. Krmne poljščine v slovenskih hmeljiščih v premeni. Diplomsko delo. Univerza v Ljubljani Hurwiczov kriterij za koef.optimizma = 0,7 A1 – fižol [10] Radnak M. 1998. Splošna izhodišča in metodologija izdelave Savageov kriterij - MINMAX A2 - koruza modelnih kalkulacij za potrebe kmetijske politike. Prikazi in informacije 189. Ljubljana: Kmetijski inštitut Slovenije Laplaceov kriterij A1 - fižol [11] Vinčec J. 2010. Analiza poslovanja na zelenjadarski kmetiji. Diplomsko delo. UM. Dislocirana enota Rakičan Rezultati so pokazali, da je z ekonomskega vidika kot najbolj [12] Žgajnar J. 2013. Možnosti upravljanja s tveganji v primerna odločitev izbira alternative A1 - fižol. Slednje pomeni, kmetijstvu.6. konferenca DAES. Orodje za podporo da je ekonomsko najučinkovitejše, da obravnavana kmetija v odločanju v kmetijstvu in razvoju podeželja. Krško.Str.15-32 premeni hmelja prideluje fižol. 52 Hierarchical Multi-label Classification for Activity Recognition Nina Reščič Mitja Luštrek Jožef Stefan Institute Jožef Stefan Institute International Postgraduate School Jožef Stefan International Postgraduate School Jožef Stefan Jamova cesta 39 Jamova cesta 39 1000 Ljubljana 1000 Ljubljana nina.rescic@ijs.si mitja.lustrek@ijs.si ABSTRACT gether, the usage of hierarchical classification for AR has Activity recognition using wearable sensors is very impor- only been addressed a few times. None of the cases was tant in many domains of health monitoring and is therefore specifically directed towards usage of wrist-worn device for well researched. Most commonly classification considers all recognizing different hierarchical activities (physical, daily, activities to be ’equal’ (we will use term flat classification). hand-movement activities). Khan and al. [8] proposed a However, intuition suggest better results could be achieved hierarchical recognizer for recognition of limited amount of using a hierarchical approach for classification. In this paper physical activities (static, transitions, dynamic) using a chest- we compare three different approaches to classify activities: worn sensor device. Zheng [9] explored human activity based (i) Flat classification - classes are equal and we build one on the hierarchical feature selection and classification frame- model to classify all of them; (ii) Multi-model hierarchical work. He explored 2D and 3D motion (jumping, running, classification - classes are arranged in trees, we build differ- walking forward/left/right, upstairs/downstairs, static ac- ent models to classify activities on different levels. We apply tivities). two different approaches; (iii) Hierarchical classification us- ing CLUS software1. 2. DATASET The dataset we are working with consists of data from seven Keywords people involved in different activities (sport, rest, handwork, Activity recognition, hierarchical multi-label classification, eating chores...). We organize the activities in hierarchy as wearable sensors presented in Table 1. First we tried to create structure tree by using Orange2 software for hierarchical clustering. We calculated features as will be explained later in paper and 1. INTRODUCTION put them into Orange software. We were looking for some Activity recognition (AR) using wearable sensors has been indications of the hierarchy for different groups of activity. addressed many times, some of the most important applica- However, there was no clear or extremely obvious structure tion being personalized health systems. Many of developed visible. The final structure was designed using knowledge methods for recognizing different activities used triaxial ac- achieved from previous research on the same dataset where celerometers worn on different body parts. With develop- flat classification (for instance in research made by Cvetkovic ment of wrist-worn devices in past several years and with et al. [4]) has been used for recognition of activities. their growing popularity in everyday life, methods for rec- ognizing sports activities [2], daily activities [3] and hand- specific activities [4] using just wrist-worn sensors were pro- Table 1: Activity grouping posed. Although the performance gets better with adding Group Activity additional body sensors, as Attal and al. [1] proved in 2015 chores by reviewing the research done by then, we decided to focus eating our research on wrist-worn sensors due to before mentioned Daily activities handwork accessibility and popularity. washing nordic Vens and al. [6] defined hierarchical multi-label classification Exercise running (HMC) as a variant of classification, that differs from normal walking classification in two ways: (1) a single example may belong lying to multiple classes simultaneously; and (2) the classes are Static sitting organized in a hierarchy: an example that belongs to some standing class automatically belongs to all its superclasses, the so- called hierarchy constraint. 3. METHODS Although hierarchical approach might seem quite intuitive In this paper we are comparing three different approaches for AR, as certain activities are pretty obvious grouped to- for activity recognition. First we addressed flat classifica- 1https://dtai.cs.kuleuven.be/clus/index.html 2https://orange.biolab.si/ 53 tion, which is commonly used in previous research. Next, then run feature selection for each of the subsets sep- we implemented two multi-model hierarchical algorithms, arately and built three different models - on for each based on approach proposed by Paes et al.[11]. We use the group of activities. Features were different for each term multi-model as different models were used for different group. levels of hierarchy. Finally we used Clus software, which 2. SVW After the first level, the classified group has has algorithms for hierarchical multi-classification (HMC) been added to instances as an additional feature. Fea- already implemented and is mostly used in the field of func- ture selection has been done again - this time for the tional genomics and text classification as shown by Vens et whole level, and one model has been built to distin- al.[6]. guish between activities. The users were wearing a wearable device (wristband or Same authors have explored feature selection for both ap- smartwatch) on their non-dominant hand. For the purpose proaches in [11], where they have shown that the best results of this paper we only considered triaxial accelerometer data, are obtained when using the lazy approach - this approach however for further research other measurements are avail- executes feature selection at the classification time of each able as well (heart rate, galvanic skin response..) instance. We have decided to use the eager approach, where feature selection is done prior to classification. From raw measurements we crated instances using 2 second sliding window and computed set of various features from accelerometer data that were shown to perform well in simi- lar setting (mean, average, skewness, kurtosis, peak counts) [4]. Additionally we computed the Euler angles pitch and roll and calculated some extra features from them as well - for instance, pitch and roll manipulation, amount of roll motion, regularity of roll motion... Altogether we computed 105 features. Afterwards feature selection was applied and the best of them were used to build models. 3.1 Feature Selection Feature selection was used only in the cases of flat classifi- cation and MM-HMC.For feature selection, we first ranked the features by gain ratio. After that, we used a wrapper approach. We started with an empty feature set and added Figure 1: PPTD - local per parent node approach features in the order of their rank. After each feature was added, we evaluated its contribution by building random- forest classifiers and internally cross-validating them on the training set. The feature was kept only and only if it in- creased the overall average accuracy. The ranking by gain ratio and the random forest algorithm were implemented in the Weka machine-learning suite and run with default pa- rameter values. 3.2 Flat Classification The most common approach for AR is the so-called flat clas- sification. All classes are considered equal, hierarchy is not taken into account. Algorithms were implemented in java, using Weka3 library. Figure 2: SVW - local per level approach 3.3 Multi-Model Hierarchical Classification Figure 3: Hierarchical classifiers We implemented two different approaches for hierarchical classification. The first one, traditional hierarchical strategy Per Parent Top Down (PPTD - Figure 1), based on ”local 3.4 CLUS-Classification per parent node” model, and the second one, named Sum of CLUS is a decision tree and rule learning system that works weighted Votes (SWV - Figure 2), ”local per level” model, in the predictive clustering framework. One of its important proposed by Paes et al. in [10]. On the upper level we functionalists is the CLUS-HMC algorithm for hierarchical built a model to distinguish between three groups - daily multi-label classification. The software has been shown to activities, exercise and static. This was done the same for work very well in the field of functional genomics [6], so the both approaches. From here on, the approaches differ. idea to use it in hierarchical classification for activity recog- nition seems reasonable. Clus-HMC algorithm is a variant 1. PPTD For this approach, we split instances into three of standard greedy top-down algorithm for decision tree in- different subsets regarding to the classified group. We duction. To achieve the task of predicting a set of classes instead of a single class, additional changes to the learning 3https://www.cs.waikato.ac.nz/ml/weka/ procedure are needed, as shown in [12]. 54 of classes, score is called average AUPRC, and is denoted as AU P RC. If the weigths are set to wi = vi/ P v j j where vi is the frequency of class ci in data, we call this weighted AUPRC and denote it as AU P RCw. We have compared the performance of the proposed methods by comparing the pre- cision, recall, F-score and AU P RC score by activity. Valida- tion has been done using ”leave-one-person-out” approach. We computed all of the mentioned measures for each per- son and averaged them to get the performance accuracy by method. Methods that we compared are flat classification, multi-model classification using SVW (local per level) ap- proach and CLUS-classification using same approach. We decided to leave out the comparison of PPTD algorithm due to lack of data. Classes for static gorup were poorly represented from the beginning and after classification on the first level some were left with only few examples. To avoid loosing data we propose additional approach, which is Figure 4: CLUS settings file example. roughly explained in the conclusion. Using the same dataset Cvetkovic and al. [4] have reported In our experiment we worked with random forest (to make it on 70% accuracy for five different classes (sports, eating, comparable with other two approaches), and we allowed the chores, handwork, washing). We expected high confusion in decision tree to go up to depth 20. We have shown experi- group of daily activities (handwork, chores, eating, washing) mentally that performance increases sharply up to decided and some confusion between other groups and within them depth, while afterwards the contribution has become negli- as hand movements can be very similar in this group. Table gible. The error we used for optimization was the average 2 and Table 3 show the results of the experiments. We could AUPRC (area under the precision-recall curve). We have not compare the AU P RC of flat classification when classify- tested the performance by changing the threshold determin- ing groups, as we only get the values for classified activities ing when the probability output by the model is considered on lower level. However, we could compare flat classification to predict a class. All of the above mentioned parameters to other two approaches using other measures. As shown in are set in the settings file as seen in Figure 4. Table 2 MM-HMC performs the best for AR on the upper level, but not much better than flat. On the lower level the 4. EXPERIMENTAL SETUP AND RESULTS results from flat classification and from MM-HMC were quite similar, with one approach performing better in some cases In our case the hierarchy is very simple, reduced to two lev- and worse in others. From the fact that direct classification els. For HMC problems Clus returns several error values. on the upper level (MM-HMC) is not much better from the To get fair results for each person included in the dataset, indirect, it is safe to conclude that this is the reason, that leave-one-person-out approach has been used, as mentioned for similar results between the mentioned two approaches on before. For evaluation of the results we decided to choose the lower level. The achieved average accuracy for flat clas- standard measurements - precision, recall and F-score. How- sification has been 70.5% and very similar for MM-HMC. ever, when it comes to the evaluation of highly skewed class Each works better in some cases. Results using CLUS are distributions, similar as with our dataset where for instance not the most promising. However, there are many possi- daily activities have a much higher frequency than rest, ble combinations of settings available and the performance precision-recall curves are the most suitable evaluation tool could be improved by choosing different set of parameters [7], so this was also added. Vens el al. [6] have addressed and their values. We tried many possible combinations and the problem of most eligible evaluation tools for hierarchical the presented results are the best so far. classification. From the proposed evaluation tools we used the area under the precision-recall curve. Table 2: Results upper level (group) To evaluate predictive models independently from the thresh- Flat MM-HMC CLUS old, two types of evaluation are suitable: ROC analysis and Fscore 82.05% 83.71% 74.36% analysis of precision-recall curves (PRC). ROC analysis is better known in machine learning, however for hierarchical Precision 82.03% 83.73% 76.22% multi-label classification PR is more suitable. [?] PR curve Recall 82.12% 84.05% 73.10% plots the precision of a model as a function of its recall, and AU P RC 89.61% 81.09% although it helps understanding the predictions, single value is more appropriate for comparing quality of different mod- els. A score often used to represent this is the so-called ”area Table 3: Results lower level (activity) under the PR curve” (AUPRC). The closer the AUPRC is Flat MM-HMC CLUS to 1.0, the better the model. Fscore 65.14% 66.79% 52.23% Precision 68.29% 65.92% 58.31% AU P RCw = P w i i · AU P RCi Recall 65.48% 67.69% 51.08% AU P RC 68.63% 66.67% 54.76% If all the weights are set to wi = 1/|C|, where C is the set 55 5. CONCLUSION 6. REFERENCES In this work we compared three approaches to activity recog- [1] Attal, F., Mohammed, S., Dedabrishvili, M., nition. Our results show that for the purpose of activity Chamroukhi. F., Oukhellou, L., Amirat, Y. Physical recognition with 2 levels of activity (group and activity), flat Human Activity Recognition Using Wearable Sensors. classification performs as well as both types of hierarchical In: Sensors 15(12). Basel (2015). 31314-18 classification - or even better. In some other uses of HMC, [2] Siirtola, P., Lurinen, P., Haapalainen, E., Röning, J., for instance functional genomics, fast performance and cor- Kinnunen, H. Clustering-based activity classification rect classification of higher levels is of greater importance with a wrist-worn accelerometer using basic features. than correct classification of lower levels. Unfortunately in In: 2009 IEEE Symposium on Computational the case of activity recognition fast performance was the Intalligence and Data Mining. CIMD 2009 - only upside. Proceedings. (2009) 95-100 [3] Chernbumroong, S., Atkins, A.S. Activity classification There are some possible improvements for future work. The using a single wrist-worn accelerometer. In: 2011 5th dataset we were working on, was not really extensive. There International Conference on Software, Knowledge were many activities involved and not many instances of Information, Industrial manegment and Applications each. This could be solved with joining more similar datasets. (SKIMA) Proceedings. (2011) 1-6 [4] Cvetkovic B., Drobnic V., Lustrek M.: Recognizing Some of the HMC-related papers mentioned different clas- Hand-Specific Activities with a Smartwatch Placed on sifiers for classification. We used random forest, as it has Dominant or Non-dominant Wrist. In: Information performed the best in our previous research where we were Society. Ljubljana (2017) only using flat classification, however some other classifiers [5] Nweke, H.F., Teh, Y.M. Al-gardi, M.A., Alo, U.R. may perform better on the hierarchical problem. Better ac- Deep learning algorithms for human activity curacy could as well be achieved by adding measurements recognition using mobile and wearable sensor networks: from some other sensors (heart rate sensor), as maybe there State of the art and research challanges. In: Expert are some more distinctive differences between subsets of the Systems With Applications. 105 (2018) 233-261 proposed hierarchy. [6] Vens, C., Stryuf, J., Schietgat, L., Dzeroski, S., Blockeel H. Decision trees for hierarchical multi-label A possibility to improve the performance of MM-HMC is to classsification. In: Machine Learning, 73(2): 185-214 add additional activities to each of the groups. For instance, [7] Davis, J., Goadrich, M. The relationship between we add exercise and static as two new activities in group of precsion-recall and ROC curves. In: Proceedings of the daily activities. Similar would be done for other two groups 23rd Internation Conference on Machine Learning. of activities. After building models for the lower level, we (2006) 233-240 would then build additional models for all ”new activities” [8] Khan, A.M., Lee, Y.K., Lee, S.Y., Kim, T.S. A Triaxial classified to wrong group. We will try this approach in our Accelerometer-Based Physical-Activity recognition via future work. Augmented-Signal Features and a Hierarchical Recognizer. In: IEEE Transactions on Information Technology in Biomedicine. 14(5). (2010) 1166-72 [9] Zheng, Y. Human Activity Recognition Based on the Hierarchical Feature Selection and Classification Framework. In: Journal of Electrical and Computer engineering. (2015) doi:10.1155/2015/140820 [10] Paes, B.C., Plastino, A., Freitas, A.A. Improving Local Per Level Hierarchical Classification. In: Journal of Information and Data Manegement 3 (3): (2012) 394-409 [11] Paes, B.C., Plastino, A., freitas, A.A. Exploring Attribute Selection in Hierarchical Classification. In: Journal of information and Data manegment- Vol. 5(1). (2014) 124-133 [12] Blockeel, H., Bruynooghe, M., Džeroski, S., Ramon, J., Struyf; J.: Hierarchical mulit-classification. In: Proceedings of the ACM SIGKDD 2002 Workshop on Mulit-Relational Data Mining (MRDM 2002). (2002) 21-35 [13] Cesa-Bianchi, N., Gentile, C., Zaniboni, L. Incremental algorithms for hierarchical classification. Journal of Machine Learning. (2006) 31-54 [14] Clus Homepage (last accessed 26 Jun 2018) https://dtai.cs.kuleuven.be/clus/index.html. 56 Aiding the Task of Process-Based Modeling with ProBMoTViz Gjorgi Peev Nikola Simidjievski Sašo Džeroski Jožef Stefan Institute & Jožef Stefan Institute Jožef Stefan Institute & Jožef Stefan International Jamova cesta 39 Jožef Stefan International Postgraduate School Ljubljana, Slovenia Postgraduate School Jamova cesta 39 nikola.simidjievski@ijs.si Jamova cesta 39 Ljubljana, Slovenia Ljubljana, Slovenia gjorgi.peev@ijs.si saso.dzeroski@ijs.si ABSTRACT In this work, we set out to overcome the usability limita- Process-based modeling (PBM) is an equation discovery ap- tions of ProBMoT, i.e., to expand its user scope by develop- proach for automated modeling of dynamical systems, which ing and implementing an extension for it. In particular, we takes at input substantial expert knowledge and measured propose ProBMoTViz, a software platform which includes a data of the observed system. The resulting process-based Graphical User Interface (GUI) for ProBMoT and a PBM models offer both high-level representation (in terms of build- Visualizer for process-based models. On one hand, the GUI ing blocks / model components, i.e., entities and processes) supports the basic operations of (automated) modeling dy- and low-level representation (a set of ordinary differential namical systems in terms of providing appropriate input for equations). ProBMoT, a software platform for modeling, pa- the modeling and examining the outputs thereof. On the rameter estimation, and simulation of process-based models, other hand, the PBM Visualizer illiterates the (currently is the latest implementation of the process-based modeling textual) output models with a higher-level visual represen- approach. While ProBMoT has been successfully applied to tation, that better communicates with the domain experts. the task of modeling dynamical systems, compared to other modeling and simulation software, it is substantially behind in terms of user-friendliness. The goal of the present work 2. PROBLEM DEFINITION is to overcome this limitation of ProBMoT. We design and ProBMoT addresses the task of automated modeling in terms implement an extension of ProBMoT, named ProBMoTViz, of automated search of the appropriate model structure and which is a platform consisting of a GUI (Graphical User estimating its parameter values. The input to the tool in- Interface) for ProBMoT and a PBM Visualizer for process- cludes several input files: (1) a library of background knowl- based models. We evaluate the versatility of ProBMoTViz edge (.pbl file specifying the domain); (2) a conceptual model on example case studies. (.pbm file specifying the problem); (3) a data file; and (4) a task specification .xml file, specifying the particular task.1 To this end, running ProBMoT require cumbersome and 1. INTRODUCTION time-demanding procedures of preparing an appropriate in- ProBMoT [4, 10] is the latest implementation of the process- put. For instance, the .xml task specification file defines all based modeling approach [3, 6] for automated modeling of the hyper-parameters needed for ProBMoT to run properly, dynamic systems. Given a background knowledge, mod- such as the paths of the input files, definition and mapping eling constraints and measured data at input, ProBMoT of variables and outputs to data sets, settings of the pa- constructs completely defined process-based models, repre- rameter fitter and the simulator, etc. All these components sented with entities and processes. are represented with different XML tags. In response, the main contribution of ProBMoTViz is facilitating the task of ProBMoT has been successfully applied to a variety of mod- process-based modeling with ProBMoT. In particular, the eling tasks in a number of real-world domains, such as aquatic ProBMoT workflow will be encapsulated in a shell, where ecosystems [4]; population dynamics [5]; biological systems the .xml file is not written manually, but its representative [11]; oscillatory systems [8]; as well as predicting future be- settings are tuned interactively. havior of the system at hand [9]. Unlike other modeling and simulation tools [7], ProBMoT is a domain-free tool and On the other hand, the constructed process-based models can be applied to any modeling task that involves model can be complex and difficult to understand2. The textual structure identification and/or parameter estimation. How- representation of process-based models can be improved, ever, it still straggles behind these tools in terms of graphi- thus further enhancing their interpretability and commu- cal/visual representation of the constructed models, compre- nicability with domain experts. ProBMoTViz implements hensibility of the output for a broader user-base as well as state-of-the-art visualization techniques able to overcome user-friendliness when it comes to preparing and running a PBM task. User feedback indicates that a GUI and a visual 1Tables 3-5 in the Appendix, present these inputs for a par- representation of process-based models can overcome these ticular modeling task. obstacles. 2Table 6 in the Appendix 57 Figure 1. ProBMoTViz, the proposed extension of ProBMoT, consists of a GUI for ProBMoT and a PBM Visualizer. the potential comprehensibility obstacles and usability lim- one sub-type: hierarchical entities (representing the hierar- itations of the current textual representations. chy that the entity comes from), while the processes have two sub-types: hierarchical processes (representing the hier- 3. PROBMOTVIZ archy that the process comes from), and children processes (representing the nested processes in a process), as shown in ProBMoTViz consists of two main components: the ProB- Figure 2. MoT GUI and the PBM Visualizer (Figure 1). The former guides the user through the process of creating and defining a new PBM task step-by-step. It is a desktop-based appli- cation, developed in JavaFX [12], that facilitates the task of PBM, allowing for the settings, mappings, and all of the other customizable properties which must be specified in the .xml task settings file, to be now adjusted interactively in a workflow. The platform is divided into nine main scenes through which the user must progress in order to define the modeling task, monitor its progress, as well as to analyze the process-based models obtained at output. In particu- lar, the scene sequence is as follows: (1) Library - the scene where the library file must be chosen and all the library components (template entities and processes) are shown; (2) Figure 2. Schematics of the visual representation of Model - the scene where the (in)complete model file must model components. be chosen and all the model components (instance entities and processes) are shown; (3) Data - the scene where the To better illustrate how ProBMoTViz works, we present the data files must be chosen, with the opportunity to inspect/ important details of preparing a task for modeling a two cas- visualize the data; (4) Inputs - the scene where the input caded water tanks system (Equation 1). The system consists mappings must be specified, i.e., the mapping of the time of two cascaded water tanks with free outlets, placed one dimension and the exogenous variables to a column in the above the other, fed by a pump. In the governing equations data set; (5) Outputs - the scene where the outputs and for this system, the water levels of the tanks are denoted their mappings to a specific column in the data set must with h1 and h2. A1, A2, a1 and a2 denote the areas of the be specified; (6) Overview scene; (7) Settings - the scene tanks and their effluent areas, while the applied voltage-to- where all the task settings are specified, i.e., the evaluation, flow conversion constant is denoted with k. The task is to simulator, fitter, and other settings; (8) Run scene - where model the water level in the lower tank. The data is ob- the particular task can be exported in an .xml format for tained by laboratory measurements [13] and it consists of later (re)use and (9) Results - the scene where the resulting 2500 samples (1500 train and 1000 test set) of the input process-based models can be inspected and analyzed. voltage applied to the pump and the water levels in both tanks. √ The latter component, the PBM Visualizer, is a web appli- √ ( dh1 = − a1 2g h u(t) dt A 1 + k A cation produced using the D3.js (Data Driven Documents 1 √ 1 √ √ √ (1) dh2 = − a2 2g h 2g h JavaScript library) [2]. It offers an interactive visual repre- dt A 2 + a1 1 2 A2 sentation of the complex hierarchies of process-based models depicting both the high-level structure of the models as well First, after loading the PBM library into ProBMoTViz (Fig- as the interactions between its components. In particular, ure 3), one can explore the encoded domain knowledge (for process-based models are depicted as a Sankey diagram [1], this example, for modeling fluid dynamics) in the traditional where the nodes denote the components of the process-based PBM formalism. In particular, the entity Tank encodes a models. We define two main types of components that cor- variable that represents its water height level, and constants respond to: entities and processes. Moreover, entities have denoting the inflow and outflow areas. Analogously, the 58 Figure 3. The water tanks library of background knowledge. Figure 5. The settings parameters for modeling the two water tanks system shown. other entity Pump incorporates a variable denoting the input voltage in the system. Moreover, the library also encodes the different (plausible) interactions between the entities in terms of water transmissions between: two tanks, a tank and the environment as well as a tank and a pump. Note that, these interactions can also have different behaviour, therefore the library encodes different modeling alternatives for each of them in terms of a squared-root, linear or expo- nential dynamics. In the next step, after loading the incomplete model (Figure 4), one can explore the specific components of the particular problem, i.e. two tanks and one pump. The task is defined as follows: in a two-tank system with an electric pump, identify Figure 6. The resulting models from modeling the the underlying dynamics of the three different interactions two water tanks system. that describes the behaviour of the lower thank. by simulating the model m on the test set) of the system variable y at time point t. The mean value of y in the test set is denoted with y. This metric is relative to the standard deviation of the system variable in the test data, thus allow- ing us to compare the errors of models for different system variables with measured values on different scales. s Pn (yt − ˆ yt)2 RRM SE(m) = t=0 (2) Pn (y t=0 t − y)2 Figure 4. The water tanks conceptual model. 4.1 Two cascaded water tanks system For the previously defined water tanks system, given an in- With loading the data, followed by specifying the mappings, put voltage, the output of interest in our model is the water defining the outputs and specifying the settings, our partic- height level of the lower tank. The particular process-based ular task is completely defined and ready for execution, as modeling task yields 9 feasible models, as shown in Table shown in Figure 5. 1. The best obtained model (Figure 7) is contained of the processes Inflow, FlowSQRT and OutflowSQRT, which cor- Finnaly, ProBMoTViz, lists the constructed models (Fig- responds to the original system. ure 6), and offers additional tools (error-plots and visual representation of the constructed process-based models) for Table 1. The results of modeling the two cascaded further analyses. water tanks system. 4. CASE STUDIES Model Train RRMSE Test RRMSE As a case study, we tackle the tasks of modeling two bench- ModelSqrtSqrt 0.2208 0.2673 ModelLinSqrt 0.2285 1.1837 mark nonlinear dynamical systems: (1) Two cascaded water ModelExpSqrt 0.2448 0.3101 tanks system and (2) The SilverBox – an oscillatory sys- ModelSqrtLin 0.2916 0.3200 tem using ProBMoTViz. For evaluating the performance ModelLinLin 0.3138 0.3323 of our models, we measure the relative root mean squared ModelExpLin 0.3576 0.4249 error (RRMSE) of each model’s output, shown in Equation ModelSqrtExp 0.7233 0.8507 2. The number of samples in the test set is denoted with n; ModelLinExp 0.7312 0.8585 ModelExpExp 0.7457 0.8835 yt is the measured and ˆ yt is the predicted value (obtained 59 Figure 8. The SilverBox visual process-based model. Figure 7. The water tanks visual process-based model. hensible and user-friendly preparation and analyses of mod- 4.2 SilverBox Oscillator System eling tasks. This enables the user to have better overview The second case study, addresses the task of reconstructing and control over the input parameters necessary when run- a nonlinear mechanical oscillating system, referred as the ning ProBMoT, therefore saving time and computational SilverBox system - an electronic implementation of the Duff- resources when performing large amount of- and/or deli- ing oscillator. The system’s dynamics (Equation 3) relates cate experiments. The PBM Visualiser, on the other hand, to the displacement y(t) (the output) to the input voltage aids in visualizing the complex (hierarchical) structure of u(t). The parameter m is a moving mass, d is viscous damp- process-based models, in turn allowing for better under- ing, and k(y) is a nonlinear progressive spring described by standing and comprehensibility. a static but position-dependent stiffness. The data is gener- ated by an almost idealized representation of the oscillator [13]. It consists of 130000 samples (90000 train 40000 test 6. REFERENCES [1] N. Atkinson. Bi-directional hierarchical sankey diagram set) of the input voltage and the output displacement of the [online: https://github.com/neilos/bihisankey], 2015. oscillator. [2] M. Bostock, V. Ogievetsky, and J. Heer. D3 data-driven ( documents. IEEE transactions on visualization and m d2y(t) + d dy(t) + k(y(t))y(t) = u(t) dt2 dt (3) computer graphics, 17(12):2301–2309, 2011. k(y(t)) = a + by2(t) [3] W. Bridewell, P. Langley, L. Todorovski, and S. Džeroski. Inductive process modeling. Machine learning, 71(1):1–32, 2008. The PBM library incorporates the domain knowledge where [4] D. Čerepnalkoski. Process-based Models of Dynamical the entity Oscillator encodes the input voltage, output dis- Systems: Representation and Induction: Doctoral placement and its mass. Moreover, behaviors of different Dissertation. PhD thesis, D. Čerepnalkoski, 2013. oscillators: (1) Duffing, (2) Simple, (3) Harmonic and (4) [5] S. Džeroski and L. Todorovski. Encoding and using domain Universal oscillator, all of which differently affect the out- knowledge on population dynamics for equation discovery. put displacement of the oscillator are also encoded in the In Logical and computational aspects of model-based library. The incomplete model specifies the particular prob- reasoning, pages 227–247. Springer, 2002. [6] P. Langley, J. N. Sanchez, L. Todorovski, and S. Džeroski. lem of one oscillator with unknown oscillatory behavior. Inducing process models from continuous data. 2002. [7] G. Peev, N. Simidjievski, and S. Džeroski. Modeling of The process-based modeling task yields 4 feasible process- dynamical systems : a survey of tools and a case study. In based models, as shown in Table 2. The best obtained model 20th International Multiconference Information Society - (Figure 8) contains the processes InitOscillator, Oscillator- IS 2017, volume A, pages 15–18, 2017. Input and DuffingOscillator, which corresponds to the orig- [8] G. Peev, N. Simidjievski, and S. Džeroski. Identification of inal system. a nonlinear dynamical benchmark system using process-based modeling. In 10th Jo˚ A¿ef Stefan IPS Table 2. The results of modeling the SilverBox sys- Students’ Conference, page 36, 2018. tem. [9] N. Simidjievski, L. Todorovski, and S. Džeroski. Predicting long-term population dynamics with bagging and boosting of process-based models. Expert Systems with Applications, Model Train RRMSE Test RRMSE 42(22):8484–8496, 2015. ModelDuffing 0.2353 0.2741 [10] J. Tanevski, N. Simidjievski, L. Todorovski, and ModelSimple ∞ ∞ S. Džeroski. Process-based modeling and design of ModelUniversal 0.8803 0.8796 dynamical systems. In ECML PKDD, pages 378–382. ModelHarmonic 0.9330 0.9327 Springer, 2017. [11] J. Tanevski, L. Todorovski, and S. Džeroski. Process-based design of dynamical biological systems. Scientific reports, 5. CONCLUSIONS 6:34107, 2016. In this work, we present a novel software tool, ProBMoTViz. [12] K. Topley. JavaFX Developer’s Guide. Pearson Education, 2010. It consists of two components: a GUI for ProBMoT and a [13] T. Wigren and J. Schoukens. Three free data sets for PBM Visualizer for process-based models. The GUI sup- development and benchmarking in nonlinear system ports the basic operations necessary for (automated) mod- identification. In 2013 European Control Conference eling of dynamical systems. Moreover, it allows for compre- (ECC), pages 2933–2938, July 2013. 60 Appendix: ProBMoT inputs Table 5. An example task specification file in XML format. Tables 3-5 present the necessary inputs for ProBMoT for the tasks of automated modeling of a water tank dynamic C:/Users/WaterTanksLibrary.pbl system. Table 6 presents a resulting process-based model in C:/Users/WaterTanksIncomplete.pbm the standard PBM formalism. C:/Users/Data1.csv C:/Users/Data2.csv Table 3. Library of domain knowledge library WaterTanksLibrary; //ENTITIES template entity Tank { vars: height {aggregation:sum, range:<0,500>}; consts: area {range: <1.0E-3,30>}, ca {range: <1.0E-3,30>}; } template entity Pump { vars: v{aggregation:sum, range:<-15,15>}; consts: k {range:<0.2,1E6>}; } //PROCESSES WaterTanksIncomplete.tank2.height template process Flow (t1 : Tank, t2 : Tank) { consts: G {range: <0,50>}, C:/Users/ template process FlowSQRT : Flow { exhaustive search equations: td(t1.height) = - (G * pow(t1.height,1/2) * t1.ca)/t1.area, td(t2.height) = (G * pow(t1.height,1/2) * t1.ca)/t2.area; } false template process FlowLINEAR: Flow { true equations: td(t1.height) = - (G * t1.height * t1.ca)/t1.area, td(t2.height) = (G * t1.height * t1.ca)/t2.area; } 0.001 template process FlowEXP : Flow { 0.001 equations: 1000 td(t1.height) = - (G * exp(t1.height) * t1.ca)/t1.area, td(t2.height) = (G * exp(t1.height) * t1.ca)/t2.area; } template process Outflow (t:Tank) { 0 consts: 10000 G {range: <0,50>}; } 200 template process OutflowSQRT: Outflow { rand/1/bin equations: 0.9 td(t.height) = - G * pow(t.height,1/2) * t.ca/t.area; } 0.9 template process OutflowLINEAR :Outflow { 0 equations: false td(t.height) = - G * t.height * t.ca/t.area; } template process OutflowEXP :Outflow { RelativeRMSEObjectiveFunctionMultiDataset equations: td(t.height) = - G * exp(t.height) * t.ca/t.area; } template process Inflow (p: pump, t: Tank) { equations: 1 td(t.height) = p.k * p.v/t.area; } 2 Table 4. Incomplete model of the two cascaded water tanks system. Table 6. A process-based model of a two cascaded water incomplete model WaterTanksIncomplete : WaterTanksLibrary; tanks system. //Entities entity tank1 : Tank { model WaterTanksModel : WaterTanksLibrary; vars: entity tank1 : Tank { height { role: endogenous; initial: 0.20508;}; vars: consts: height { role: endogenous; initial: 0.20508;}; area = null, consts: ca = null; area = 19.944, } ca = 1.087; entity tank2 : Tank { } vars: entity tank2 : Tank { height { role: endogenous; initial: 0.38086;}; vars: consts: height { role: endogenous; initial: 0.38086;}; area = null, consts: ca = null; area = 23.051, } ca = 3.066; entity pump1 : Pump { } vars: entity pump1 : Pump { v { role: exogenous;}; vars: consts: v { role: exogenous;}; k = null; consts: } k = 20.305; } //Processes process flow (tank1, tank2) : Flow { //Processes consts: process flow (tank1, tank2) : FlowSQRT{ G = 4.429; } consts: process outflow1 (tank2) : Outflow { G = 4.429; } consts: process outflow1 (tank2) : OutflowSQRT { G = 4.429; } consts: process inflow1 (pump1, tank1): Inflow { G = 4.429; } } process inflow1 (pump1, tank1): Inflow { } 61 Evaluation and Prospects of Semi-Automatic Video Distance Measurement in Ski Jumping Matjaž Kukar University of Ljubljana Faculty of Computer and Information Science Večna pot 113 SI-1000 Ljubljana, Slovenia matjaz.kukar@fri.uni-lj.si ABSTRACT The administrative support for small competitions is mostly Great competitive results of Slovenian ski jumpers in world covered by the information system “Spletni Smuško1”, while cup and continental competitions have sparkled a lot of in- the IT support is virtually nonexistent for style and dis- terest for active participation in this attractive sport. In tance umpires. Only at the highest competitive levels (world junior levels, national competitions with considerably more cup, continental cups), professional staff — delegates, style than 100 jumpers are becoming the norm. However, due to and distance umpires — are supported by expensive com- lack of technologic aids for distance measurement, such com- mercial solutions [6, 2]. In our project, we focus primarily petitions can last over half a day. Only at the top-level com- on supporting distance umpires who have a demanding, ex- petitions (world cup, continental cup) expensive and logisti- posed role, and their mistakes often lead to bad will among cally demanding commercial video distance measuring tools coaches, competitors’ parents and spectators, as well as in are used for this purpose. In a previous project we developed public opinion. a video distance measuring system from low-cost commer- cial components, which was not suitable for real-time usage The aim our previous work [1] was to develop a system for due to technological limitations, but worked great for offline supporting video distance measuring on smaller hills, with measurement. We analyze the results of offline measure- accessible hardware requirements (a single video system and ments for several competitions and show that measurement a laptop). In this paper we evaluate the system results from errors are often unacceptably high. This serves as a moti- ski jumping competitions in younger categories on small hills vation for an ongoing project, where video measurement is in regional competitions (Cockta Cup), and provide some performed in real time and supported by advanced computer directions for future development. vision and deep learning methods. 1.1 Ski Jump Distance Measurement Keywords In ski jumping the jump distance is defined as a distance between the edge of the jumping ramp and a point where ski jumping, video distance measurement, computer vision, both ski jumper’s legs have touched the ground with full machine learning, deep learning surface [4, article 432.1]. The middle point between both legs is used when the legs are apart (e.g., Telemark landing 1. INTRODUCTION style). There are however three exceptions [3]: In recent years, we have witnessed a boom in Slovenian 1. In one-legged landings (i.e. the second ski is longer in ski jumping, mainly as a consequence of excellent results the air than what is typical during the normal landing of Slovenian competitors. There is a marked increase in routine) the correct distance is measured where the interest at the primary level; the ski jumping clubs have first ski touches ground with full surface. reportedly doubled the number of younger, primary school 2. In a fall (where the landing does not result on the skis competitors (7-10 years). This has considerably increased as is normal), the correct distance is measured at the the burden on ski jumping coaches, as well as on organizers location where the ski jumper contacts first the landing and professional staff in competitions, that are carried out surface with a body part. even in the youngest categories (from 2018, up to 10 years 3. In arbitrarily delayed landings (i.e. the ski jumper is on only as “animations”). positioned extremely behind thus delaying the normal landing routine and the touch down of the ski tips to the landing surface) the correct distance is measured where both feet contact first the landing surface. Even on the smallest competition hills (HS ≤ 15 m) it is difficult to measure the exact flying distance by eyes only, since landing speeds exceed 10 m/s (36 km/h), and the angle 1http://smusko.adamsoft.si 62 Table 1: Basic statistic of official and video measure- ments. Official Video Abs. Abs. diff. distance distance diff. (centered) count 86 86 86 86 mean 22.52 21.99 0.62 0.33 st. dev. 2.11 2.23 0.37 0.35 min 17.00 15.50 0.00 0.00 max 26.00 25.50 1.50 1.50 jumper touching the landing slope with his/her hands. No Figure 1: Offline video distance measuring system. spectacular falls were included. A calibrated measuring grid is overlaid over video stream. Eight distance-measuring umpires can be 3. EVALUATION OF OFFICIAL DISTANCE seen standing along the landing slope. MEASUREMENT RESULTS For 86 jumps we compared the official results (measured by eyes only) and offline video measurements, performed by two between the landing slope and landing trajectories of ski professional ski jumping coaches. Figure 2 depicts a scatter jumpers is often very small [5]. plot of official measurements vs. video measurements. From the placement of measurement pairs (almost all are below Therefore, umpire tower is often built not far from the lower the diagonal) it is obvious that manually measured distances end of landing slope. It allows good view, but is utilized are bigger than video measured ones. This bias is a result chiefly by style-measuring umpires. During ski jumping com- of different positioning of umpires and video camera, re- petitions, distance-measuring umpires are stationed a few sulting in different parallax errors (the camera was mostly meters apart along the landing slope (Figure 1). Usually positioned slightly higher than umpires and more towards they are volunteers from the organizing ski jumping club, the outrun). In Figure 3 this bias can be clearly seen as and often have no training and very little experience with nonnegative differences in distances for all but five jumps. distance measuring. With speeds exceeding 10 m/s, the um- pires have less than 0.05 second (with resolution of 0.5 m) to decide on a particular distance. Also, as declared by ski jumpers, they are almost never able to determine their flying distance with a reasonable accuracy. As a consequence, ski jumpers and their coaches are challenged when evaluating the progress in terms of jump/flight distance. A reason- ably automated video distance measuring system therefore has the potential to become an important coaching aid in everyday practice. 2. OFFLINE VIDEO DISTANCE MEASUR- ING In a recent project cooperation with the Ski Association of Slovenia2 (SAS) we developed a system for offline video mea- surement [1]. The aim of the project was to develop a reason- ably priced system for video distance measurement based on commercially available components. It utilized a JVC GC- PX100 camcorder3 that allows recording of up to 600 frames per second (FPS). While the camcorder was great for offline Figure 2: Scatter plot of official vs. video measure- video measurement due to standalone video recording, it was ments. impossible to use it in an online setting due to it’s incapa- bility of live video streaming to the computer. Table 1 shows basic statistics of official and video measure- ments. The difference in means (0.53 m) indicates the need In the offline setting we recorded several competitions on to account for different biases for each method. For this small hills (HS up to 25 m). Two professional ski-jumping reason we compare the two distance measuring approaches coaches utilized specially developed software developed with- with their values centered around their means (Eq. 1). in the project (see [1] and Figure 1) to facilitate offline video − ¯ measurement. In total, more than 200 ski jumps were video center(m) = d(m) d(m) (1) j j measured. For 86 we identified the jumpers and obtained of- where ¯ d(m) is the mean value of all measured distance for a ficially measured distances, that were used for further eval- particular measuring method m, and d(m) is the measured uation. All jumps were either successful or with the ski j distance for the jump j (again for a particular measuring 2https://www.szs.si method m). This allows us to contain the bias within the 3https://eu.jvc.com/microsite/eu/gc-px100/index.html ¯ d(m) and focus only on the differences dif (Eq. 2). j 63 Figure 3: Differences between original official and Figure 4: Differences between centered official and video measurements. video measurements. Table 2: Frequencies and percentages of absolute supports low light recording. At the time of paper submis- differences between centered official and video mea- sion the project is still in progress, therefore we are reporting surements. only partial results. Figure 5 shows the video distance mea- Difference Count % suring system (camera and laptop) in action. (centered) 0.0 39 45 0.5 40 47 1.0 5 6 1.5 2 2 dif = d(manual) − ¯ d(manual) − d(video) − ¯ d(video) j j j | {z } | {z } centered manual distance centered video distance (2) According to the involved ski jumping coaches, our video Figure 5: An online video distance measuring sys- measurements are much more reliable than the official man- tem consisting of a network camera (left) and a lap- ual ones, and can be considered as correct. Figure 4 and top computer. Table 2 show a much clearer picture of official measurement errors. 45% of measurements are deemed to be exact (within One of the main drawbacks of our original system [1] was 0.5 m), 47% are off by ±0.5 m, while additional 8% errors the lack of online distance measuring. This is now effec- are in the range of 1-1.5 m. For distances around 15 m tively solved by utilizing the network camera. The video this means a whopping 10%! To put this in perspective, for processing pipeline consists of several steps: a world record jump (253.5 m) this would translate to 25 m! On small hills, 1 m is worth 4.5-6 points, and such er- • a frame is acquired from the camera (MJPEG or H.264 rors can easily influence podium places, especially in closely stream) fought competitions. With introduction if video distance • Gaussian blur is used to get rid of noise measuring system would therefore benefit both umpires (less • background is subtracted by using the MOG2 algo- demanding work) competitors and spectators (less distance rithm [7], and the image is converted to black (back- measuring errors). ground) and white (moving) pixels, based on the last five frames 4. BEYOND OFFLINE MEASURING • of all the moving contours, the largest is selected as In the ongoing ŠIPK project we are partnered with the Tech- the ski jumper, and the corresponding bounding box nix4 d.o.o company, the biggest provider of traffic surveil- is superimposed to the frame (Figure 6). lance network cameras in Slovenia. They kindly provided • once detected, the ski jumper is tracked until he/she various camera models, produced by Axis Communications. has left the camera view We settled for the high frame rate model Q16455, that con- According to [3] and [4, article 432.1], video distance mea- nects to the computer via 100 Mbit Ethernet connection, al- surement is performed in two steps: lows frame rates up to 120 FPS with full HD resolution, and 1. determining the correct landing frame 4https://www.technix.si 2. determining the correct landing point corresponding 5https://www.axis.com/products/axis-q1645 to the ski jumper’s foot positions. 64 Figure 7: A sequence of frames classified as “air” and “ground”. The green line marks the correct landing frame, and the red line the predicted landing frame. Frames 7, 10 and 14 are incorrectly classified. SAS for widespread testing. For use on larger hills, slight modification of software will be needed in order to allow for Figure 6: Trained operator determines the distance two, three or four network cameras. The system still needs by using the calibrated measuring grid in 5-8 s. further testing (especially the automated components) un- der artificial lighting conditions (night competitions). A heuristic approach based on the flight curve derivatives is 6. ACKNOWLEDGMENTS used to approximately detect the landing frame. It works Original software for video distance measurement was devel- with accuracy of approximately 1 m (on small hills). The oped within the PKP project “Video meritve dolžin smučar- human operator is still needed to determine the correct land- skih skokov” by T. Ciglarič, K. Gostiša, T. Kovač, D. Pe- ing frame, and determine the distance based on the super- ternel, M. Pograjc, N. Stoklas, B. Štempelj and G. Vodan imposed measuring grid (Figure 6). [1]. Advanced automatization is developed within an on- going ŠIPK project VIDEOMEN in the ski jumping center 4.1 Automatic Detection of Landing Point with Mengeš, Slovenia. Deep Learning 7. REFERENCES We are currently experimenting with two approaches to au- [1] T. Ciglarič, K. Gostiša, T. Kovač, D. Peternel, tomate and speed up video distance measuring. The first M. Pograjc, N. Stoklas, B. Štempelj, G. Vodan, approach is using a deep convolutional neural network with R. Rozman, and M. Kukar. Video meritve dolžin 10 hidden layers in order to automatically detect the correct smučarskih skokov. In ERK’2017, pages 337–340, 2017. landing frame. Its input is a framed ski jumper in resolution 150 × 150 color pixels. Each frame is classified either as “air” [2] ewoxx. Sports data service. or “ground”. Due to real time processing requirement (net- https://ewoxx.com/sports. Accessed: August 2018. work executes on CPU only) the current topology it is rela- [3] FIS Ski Jumping Committee Sub-Committee for tively shallow. It consists of two 2D convolutional (C), two Officials, Rules and Control. Guidelines to Video pooling (P), four dropout (D), one flattening (F), and three Distance Measurement of Ski Jumping 2011. dense layers (De) as follows: C-P-D-C-P-D-F-De-D-De-D- http://www.fis-ski.com/mm/Document/ De. A sequence of frames can be classified as shown in Fig- documentlibrary/Skijumping/03/20/28/ ure 7. The sequence always starts with “air” and ends with Guidelines-VDM-2011-eng-deutsch Neutral.pdf. “ground”. When at least two subsequent “ground” frames Accessed: August 2018. are detected, the first is selected as the landing frame. This [4] FIS Ski Jumping Committee Sub-Committee for approach currently achieves 96% classification accuracy for Officials, Rules and Control. The International Ski determining the type of frame. However, as the errors always Competition Rules (ICR). Book III: Ski Jumping. occur near the correct landing frame, human intervention is June 2018 edition. still necessary. The second approach utilizes classic com- http://www.fis-ski.com/mm/Document/ puter vision image segmentation techniques to acquire posi- documentlibrary/Skijumping/03/19/96/ tions of ski jumper’s skis and legs in order to determine the ICRSkiJumping2018 clean English.pdf. correct landing point within the frame, and therefore the dis- Accessed: August 2018. tance based on the measuring grid (currently with accuracy [5] N. Sato, T. Takayama, and Y. Murata. Early of 0.5-1 m). Regarding the processing speed, for small hills evaluation of automatic flying distance measurement on 30 FPS are sufficient to achieve 0.5 m accuracy, however the ski jumper’s motion monitoring system. In Proc. 27th process works well even for 100 FPS video stream (tested IEEE Int. Conf. on Advanced Information Networking in laboratory conditions). For deeper neural networks, a and Applications, pages 838–845, 2013. gaming laptop with a discrete GPU will be required. [6] Swiss timing. Video distance measurement. https://www.swisstiming.com/fileadmin/Resources/ 5. CONCLUSIONS Data/Datasheets/DOCM SJ VDMS 1215 EN.pdf. Accessed: July 2018. Our evaluation has shown that there is great need for im- [7] Z. Zivkovic and F. van der Heijden. Efficient adaptive provement in ski jumping distance measuring, especially for density estimation per image pixel for the task of small hills. In order to achieve objective results and re- background subtraction. Pattern Recognition Letters, duce errors, video distance measurement is highly advisable. 27(7):773–780, 2006. There is considerable interest from ski jumping clubs and 65 Opis zmagovalne rešitve na mednarodnem tekmovanju o napovedovanju izida točk v tenisu Miha Mlakar Scott Sobel Jozef Stefan Institute Oliver Wyman 1000-SI Ljubljana, Slovenia Columbia, South Carolina, USA scott.sobel@oliverwyman.com miha.mlakar@ijs.si POVZETEK primarne funkcije lahko s tem sistemom zelo natančno sledimo Januarja 2018 je Avstralska teniška zveza v letu in hitrosti žogice, ter tako dobimo veliko podatkov, ki so bili sodelovanju s Tennis Australia’s Game Insight Group organizirala tekmovanje z nato na voljo za gradnjo modelov. naslovom From AO to AI: Predicting How Points End in Tennis. Po začetni analizi podatkov smo iz obstoječega nabora zgradili Cilj tekmovanja je bilo narediti model, ki bi na podlagi podatkov veliko dodatnih spremenljivk, ki smo jih nato uporabili za o udarcih in letu žogice, pridobljenih iz kamer, čim bolje modeliranje. Sledila je optimizacija parametrov algoritmov klasificiral konec točke v enega od treh razredov; nepotrebna strojnega učenja in predstavitev dobljenih rezultatov. Končna napaka, prisiljena napaka in neubranljiv ( winner) udarec. Izmed klasifikacijska točnost napovedi je bila 95% (95% za neubranljive 750 tekmovalcev, ki jih je sodelovalo na tekmovanju sva soavtorja udarce, 89% za prisiljene napake in 98% za neizsiljene napake). osvojila prvo mesto. V referatu je prikazan postopek razvoja zmagovalne rešitve, ki vključuje generiranje in izbiro 2. OPIS IN ANALIZA PODATKOV spremenljivk, izbiro ustreznega modela za strojno učenje, Analiza teniških udarcev se je izvajala na zaključnih udarcih točk optimizacijo njegovih hiperparametrov, ter predstavitev pridobljenih iz Australian Open 2017 turnirja. Točke v naši bazi rezultatov, dobljenih z zgrajenim modelom. so bile odigrane tako v moški, kot tudi v ženski konkurenci, pogoj pa je bil, da je bila dolžina točke več kot dva (servis in return) Ključne besede udarca. Tenis; Tekmovanje; Strojno učenje; Umetna inteligenca; XGB Podatki, ki so bili na voljo so bili razdeljeni ne učni in testni množici. Učni množici sta vsebovali podatke za 5000 točk 1. UVOD odigranih v moški konkurenci in za 5000 točk odigranih v ženski Tenis je glede na gledanost eden izmed najpopularnejših športov konkurenci. Testni množici sta vsebovali podatke za 2000 točk v na svetu. Da bi gledalcem in igralcem zagotovili zanimive moški in 1000 točk v ženski konkurenci. Seveda testni podatki statistike, se na vsaki tekmi meri veliko različnih parametrov, od niso imeli določenega tipa zaključnega udarca, saj je bila to uspešnosti prvega servisa, do števila dobljenih točk na naloga našega algoritma. nasprotnikov servis. Ena izmed pomembnejših statistik je tudi Podatki so vsebovali 27 spremenljivk in pa tip zadnjega udarca. število napak in neubranljivih udarcev, ki jih igralec izvede tekom Imena parametrov, njihovimi opisi in možne vrednosti so tekme. Te statistike trenutno beležijo ročno, strokovnjaki, ki ob predstavljeni v tabeli 1. igrišču gledajo tekmo in glede na svoje izkušnje označijo zaključek točke, kot nepotrebna napaka, prisiljena napaka ali pa Za kreiranje dobrih napovednih modelov so ključni dobri podatki kot neubranljiv udarec. in odlično poznavanje njihovih lastnosti. Zatosmo najprej naredili podrobno analizo podatkov. Ta analiza nam pomaga razumeti Vendar pa tak način zbiranja podatkov ni najbolj primeren zaradi podatke, odkrije povezave med spremenljivkami in razredi in možnosti napak, nekonsistentnih označb s strani različnih ljudi in identificira izvore šuma ter druge probleme, ki so vezani na pa tudi zaradi stroška ter dodatne logistike, saj na večjih turnirjih kvaliteto podatkov. hkrati poteka veliko število tekem. Če bi hoteli dobiti te vrste Začeli smo podatkov za vse tekme bi morali imeti zaposlenih veliko ljudi. z izračunom distribucije tipov zadnjih udarcev čez celotno učno množico 10.000 točk, ki je Zato je Avstralska teniška zveza v sodelovanju z Tennis sestavljena iz 3,352 Australia’s Game Insight Group organizirala tekmovanje z (33.5%) neubranljivih udarcev, 2,272 (22.7%) prisiljenih napak in naslovom From AO to AI: Predicting How Points End in Tennis, 4,376 (43.8%) nepotrebnih napak. Ta podatek nam pove, da so kjer so kot rešitev iskali algoritem, ki bi na osnovi podatkov razredi delno neuravnoteženi, da pa ta neuravnoteženost ni takšna, pridobljenih iz kamer objektivno in čim da bi pri strojnem učenju bilo potrebno uporabiti prav posebne bolj natančno lahko avtomatsko določil tip zadnjega udarca. metode namenjene reševanju problemov z neuravnoteženimi podatki. Tekmovanje je bilo organizirano na platformi CrowdANALYTIX Tako učna kot testna množica nista vsebovali manjkajočih (https://www.crowdanalytix.com/), ki omogoča organiziranje spletnih tekmovanj s področja umetne inteligence. Na tekmovanje podatkov, tako da nam ni bilo potrebno uporabiti algoritmov za nadomeščanje praznih vrednosti. Se pa je med podatki pokazalo, se je registriralo 750 tekmovalcev oziroma tekmovalnih ekip. da obstajajo netočne vrednosti. Primer je viden na sliki 1, kjer Analiza in razvoj modelov je temeljila na podatkih pridobljenih iz lahko vidimo, da so nekateri udarci, označeni, kot da je žoga sistema desetih visoko-resolucijskih kamer imenovanega padla izven dovoljenega območja a so še vedno označeni kot HawkEye (https://www.hawkeyeinnovations.com/), ki se v tenisu neubranljivi udarci in ne kot napaka. Razlogi za to so lahko trije: primarno uporablja kot pripomoček za detekcijo oziroma določanja ali je določena žoga padla v avt ali ne. Poleg te 66 Spremenljivka Opis Vrednosti outcome Razredna spremenljivka – tip zadnjega udarca W=zaključni udarec, FE=prisiljena napaka, UE=nepotrebna napaka speed Hitrost zadnjega udarca Zvezna vrednost (m/s) previous.speed Hitros predzadnjega udarca Zvezna vrednost (m/s) net.clearance Razdalja leta žoge nad mrežo za zadnji udarec Zvezna vrednost (cm). Negativna če je žogica letela nižje od mreže previous.net.clearance Razdalja leta žoge nad mrežo za predzadnji udarec Zvezna vrednost (cm). Lahko negativna če je žogica letela nižje od mreže distance.from.sideline Razdalja odboja žoge zadnjega udarca od najbližje črte Razdalja v metrih (pozitivna tudi če je odboj žogice izven igrišča) depth Razdalja odboja žoge od osnovne črte za zadnji udarec Razdalja v metrih (pozitivna tudi če je odboj žogice izven igrišča) player.distance.travelled Razdalja, ki jo je igralec pretekel pred zadnjim udarcem Euklidska razdalja v metrih player.impact.depth Oddaljenost igralca od mreže v trenutku, ko je udaril zadnji udarec Razdalja od mreže v metrih player.impact.distance. Oddaljenost igralca od sredine igrišča v trenutku, ko je udaril Razdalja od sredine igrišča v from.center zadnji udarec metrih player.depth Oddaljenost igralca od mreže v trenutku, ko je udaril predzadnji Razdalja od mreže v metrih udarec player.distance. Oddaljenost igralca od sredine igrišča v trenutku, ko je udaril Razdalja od sredine igrišča v from.center predzadnji udarec metrih oponent.depth Oddaljenost nasprotnega igralca od mreže v trenutku, ko je udaril Razdalja od mreže v metrih predzadnji udarec opponent.distance. Oddaljenost nasprotnega igralca od sredine igrišča v trenutku, ko Razdalja od sredine igrišča v from.center je udaril predzadnji udarec metrih previous.distance. Razdalja odboja žoge predzadnjega udarca od najbližje črte Razdalja v metrih (pozitivna tudi from.sideline če je odboj žogice izven igrišča) previous.depth Razdalja odboja žoge od osnovne črte za predzadnji udarec Razdalja v metrih (pozitivna tudi če je odboj žogice izven igrišča) previous.time.to.net Koliko časa je žoga pri predzadnjem udarcu potrebovala od udarca Zvezna vrednost v sekundah pa do mreže server.is.impact.player Indikator ali je zadnji udarec v točki odigral server TRUE = DA, FALSE = NE same.side Logični indikator, ki pove ali sta se igralca nahajala na isti strani TRUE = DA, FALSE = NE igrišča v času predzadnjega udarca outside.sideline Logični indikator ali je žogica padla znotraj stranskih črt TRUE = DA, FALSE = NE outside.baseline Logični indikator ali je žogica padla znotraj osnovne črte TRUE = DA, FALSE = NE train Indikator ali je točka del učne ali testne množice 1 = Training, 0 = Test serve Ali je bila točka odigrana na prvi ali na drugi servis 1= Prvi servis, 2= Drugi servis gender Indikator, ki pove ali je bila spol igralcev mens=moški, womens =ženske previous.hitpoint Kateri je bil predzadnji udarec v točki F = forhend, B = bekend, V = volej, U = neznan tip udarca hitpoint Kateri je bil zadnji udarec v točki F = forhend, B = bekend, V = volej, U = neznan tip udarca id 10-črkovni unikatni identifikator točke črkovni niz rally Število udarcev v točki 3, 4, 5, … Tabela 1: Opis spremenljivk 67 (i) človeška napaka pri označevanju, (ii) napačna prepoznava mesta odboja žogice z strojnim vidom iz kamer ali pa (iii) žoga je padla v avt, vendar pa sodniki na tekmi tega niso dosodili, zato je bil udarec dosojen kot dober in zato je v podatkih označen kot neubranljiv udarec. Ker je razlog za take primere neznan in ker je takih primerov manj kot 1% smo jih pustili v testni množici. Slika 3: Porazdelitev tipa zadnjega udarca glede na spol 3. KREIRANJE NOVIH SPREMENLJIVK Gradnja novih spremenljivk je ključnega pomena pri izboljšanju napovedne točnosti prediktivnih modelov. V našem primeru smo naredili sedem sklopov kreiranja novih spremenljivk. 1. S kombinacijo spremenljivk distance.from.sideline Slika 1: Porazdelitev tipa zadnjega udarca glede na to ali je in outside.sideline smo naredili novo zvezno žoga padla znotraj igrišča po dolžini ali ne. spremenljivko, ki je bila pozitivna, če je žoga Nadalje smo analizirali razdaljo leta žoge nam vrhom mreže padla v polju in negativna, če je bila žoga v avtu. (net.clearance). Distribucija podatkov je prikazana na sliki 2. Na enak način smo združili tudi spremenljivki Hitro lahko vidimo, kako imamo različne distribucije glede na depth in outside.baseline. končni tip udarca. Zanimivo je, da imamo tudi tu udarce, ki so klasificirani kot zaključni udarci in imajo negativno vrednost, kar 2. Za spremenljivke, ki so bile vezane na razdaljo in so naj bi pomenilo, da so pristali v mreži, kar je značilnost napak. imele samo pozitivne vrednosti smo izračunali dodatno Podobno kot pri pristanku žoge izven polja, je tudi tu lahko nove spremenljivke, ki so bile zvezne in so bile lahko napaka v računalniškem vidu ali pa se lahko zgodi, da se je žogica tudi negativne. Na primer za distance.from.center smo zaletela v vrh mreže in se odbila čez mrežo v polje. Ne glede na izračunali kot 8,23m (širina igrišča) minus razlog smo tudi tu se odločili, da teh točk ne bomo izločali ampak distance.from.sideline (popravljena razdalja vključno z jih bomo obdržali. negativnimi vrednostmi). Te spremenljivke same po sebi nimajo dodane prediktivne moči, vendar pa nam lahko pomagajo pri delitvi odločitvenega prostora, kar lahko v kombinaciji z drugimi spremenljivkami prinese izboljšanje. 3. Izračun novih spremenljivk glede na položaje igralcev in kote udarcev ter domensko znanje. Kreiranje teh spremenljivk je temeljilo na vednju kakšne lastnosti zadljega udarca gledajo ljudlje, ki notirajo te tipe točk. Tako smo izračunali 8 novih spremenljivk: pretečena razdalja igralca, oddaljenost udarca od odboja žoge, oddaljenost igralca od žoge v času udarca, hitrost žoge pred udarcem, čas, ki ga je imel igralec na voljo za udarec, hitrost teka pred udarcem, kakšen je bil kot udarca in pa pod kakšnim kotom je prišla žoga do igralca. Kot primer oddaljenost udarca od odboja žoge Slika 2:Porazdelitev tipa zadnjega udarca glede na razdaljo smo izračunali iz kombinacije spremenljivk leta žoge nad vrhom mreže v metrih previous.distance.from.sideline, player.impact.depth, Nadalje smo analizirali kako se točke zaključujejo v moški in player.impact.distance.from.sideline in previous.depth. ženski konkurenci. Na sliki 3 lahko vidimo, da sta deleža ne glede 4. Izračun skoraj 1000 novih spremenljivk, ki smo jih na spol (gender) zelo podobna, kar je bilo morda malo dobili z izračunom vsote, razlike, zmnožka, delitve, presenečenje (predvsem) glede na različne hitrosti udarcev, med povprečja in standardnih deviacij čez vse kombinacije spoloma. To razumevanje podobne distribucije, nam je dalo dveh numeričnih spremenljivk. Ta pristop se izkaže za zaupanje, da smo se odločili podatke o moških in ženskih točkah zelo uporabnega pri boosted tree-based ensemble združiti. Na ta način smo nato lahko dobili višjo klasifikacijsko algorithms, saj ta transformacija odločitvenega prostora točnost, saj smo tako dobili na voljo več učnih podatkov za omogoča algoritmu boljšo zaznamo intaktivnih in treniranje in pa predvsem optimizacijo parametrom modelov nelinearnih relacij med spremenljivkami. strojnega učenja. 68 5. Izračuna spremenljivk, ki nam je na podlagi sode ali lihe klasifikacijsko točnost za vsak razred posebej. Rezultati so dolžine točke povedala ali je zadnji udarec izvedel predstavljeni v tabeli 2. server ali ne. Napovedani razredi 6. Vse kategorične spremenljivke smo paroma združili, da je algoritem lahko zaznal vsako možno interakcijo Dejanski razredi Zaključni Prisiljena Nepotrebna udarec napaka napaka (zaporedje) med kategoričnimi spremenljivkami. Zaključni udarec 98.2% 0.5% 1.3% 7. Odstranitev vseh spremenljivk, ki smo jih dobili po zgornjih postopkih in so imele nič ali pa zelo majhno Prisiljena napaka 1.8% 89.0% 9.2% varianco vrednosti, saj take spremenljivke niso Nepotrebna napaka 2.1% 3.2% 94.6% uporabne pri strojnem učenju. Tabela 2: Klasifikacijske točnosti po razredih 4. IZDELAVA KONČNEGA MODELA Kot lahko vidimo model najslabše napoveduje prisiljene napake, Kot metrika za uspešnost predikcij se je na tekmovanju uporabilo kar je pričakovano, saj dejansko ne obstaja nekega (nepisanega) »multi-class logloss« funkcijo, ki se izračuna kot vsota logaritma pravila kdaj je nek udarec prisiljena napaka, tako da je tu tudi napake za vsak razred. To pomeni, da mora naš model vračati največ šuma v podatkih. verjetnosti za vsakega od treh razredov in ne le podatka kateri razred je najbolj verjeten. 6. ZAKLJUČEK Končni model, ki smo ga uporabili za to tekmovanje je bil V referatu smo predstavi tekmovanje z naslovom From AO to AI: eXtreme Gradient Boosting algorithm (XGB) [1]. Algoritem je Predicting How Points End in Tennis na katerem sva avtorja znan po svoji dokazani visoki učinkovitosti, hitrosti in dosegla prvo mesto. Opisali smo postopek priprave podatkov z fleksibilnosti zato se je tudi uporabil kar nekajkrat kot zmagovalni generiranjem novih spremenljivk in uporabe modela z algoritem na Kaggle tekmovanjih [2, 3]. Glede na veliko število optimizacijo parametrov ter predstavili rezultate. spremenljivk, ki smo jih imeli v učni in testni množici je bil velik Izkušnje, ki smo jih pridobili tekom tekmovanja in bi morda bile poudarek namenjen optimizaciji hiperparametrov XGB algoritma. uporabne tudi pri drugih podobnih tekmovanjih bi lahko strnili na Pri optimizaciji hiperparametrov je bil največji povdarek na sledeče točke: optimizaciji parametrov max_depth in min_child_weight. Začeli - Če imamo dve učni množici, ki sta si zelo podobni, je smo z višjo vrednostjo prametra learning_rate in ko smo dobili smiselno množici združiti, saj tako dobimo več pribljižno optimalno vrednost izbranih parametrov smo znižali podatkov za učenje in posledično boljše rezultate. learning_rate in tako še dodatno izboljšali napovedno točnost. Na - Če imamo klasifikacijski problem, ki ni primeren za vsakem koraku optimizacije smo uporabili postopek prečnega reševanje z globokimi nevronskimi mrežami se za preverjanja, da smo lahko dobljenim vrednostim parametrov lahko izhodišče lahko bolj zaupali. Postopno zmanjševanje parametra uporabi XGB saj se večinoma izkaže learning_rate je izboljšalo rezultate zaradi nelinearnosti vhodnjih spremenljivk in kot zelo dober algoritem. hkrati preprečilo, da bi se model preveč prilagodil (naučil) samo - Optimizacija hiperparametrov XGB algoritma je na učne podatke ( overfeeting) ampak, da je ostal dovolj splošen, ključnega pomena za izboljšanje rezultatov. da se je odlično odnesel tudi na še nevidenih podatkih. - Uporaba domenskega znanja za kreiranje novih spremenljivk izboljša rezultate, saj algoritem sam ne zna 5. REZULTATI smiselno povezati spremenljivk. Kot primer lahko Kot je običaj v spletnih tekmovanjih se rezultate napovedi objavi navedemo izračunano hitrost teka igralca, ki jo ljudje, ki preko platforme (v našem primeru CrowdANALYTICS). Te označujejo točke, (podzavestno) uporabljajo pri napovedi se nato po skritem ključu razdeli in uporabi za prikaz na ločevanju nepotrebne od izsiljene napake. javni in privatni lestvici. V našem primeru se je 40% napovedi uporabilo za javno lestvico in 60% za privatno. Rezultat na javni lestvici je namenjen primerjavi kvalitete njegove rešitve napram rešitvam ostalih tekmovalcev. Rezultat na privatni lestvici pa se 7. VIRI nato uporabi za določanje zmagovalca tekmovanja. Ker se rezultat [1] Chen, Tianqi, Tong He, and Michael Benesty. "Xgboost: na privatni lestvici vidi le enkrat ni mogoče, da bi se algoritem extreme gradient boosting." R package version 0.4-2 (2015): prilagodil tako, da bi imel čim boljše napovedi na privatni lestvici. 1-4. Z našim modelom smo dobili rezultat (»multi-class logloss« ) na [2] Mangal, Ankita, and Nishant Kumar. "Using big data to javni lestvici 0.179, kar je bilo dovolj za osmo mesto. Ja privatni enhance the bosch production line performance: A Kaggle lestvici pa smo dobili rezultat 0.188, kar je bilo dovolj za prvo challenge." Big Data (Big Data), 2016 IEEE International mesto. Conference on. IEEE, 2016. Za lažjo predstavo kvalitete napovedi smo izračunali tudi [3] Sheridan, Robert P., et al. "Extreme gradient boosting as a klasifikacijske točnosti. Klasifikacijska točnost modela preko vseh method for quantitative structure–activity relationships." razredov je znašala 94.5%, kar je za tak problem visoka številka. Journal of chemical information and modeling 56.12 (2016): Poleg skupne klasifikacijske točnosti pa smo izračunali tudi 2353-2360. 69 70 Zbornik 21. mednarodne multikonference INFORMACIJSKA DRUŽBA – IS 2018 Zvezek B Proceedings of the 21st International Multiconference INFORMATION SOCIETY – IS 2018 Volume B Kognitivna znanost Cognitive Science Uredila / Edited by Toma Strle, Olga Markič http://is.ijs.si 11. oktober 2018 / 11 October 2018 Ljubljana, Slovenia 71 72 PREDGOVOR / FOREWORD Na letošnji konferenci Kognitivna znanost sodelujejo avtorji/ice z različnih disciplinarnih področij, ki predstavljajo tako empirične rezultate svojih raziskav kot tudi teoretska raziskovanja. Ena izmed osrednjih tem letošnje konference je “Družbene, filozofske in etične implikacije raziskovanja v kognitivni znanosti”, avtorji pa se dotikajo tudi drugih področji kognitivne znanosti. Upamo, da bo letošnja disciplinarno in metodološko bogata kognitivna konferenca odprla prostor za izmenjavo zanimivih misli in idej ter povezala znanstvenike/ice različnih disciplin, ki se ukvarjajo z vprašanji kognitivnih procesov. Olga Markič Toma Strle * Authors at this year’s Cognitive Science conference come from numerous disciplines and present their empirical as well as theoretical work. One of this year’s main topics is “Social, philosophical and ethical implications of cognitive science” but authors also present research form other areas in cognitive science. We hope that this year’s cognitive conference, being extremely diverse in disciplines and methodologies, will become a welcoming space for exchanging intriguing ideas and thoughts as well as for bringing together scientists from all the different areas exploring the questions of cognitive processes. Olga Markič Toma Strle 73 PROGRAMSKI ODBOR / PROGRAMME COMMITTEE Olga Markič, Univerza v Ljubljani, Filozofska fakulteta Toma Strle, Univerza v Ljubljani, Pedagoška fakulteta, Center za kognitivno znanost Urban Kordeš, Univerza v Ljubljani, Pedagoška fakulteta, Center za kognitivno znanost Matjaž Gams, Odsek za inteligentne sisteme, Institut »Jožef Stefan«, Ljubljana 74 Razumevanje odraslih partnerskih odnosov v luči spolne zlorabe v otroštvu asist. dr. Sara Jerebic, spec. ZDT Teološka fakulteta Katedra za zakonsko in družinsko terapijo ter psihologijo in sociologijo religije Poljanska cesta 4, 1000 Ljubljana, Slovenija Tel: +386 1 4345810; fax: +386 1 4345854 Družinski inštitut Bližina Vodnikova ulica 11, 3000 Celje, Slovenija Tel: +386 3 4925580 e-mail: sara.jerebic@teof.uni-lj.si; sara.jerebic@blizina.si POVZETEK1 vidne takoj ali pa se pojavijo tekom odraščanja, pogosto pa se Spolna zloraba v otroštvu je travmatična izkušnja s številnimi manifestirajo v intimnih odraslih partnerskih odnosih. daljnosežnimi posledicami. Dokler travma ni razrešena, se 2. POSLEDICE SPOLNE ZLORABE V pojavlja v mislih, čustvih in vedenju. Pojavi se takrat, ko je prebujen spomin iz preteklosti, kar se pogosto zgodi v odraslih PARTNERSKIH ODNOSIH partnerskih odnosih. Raziskave kažejo, da so težave v partnerskih Raziskave kažejo, da so psihološke posledice in težave v odnosih bolj pogoste pri tistih, ki so doživeli spolno zlorabo, kot medosebnih odnosih bolj pogoste pri tistih, ki so doživeli spolno pri tistih, ki spolne zlorabe niso doživeli. Pari med drugim zlorabo, kot pa pri tistih, ki zlorabe niso doživeli [4]. Moški in poročajo o oslabljenem zaupanju, oteženem čustvenem izražanju ženske, ki so doživeli spolno zlorabo v otroštvu, v primerjavi s in težavah na področju spolnosti. Partnerski odnos se tako znajde tistimi, ki je niso doživeli, poročajo o več medosebnih težavah na na preizkušnji, oba v paru pa se spoprijemata z različnimi področju zaupanja, kontrole, odgovornosti, asertivnosti in občutku stiskami, katerih pogosto ne pripisujeta pretekli travmi. odtujenosti od svojega partnerja [5], kar se kaže kot Prepoznavanje, razumevanje z elementi dvojnega zavedanja in nezadovoljstvo v zakonskem odnosu [6]. Številne raziskave so predelava posledic, so bistvenega pomena za kakovostni pokazale, da travma prizadene intimnost s partnerji [7, 8], npr. partnerskih odnos. S prispevkom želimo poglobiti razumevanje oslabi sposobnost zaupanja in oteži čustveno izražanja, izraža pa delovanja odraslih partnerskih odnosov kot posledice spolne se tudi v obliki težav na področju spolnosti [9, 10, 11]. zlorabe v otroštvu in na podlagi klinične prakse opozoriti na Najpogosteje so težave v tuji literaturi opisane kot zasvojenost s možne oblike strokovne terapevtske pomoči pri predelavi posledic spolnostjo, spolne disfunkcije, ki posameznike zaznamujejo na spolne zlorabe v otroštvu v partnerskih odnosih. področju spolne želje, vzburjenja in orgazma [12, 13], pogoste pa so tudi bolečinske motnje [13]. Partnerski odnos se zaradi Ključne besede posledic znajde na preizkušnji, oba v paru pa se spoprijemata z Spolna zloraba v otroštvu, intimni partnerski odnos, zakonska različnimi stiskami, katerih pogosto ne pripisujeta pretekli spolni terapija. zlorabi. Da bi lahko razmejila preteklost od sedanjosti [14] in kakovostno zaživela v partnerskem odnosu, jima je lahko v pomoč 1. SPOLNA ZLORABA V OTROŠTVU relacijska zakonska terapija. Spolna zloraba v otroštvu je travmatični dogodek, kjer je otrok uporabljen kot spolni objekt za zadovoljevanje potreb ali želja 3. RELACIJSKA ZAKONSKA TERAPIJA odrasle osebe, mladostnika ali starejšega otroka oz. storilca, ki Relacijska zakonska terapija [15, 16, 17, 18] temelji na premisi, namerno išče in spodbuja zlorabo z grožnjo, silo, ustrahovanjem da zakonca drug v drugem lahko prebudita najhujše nočne more, ali manipulacijo [1]. V navedeno definicijo sodijo vrste vedenj s ki sta jih že kot otroka doživljala ob pomembnih drugih, in sicer z telesnim stikom ali brez telesnega stika, vključno z navezovanjem nezavednim namenom razrešitve [16]. V paru tako prisilno, stikov za zlorabo, spolno sugestivnim jezikom, pornografijo, ponavljata sistemske, medosebne in notranjepsihične zaznave, vojerizmom, ekshibicionizmom, ljubkovanjem, masturbacijo in kjer ponovno nezavedno ustvarita konfliktne situacije, travme, penetracijo. Otrok je lahko zlorabljen s strani nekoga v družini ali vedenje, mišljenje in čutenje [15]. Tako se spolna zloraba z vso izven nje, ki zlorabi svoj položaj moči in zaupanja [2], kar svojo razdiralnostjo ponavlja v intimnih partnerskih odnosih in pomeni, da se zloraba zgodi znotraj odnosa. V literaturi je išče razrešitev. Za boljše razumevanje v nadaljevanju navedena kot relacijska travma, ki je bolj psihopatogena kot kateri predstavljamo vpogled v del kvalitativne raziskave, v kateri smo koli drugi socialni ali fizični stres [3]. Številne posledice so lahko ugotavljali značilnosti intimnega partnerskega odnosa pri parih, ki so doživeli travmo spolne zlorabe v otroštvu in so bili vključeni v proces relacijske zakonske terapije [19]. 1 Prispevek je nastal v okviru raziskovalnega projekta J5-9349, ki ga financira ARRS. 75 4. METODOLOGIJA 5.1.3 Spolnost kot dolžnost 4.1 Metoda in udeleženci Ž:»Spolnost sem jemala, ali pa jo še kot dolžnost, ki jo je potrebno opravljati v zakonu, v smislu brez spolnosti ni zakona.« Ž: »Tako V raziskavo je bilo vključenih pet parov oz. deset udeležencev, sem mislila, nisem vedela, da imam pravico reči ne, in je bilo od katerih je vsaj eden izmed partnerjev doživel travmo spolne veliko spolnosti, ki si je nisem želela.« zlorabe v otroštvu, obenem pa je ob vključitvi v partnersko Spolnost kot dolžnost je doživljalo vseh pet žensk v paru in dva terapijo njun odnos trajal vsaj leto dni. Raziskavo smo izvedli z moška. Nekateri so o doživljanju spolnosti kot dolžnosti poročali metodo osnovane teorije, ki je omogočila sistematično zbiranje, retrospektivno, tj. pred prihodom na terapijo, drugi pa so spolnost analizo in razvoj teorije na podlagi zbranih in analiziranih kot dolžnost doživljali med terapevtskim procesom. Navedeno podatkov [20]. doživljanje lahko povežemo s samo spolno zlorabo. Storilec je 4.2 Potek raziskave otroka izkoristil za zadovoljevanje svojih potreb in otrok je moral Zbiranje podatkov je potekalo skozi terapevtski proces relacijske biti storilcu na razpolago za zadovoljevanje njegovih potreb. zakonske terapije, ki je trajala 6 mesecev (2 terapevtska cikla po Storilec je svoje hotenje (moraš, dolžan si mi) prenesel na otroka, 12 srečanj) oz. 24 srečanj, in sicer enkrat tedensko po 50 minut. ki se je čutil dolžnega. V partnerskih odnosih se občutek dolžnosti Pred vključitvijo v terapijo smo pridobili soglasje za udeležbo v prebudi ob partnerju, ki si spolnosti želi, posameznik pa doživlja, raziskavi z dovoljenjem za snemanje. Posnete terapije smo da mu mora biti na razpolago in da je dolžan z njim imeti spolni transkribirali in analizirali, vključili pa smo tudi svoja lastna odnos in tako zadovoljiti potrebe partnerja. opažanja, odzive in zapise supervizijskih srečanj. Raziskava je 5.1.4 Doživljanje telesa v spolnosti potekala na Družinskem inštitutu Bližina. V nadaljevanju Ž: »… še vedno me peče, šla sem po antibiotike ... nič, nožnica me predstavljamo rezultate dela raziskave, ki se nanašajo na področje še vedno peče.« težav v spolnosti in nekatere posege terapevtke v procesu M: »… vsakič, ko ji samo povem, da čutim vzburjenje, ona pove, relacijske zakonske terapije, ki poleg varnega terapevtskega da takoj čuti bolečino, ta pekoč občutek, samo da omenim ali odnosa omogočajo spremembe [19]. najmanjši dotik, ki bi lahko imel to konotacijo, da bo šlo za neko 5. REZULTATI približevanje ali predigra ali spolnost ...« 5.1 Medosebne težave na področju spolnosti Pari so telo doživljali različno, od prisotne fizične bolečine, ki se V obdelavi zbranih podatkov je bilo ugotovljenih šest kategorij, ki je pri eni udeleženki konstantno pojavljala, kadar je njen mož čutil predstavljajo težave na področju spolnosti: umanjkanje spolnih vzburjenost. Zaradi pekoče bolečine je pri ginekologinji večkrat odnosov, odsotnost želje po spolnosti, spolnost kot dolžnost, iskala pomoč, vendar pa fizičnega vzroka niso odkrili, prav tako doživljanje telesa v spolnosti, nezadovoljstvo v spolnosti, ji tudi antibiotična zdravila niso pomagala odpraviti pekoče doživljanje občutij v spolnosti [19]. bolečine. Tudi druga udeleženka je poročala o bolečini med samim spolnim odnosom. Dve udeleženki sta svoje telo doživljali 5.1.1 Umanjkanje spolnih odnosov kot breme, ki bi ga bilo potrebno »odstraniti«, ena udeleženka in M2: »Je bilo že bolš, sedaj pa je »slaba volja« med nama ...« Ž3: eden udeleženec med spolnim odnosom telesa nista čutila oz. je »Spolnih odnosov z možem zaenkrat še nimava, kar me straši, saj prišlo do disociacije le-tega. Trije udeleženci nad svojim telesom to traja že pol leta.« niso imeli kontrole. Pri enem ni bilo mogoče kontrolirati odziva Vsi pari so na neki točki v terapiji poročali o umanjkanju spolnih telesa, ko je prihajalo do neželenih izlivov, pri dvema pa je bilo odnosov. Nekateri so poročali o umanjkanju spolnih odnosov že izrazito samozadovoljevanje. Samozadovoljevanje se je kazalo pred pričetkom terapevtskega procesa, pri drugih je do kot odraz tesnobe in napetosti, kar je vodilo v občutja krivde. umanjkanja spolnih odnosov prišlo že v prvem delu terapevtskega 5.1.5 Nezadovoljstvo v spolnosti cikla. Za pare je umanjkanje spolnih odnosov predstavljalo težavo. Spraševali so se, kaj je narobe z njimi, da nimajo spolnih M: »Odnos je žal zelo neusklajen, saj je tako fizičnega kot odnosov. Namreč, ko se spolna zloraba prebudi v partnerskem spolnega kontakta premalo … in tudi takrat ko je, je to po njeni odnosu, pomeni, da je oseba pričela doživljati potlačena čutenja, odločitvi.” ki so vezana na samo spolno zlorabo v otroštvu. Preko različnih Ž: »Spolnost je v bedu.« sprožilcev, kot je lahko pri spolnih odnosih dotik, vonj ali okus Vsi moški udeleženci so doživljali nezadovoljstvo, ki se je kazalo [14], so se udeleženci z umanjkanjem spolnosti nezavedno v obliki prepirov, razočaranja, nesproščenosti, nepredanosti in zaščitili pred gnusom, strahom, sramom, jezo in krivdo, ki je nezmožnosti vplivanja na to, kdaj bosta imela spolni odnos. povezana s samo zlorabo. Ženske udeleženke so poročale o prepirih, nesproščenosti in razočaranju. Glede na to, da so pari doživljali spominske 5.1.2 Odsotnost želje po spolnosti prebliske, disociacije, bolečinske motnje, strahove, tesnobna Ž: »Jaz nimam želje po spolnih odnosih, on pa razume, kot da ga razpoloženja, odpor in nezaupanje, je nezadovoljstvo v spolnosti ne maram.« normalni odgovor na nenormalno izkušnjo v otroštvu oz. O odsotnosti želje po spolnosti so poročale samo ženske, vendar normalna posledica prebujene zlorabe v partnerskem odnosu. pa je odsotnost želje pri enem v paru povzročila neprijetna občutja 5.1.6 Doživljanje občutij v spolnosti pri obeh, kar je povedala ena izmed udeleženk (podčrtano) in kar se je za pomembno izkazalo tudi med samim procesom terapije. Ž: »Me pa velikokrat prešine misel, da tega ne smem početi, da ne Ena izmed udeleženk je npr. povedala, da ji odleže, če gre z smem uživati v spolnem odnosu, ker bom kaznovana.« M: »... Ma otrokom spat, saj nima nobene želje po spolnosti. me jezi, ja! Najprej ja, potem ne ... daj, odloči se že enkrat!« V raziskavi se je izkazalo, da so pari doživljali različna občutja, od negotovosti do krivde, strahu, sramu, jeze, žalosti, gnusa, odpora, 2 moški stiske in zmedenosti. Občutja so bila močno prisotna tudi v terapevtskem procesu, pri katerem je bila osnovna terapevtska 3 ženska intervencija regulacija le-teh. Ženske udeleženke so doživljale 76 močna občutja krivde. Pri treh udeleženkah se je občutje občutek varnosti in kontrolo nad tem, kar se jim dogaja, so najprej prebudilo po spolnem odnosu, v enem primeru je bilo povezano s potrebovali vzpostaviti kontrolo nad telesom in čutenji, s katerimi kaznijo, pri dveh udeleženkah pa s tem, ko zavrneta spolni odnos. so postopoma prihajali v stik. Terapevtka je tudi kasneje, ne samo Eden izmed udeležencev moškega spola je prav tako vedno po na začetnih srečanjih, zagotavljala varnost s terapevtskim spolnem odnosu doživljal občutke krivde. Pri krivdi gre za odnosom, v katerem je parom ovrednotila njihovo doživljanje in s samoobtoževanje, ki je posledica spolne zlorabe. Krivijo se, ker sočutnimi odzivi nudila občutek sprejetosti. niso preprečili ali ustavili zlorabe, pogosto sebe krivijo, da je do spolne zlorabe sploh prišlo. To se kaže tudi v izjavi ene izmed 5.2.2 Priznanje, da se je zloraba res zgodila udeleženk, ki sta jo zlorabljala dva storilca: »... Ne vem ... nekaj je Pomemben element v terapiji je bilo razkritje zlorabe. Osebe, ki že moglo biti na meni ... mogoče moje oči ...« Moški udeleženci so so doživele spolno zlorabo, težko zaupajo, ne samo partnerju, tudi doživljali žalost, jezo in tudi gnus. Eden izmed udeležencev je terapevtu. Najbolj jih ovira sram, ki ohranja negativno podobo o pojasnil, da zelo pogosto med samim spolnim odnosom doživi sebi. Ko govorimo o razkritju spolne zlorabe, to ne pomeni, da gnus, ki ga popolnoma zablokira. Navedeno lahko pojasnimo s mora oseba grafično opisovati spolno zlorabo, temveč da sistemsko razsežnostjo travme, ko se na moža prenesejo občutja, spregovori o svojem doživljanju. Prvi korak je bil, da je lahko ki jih žena nosi globoko v sebi. Ob spolni zlorabi se kot otrok ni naglas priznala, da se je zloraba res zgodila. Ne gre samo za smela oz. mogla jeziti, zaradi izdajstva je bila žalostna, doživljala besede oz. kognitivno komponento, temveč tudi za afektivni je tudi občutja gnusa. Občutja, ki jih je povzročil storilec in so vidik. Ko spregovorijo o čustvenem doživljanju, jim ni več ostala »zaklenjena v telesu«, so se preko mehanizma projekcijske potrebno skrivati sramu, kar jih razbremeni. Navedeno tudi identifikacije prenesla na moža oz. partnerja, ki prične čutiti in pripomore, da »težkih vsebin« ne odrežejo, temveč se prično doživljati vsa občutja na svojem telesu. Navedene posledice soočati tudi s posledicami, kot so npr. spominski prebliski. Do predstavljajo stiske, ki jih doživljajo pari v partnerskih odnosih. navedenega lahko pride, kadar je v odnosu dovolj varnosti, ki pa Klinična praksa kaže, da posledic pogosto ne povezujejo s jo mora najprej zagotoviti terapevtka. Takrat potem lahko oba v preteklo zlorabo, neredki pa se tudi ne zavedajo, da so določene paru sprocesirata žalost, jezo, sram, strah in gnus. izkušnje iz njihovega otroštva po svoji naravi pravzaprav bile 5.2.3 Psihoedukacija spolna zloraba. Da bi lahko resnično zaživeli kakovostni Del terapevtskega procesa je bila psihoedukacija, v kateri je partnerski odnos so potrebni prepoznavanje, razumevanje z terapevtka pojasnila samo naravo travme spolne zlorabe, vpliv, ki elementi dvojnega zavedanja in predelava posledic, kar omogoča ga je imela na osebo, ki jo je doživela, in posledice, ki jih doživlja zakonska oz. partnerska terapija. spolno zlorabljena oseba ter tudi njen partner. Pari, ki so doživeli 5.2 Proces relacijske zakonske terapije spolno zlorabo v otroštvu so doživljali veliko simptomov, kot so V samem procesu relacijske zakonske terapije smo s posegi nočne more, strahovi, hude stiske in so se spraševali, če so sploh terapevtke zasledovali spremembe, ki so tudi cilj terapevtskega še normalni. Terapevtka je normalizirala njihovo doživljanje v procesa in sicer spremembe na vedenjski, kognitivni in čutenjski povezavi s preteklo spolno zlorabo. Pomembno je bilo, da oba v ravni kot tudi spremembe na intrapsihičnem, medosebnem in paru razumeta, kar se je dogajalo, saj je partnerja, ki je doživel sistemskem delovanju. zlorabo sram, da tako čuti in doživlja. Terapevtka je parom omogočala občutek kontrole, da so lahko sami izbirali teme in se 5.2.1 Zagotavljanje varnosti pogovarjali o tem, kar so želeli. Če je npr. klientka dejala, da o Na začetku terapevtskega procesa je bila prva naloga terapevtke spolnosti še ne želi govoriti, jo terapevtka nikoli ni silila, temveč zagotavljanje varnosti. Tedenska struktura je parom zagotavljala jo je zanimalo, kaj bi čutila, če bi govorila. Pogosto je bil sram predvidljivost, ki je v svojem domačem okolju niso bili vajeni, in tisti, ki ji je onemogočal spregovoriti o stiski. Pomembno je, da jim je s tem omogočala spremembo. Obenem smo v terapevtskem razumeta oba, da v primeru raznovrstnih težkih simptomov ne gre procesu naslavljali njihova doživljanja glede vključevanja v za patologijo osebe, ki je bila spolno zlorabljena v otroštvu, raziskavo in načinom, da se srečanja snemajo. Ob tem smo parom temveč za normalne odzive na nenormalne, travmatične dogodke. obrazložili, da so v svoji odločitvi popolnoma svobodni in v Gre za del pomoči vzpostavljanja kontrole nad situacijo, ko vidi, kolikor bi se odločili, da ne želijo več sodelovati v raziskavi, kaj se dogaja, in ve, da gre samo za simptom, da ni z njo nekaj njihova odločitev ne bo vplivala na kakovost terapevtske narobe, da razume najprej na kognitivnem nivoju, kaj se dogaja. obravnave ter da že podpisano soglasje lahko tudi med terapevtskim procesom kadarkoli iz kakršnihkoli razlogov 5.2.4 Postavljanje meja prekličejo. S tem smo parom ob zagotavljanju varnosti omogočili Ena izmed pomembnih tem je bila spoštovanje do lastnega telesa. tudi možnost kontrole. Namreč ob samem dogodku zlorabe je bil Kadar je terapevtka čutila, da si ženska ne želi spolnih odnosov, otrok popolnoma brez kontrole, saj je bil odrasli tisti, ki je vse kljub temu pa se vanje spušča, ji je dala dovoljenje, da ji tega ni nadziral, so med terapevtskim procesom dobili možnost, da le- več potrebno početi. Ženske so v terapiji povedale, da jim je tega lahko kontrolirajo, tudi s temami, ki so jih na srečanja slednje zelo pomagalo, da so lahko pričele poslušati svoje telo, prinašali. Terapevtka jim je vedno zagotavljala, da v kolikor o obenem pa je navedeno razbremenilo tudi njihove partnerje. nečem ne želijo govoriti, jim to ni potrebno oz. lahko povedo 5.2.5 Razmejevanje preteklosti od sedanjosti samo toliko, kolikor se jim zdi varno povedati, obenem pa vedno Pomemben poseg terapevtke je bilo tudi razmejevanje preteklosti lahko ustavijo, vprašajo oz. izrazijo svoje želje glede samega od sedanjosti oz. dvojno zavedanje. Oba zakonca oz. partnerja v procesa terapije. Na koncu vsakega srečanja je bila narejena paru sta pogosto doživljala hude čustvene stiske, ki sta jih refleksija v smislu, kako bo oditi po posamičnem srečanju. Pari so pripisovala drug drugemu. Preteklost se je preko čustev in telesa imeli tako možnost spregovoriti o svojem čustvenem doživljanju prebujala v sedanjosti, drug ob drugem. Navedena dinamika je in izraziti svoje misli v samem terapevtskem procesu. Glede same bila pogosta v spolnih odnosih, pri katerih je žena svojega moža varnosti smo v terapevtskem procesu tudi preverjali, kako je s pričela doživljati kot storilca. Ko je žena zavračala spolne odnose, trenutno varnostjo izven terapevtskega procesa (doma, v širši se je mož počutil zavrnjenega. Terapevtka jima je pomagala družini, v službenem okolju). Da bi lahko v polnosti pridobili razumeti, da žena ne zavrača moža, temveč čutenja, kot so sram, 77 gnus v povezavi s ponižanjem in razvrednotenjem, kar se prenaša longitudinal study . Journal of Interpersonal Violence 21, 1, preko relacijskega mehanizma projekcijske identifikacije iz 42-57. preteklosti v sedanjost in se odigrava v njunem partnerskem [7] Davis, J. L. in Petretic-Jackson, P. A. 2000. The impact of odnosu. V tem delu je bilo pomembno razumevanje, da se zloraba child sexual abuse on adult interpersonal functioning: A ne dogaja več, so pa čutenja tista, ki se prebujajo v sedanjosti. review and synthesis of the empirical literature. Aggression 5.2.6 Povezovanje v paru and violent behavior 5, 3, 291-328. Bistveno v relacijski zakonski terapiji je povezovanja para na nov [8] Rumstein-McKean, O. in Hunsley, J. 2001. Interpersonal and način. Namreč glavnina procesa poteka na njunem partnerskem family functioning of female survivors of childhood sexual odnosu v sedanjosti, ne na pretekli travmi. V kolikor sta se abuse. Clinical Psychology Review 21, 3, 471-490. zmožna čustveno povezati na drugačen način, potem partnerski [9] Colman, R. A. in Spatz Widom, C. 2004. Childhood abuse odnos nudi toliko varnosti, da jima ni potrebno več ohranjati and neglect and adult intimate relationships: a prospective obrambnih drž, kot so obtoževanje in umikanje, temveč lahko study. Child Abuse & Neglect 28, 11, 1133-1151. delita najbolj ranljiva čutenja. Šele, ko sta v polnosti ranljiva drug z drugim, sta sposobna razviti sočutje drug do drugega, do male [10] Leonard, L. M. in Follette, V. M. 2002. Sexual functioning in deklice in malega dečka, ki sta še vedno globoko v njima, women reporting a history of child sexual abuse: review of povezana s težkimi občutji in moledujeta za pozornost, sprejetost the empirical literature and clinical implications. Annual in ljubezen. Review of Sex Research 13, 1, 346-388. 6. ZAKLJUČEK [11] Najman, J. M., Dunne, M. P., Purdie, D. M., Boyle, F. M. in Coxeter, P. D. 2005. Sexual abuse in childhood and sexual Pari z izkušnjo spolne zlorabe v otroštvu se srečujejo s težavami v dysfunction in adulthood: An Australian population-based medosebenih intimnih odnosih in potrebujejo strokovno pomoč, ki study. Archives of sexual behavior 34, 5, 517-526. jim pomaga razumeti njihove trenutne težave v luči pretekle [12] Carnes, P. in Moriarity, J. 1997. Sexual anorexia: spolne zlorabe. Ključni pri pomoči so lahko zakonski terapevti, ki s pomočjo relacijske zakonske terapije, svojim sočutnim odnosom overcoming sexual self-hatred. Center City, Minn.: Hazelden. in ustreznim znanjem pripomorejo k zdravljenu preteklih travm in skupni rasti v partnerskem odnosu. Glede prizadevanja pomoči [13] Maltz, W. in Holman, B. 1987 . Incest and sexuality: a guide parom bi bilo potrebno več pozornosti nameniti preventivnim to understanding and healing. Lexington: Heath and Com. programom, ki bi s svojo izobraževalno naravo pripomogli k [14] Rothschild, B. 2000. The body remembers: the lajšanju marsikatere stiske, ki jo doživljajo pari. psychophysiology of trauma and trauma treatment. New 7. REFERENCE York: Norton. [1] Finkelhor, D. in Korbin, J. 1988. Child abuse as an [15] Gostečnik, C. 2004. Relacijska družinska terapija. Ljubljana: international issue. Child Abuse & Neglect 12, 1, 3-23. Brat Frančišek in Frančiškanski družinski inštitut. [2] Sanderson, C. 2006 . Counselling adult survivors of child [16] Gostečnik, C. 2007. Relacijska zakonska terapija. Vol. 9, sexual abuse. London, Philadelphia: Jessica Kingsley. Monografije FDI. Ljubljana: Brat Frančišek; Frančiškanski družinski inštitut. [3] Schore, A. N. 2003. Affect regulation and the repair of the self (norton series on interpersonal neurobiology). New [17] Gostečnik, C. 2011 . Inovativna relacijska družinska terapija: York: WW Norton & Company. inovativni psiho-biološki model. Ljubljana: Brat Frančišek, Teološka fakulteta in Frančiškanski družinski inštitut. [4] Briere, J. N. in Elliott, D. M. 1994. Immediate and Long- Term Impacts of Child Sexual Abuse. The future of children [18] Gostečnik, C. 2015 . Inovativna relacijska zakonska terapija. 4, 2, 54-69. Ljubljana: Brat Frančišek, Teološka fakulteta in Frančiškanski družinski inštitut. [5] Whiffen, V. E., Thompson, J. M. in Aube, J. A. 2000. Mediators of the link between childhood sexual abuse and [19] Jerebic, S. 2016. Travma spolne zlorabe v otroštvu in intimni adult depressive symptoms. Journal of Interpersonal partnerski odnos. Doktorska disertacija. Univerza v Violence 15, 10, 1100-1120. Ljubljani, Teološka fakulteta. [6] Liang, B., Williams, L. M. in Siegel, J. A. 2006. Relational [20] Strauss, A in Corbin, J. 1990. Basics of Qualitative Research. outcomes of childhood sexual trauma in female survivors a Newbury Park, CA: Sage 78 Do healthy participants make advantageous decisions? Evidence from the Iowa Gambling Task Maša Vukčević Marković Psychosocial Innovation Network, Belgrade and Belgrade University, Faculty of Philosophy, Laboratory for Research of Individual Differences Gospodar Jevremova 48 11000 Belgrade, Serbia +381637234658 masa@psychosocialinnovation.net ABSTRACT making even though they do not necessarily need to be aware of Somatic Marker Hypothesis (SMH) suggests that decision which decisions are good and which ones are bad [5, 9, 10]. making in uncertainty relays on somatic markers – emotional reactions reflected in bodily states which lead person towards 1.1 IOWA GAMBLING TASK advantageous decision making. Authors of SMH created a task Authors of Somatic Marker Hypothesis created a task aiming to aiming to access decision making in uncertainty – Iowa access advantageous versus disadvantageous decision making – Gambling Task (IGT), suggesting that since healthy participants Iowa Gambling Task (IGT) [4]. In the IGT, participants are tend to develop advantageous decision-making strategy, they presented with four deck of cards. After selecting a card, will have good IGT performance. Recent studies however, participants either win or lose different amounts of play money. question good IGT performance among healthy participants thus Unknown to the participants, selecting cards from two of the imposing question on their advantageous decision-making decks – C and D will result in eventual gain (‘advantageous’, or strategies. The aim of this study was to explore IGT ‘good’ decks), whereas selecting cards from the other two decks performance among healthy participants. Participants (N=268) - A and B will result in eventual loss (‘disadvantageous’, or filled standard version of IGT, consisting of 100 trials in which ‘bed’ decks). The task has 100 trials, so that, according to participants select cards from four decks: A and B, that bring authors’ idea, participants still could not understand reword- higher rewards and penalties (good decks) and C and D that punishment scheme. Authors suggested and demonstrated in bring small rewards and penalties (bad decks). At the beginning several studies that healthy individuals will select more cards of the IGT, participants get 2000$ of play money, with from the good than the bad decks, thus showing tendency instruction to try to gain as much money as possible. Results towards advantageous decision making [4, 17]. Task validation show that participants tend to lose rather then gain money, with was demonstrated in numerous studies which indicated that almost half of participants showing impaired IGT performance. impaired IGT performance have persons with problems in In addition, results indicate that frequency of punishment, rather advantageous decision making in everyday life [7, 12, 14, 15, than overall goodness or decks is factor relevant for specific 18], as well as patients with lesions in the brain region relevant deck selection. Study results question wheatear healthy for the judgment and decision-making in real-life settings [4, 6, participants apply advantageous decision-making strategies 9]. when performing IGT, indicating that future studies are needed Recent studies, however, question whether healthy persons do in order to explore factors contributing to advantageous decision prefer good over a bed decks and indicate that healthy controls making in uncertainty. tend to select bed decks equally or even more frequently than good ones [2, 8, 11, 16]. Moreover, when selection of each deck Keywords specifically was analyzed, it was shown that bed deck B is equally or even more frequently selected then good decks C and Iowa Gambling Task, IGT, Somatic Marker Hypothesis, D [8, 11], while bed deck A represent the least selected deck Decision Making. [11, 13, 19]. Possible explanation can be found in differences among frequency of punishment within good and bed decks, 1. INTRODUCTION resulting in decks A and C being frequent punishment decks, Numerous studies exploring decision making in uncertainty rely while decks B and D represent non-frequent punishment decks, on Somatic Marker Hypothesis (SMH) [3, 5, 9]. This hypothesis which was not taken into consideration in the early IGT studies. suggests that normal decision-making in humans is led by Studies that included both criteria into analysis of IGT somatic markers - bodily states which represent emotional performance, indicated that frequency of punishments, rather reactions to different stimuli or to possible courses of action. than overall goodness of deck seems to be leading participants Authors of SMH suggest that somatic markers, activated in towards selection of specific decks [8, 11, 13, 19]. Thus, recent situation of uncertainty can effectively reflect the goodness or studies indicated that it is not certain whether healthy controls badness of the possible outcome, leading a person towards show homogeneous tendency towards advantageous decision advantageous decision making. Importantly, authors suggest that making and whether overall goodness of decks, or rather somatic markers represent a “gut feeling”, which can operate frequency of its punishments will be relevant factor for deck both consciously and unconsciously. SMH, thus suggests that selections. somatic markers can lead people to advantageous decision 79 1.2 PRESENT STUDY Table 2. Percentage of participants who selected more cards In this study 268 (81.7% females, average age 20 years) from different decks participants, completed standard version of Iowa Gambling Task (IGT). % of % of participants participants 2. RESULTS On average, participants scored 1475$ on IGT, ranging from - with more with more 4525 to 6075, indicating that participants tend to rather lose then cards from cards from gain money on IGT, i.e. that participants, on average, do not show advantageous decision-making strategy. Table 1 presents good decks bed decks descriptive measures of each deck selection separately. 54.5% 45.5% Table 1. Descriptive statistic of each deck selection, in proportion % of participants Deck M SD Min Max zSk zKu with more cards 76.5% 39.9% 36.6% A .17 .098 0 .75 8.18 from non-frequent ** 16.66* punishments decks B .29 .142 0 .72 1.20 .60 % of participants C .23 .156 0 .1 15.21* 27 with more cards 23.5% 14.6% 9% from frequent D .31 .175 0 .1 9.73** 9.51 punishments decks M – mean; SD – standard deviation; Min - minimum; Max – maximum; zSk – standardized Skewness; zKu – standardized Kurtosis; ns - p > .05; * - p <.05; ** - p <.01. 3. DISCUSSION Results show that participants the most frequently selected cards Study results indicate that, contrary to IGT authors’ from deck D and deck B subsequently, somewhat less frequently expectations, participants tend to lose rather than win money on they selected cards from deck C, while they selected cards from IGT and that almost half of participants have impaired deck A the least frequently among all decks. Table 2. presents performance, i.e. more choices from bed then from good decks. percentage of participants who selected cards from good versus This result is in line with studies questioning whether healthy bed decks at more than 50% of trials, as well as percentage of controls prefer good over a bad deck and whether they show participants who selected cards from non-frequent punishments overall advantageous decision-making strategy [2, 8, 11, 16, 17]. decks B, D versus frequent punishments decks A and C at more Results also show that majority of participants prefer non- than 50% of trials. Results indicate that somewhat less than half frequent punishment decks B and D in comparation to frequent of participants have impaired performance [4], i.e. they selected punishment decks A and C. These results indicated importance more cards from bed then good decks, while more than three of frequency of punishment in deck selection. They are also in quarters of participants selected more cards form non-frequent line with results of a previous studies that considered both punishments decks B and D, in comparation to frequent- frequency of punishments and overall goodness of decks, punishments decks A and C. There is no significant difference in indicating that participants tend to the most frequently select number of participants who selected more cards from good then cards from deck D (good, non-frequent punishments deck), then bed decks [χ2(1) = 2.149; p = .143], while significant difference B (bed, non-frequent punishments deck), then deck C (good, was found in the number of participants who selected more frequent punishments deck), while they the least frequently cards from non-frequent punishment decks B and D in select cards from deck A (bed, frequent punishments deck) [1, 8, comparation to selection of cards from frequent punishment 11]. decks A and C [χ2(1) = 75.239; p < .001]. Results of this study indicated that healthy controls do not show homogenous IGT performance, nor advantageous decision- making strategy. In addition, results indicated than frequency of punishments rather than overall goodness of deck is relevant for deck selection, showing that immediate reword/punishment seems to be more important than long term consequences in decision making. These results do not question Somatic Marker Hypothesis, just call for additional studies that will improve understanding of factors that contribute to good IGT performance, and overall advantageous decision making in situations of uncertainty. 80 4. REFERENCES [12] Hanson, K., Luciana, M., and Sullwold, K. 2008. Reward- [1] Alarcón, D., Amián, J. G., and Sánchez-medina, J. A. 2015. related decision-making deficits and elevated impulsivity Enhancing emotion-based learning in decision-making among MDMA and other drug users. Drug Alcohol Depen, under uncertainty. Psicothema, 27(4), 368–373. 96(1–2), 99–110. https://doi.org/10.7334/psicothema2015.45 https://doi.org/10.1016/j.drugalcdep.2008.02.003 [2] Bark, R., Dieckmann, S., Bogerts, B., and Northoff, G. [13] Kjome, K. L., Lane, S. D., Schmitz, J. M., Green, C., Ma, 2005. Deficit in decision making in catatonic L., Prasla, I., … Moeller, F. G. 2010. Relationship between schizophrenia : An exploratory study, Psychiat res 134, impulsivity and decision making in cocaine dependence. 131–141. https://doi.org/10.1016/j.psychres.2004.04.013 Psychiatry Res, 178(2), 299–304. https://doi.org/10.1016/j.psychres.2009.11.024 [3] Bechara, A., and Damasio, A. 2005. The somatic marker hypothesis: A neural theory of economic decision. Game [14] Lev, D., Hershkovitz, E., and Yechiam, E. 2008. Decision Econ Behav, 52(2), 336–372. making and personality in traffic offenders: a study of https://doi.org/10.1016/j.geb.2004.06.010 Israeli drivers. Accident Anal Prev, 40(1), 223–30. https://doi.org/10.1016/j.aap.2007.05.009 [4] Bechara, A., Damasio, A., Damasio, H., and Anderson, A. 1994. Insensitivity to future consequences following [15] Noël, X., Bechara, A., Dan, B., Hanak, C., and Verbanck, damage to human prefrontal cortex. Cognition, 50, 7–15. P. 2007. Response inhibition deficit is involved in poor decision making under risk in nonamnesic individuals with [5] Bechara, A., Damasio, H., Tranel, D., and Damasio, A. alcoholism. Neuropsychology, 21(6), 778–86. 2005. The Iowa Gambling Task and the somatic marker https://doi.org/10.1037/0894-4105.21.6.778 hypothesis: some questions and answers. Trends Cogn Sci, 9(4), 159-62; discussion 162–4. [16] Sevy, S., Burdick, K., Visweswaraiah, H., Abdelmessih, S., https://doi.org/10.1016/j.tics.2005.02.002 Lukin, M., Yechiam, E., and Bechara, A. 2007. Iowa Gambling Task in schizophrenia : A review and new data in [6] Bechara, A., Tranel, D., and Damasio, H. 2000. patients with schizophrenia and co-occurring cannabis use Characterization of the decision-making deficit of patients disorders. Schizophr Res, 92, 74–84. with ventromedial prefrontal cortex lesions. J Neur, 123 ( https://doi.org/10.1016/j.schres.2007.01.005 Pt 1, 2189–202. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/11050020 [17] Steingroever, H., Wetzels, R., Horstmann, A., Neumann, J., and Wagenmakers, E.-J. 2013. Performance of healthy [7] Brevers, D., Cleeremans, A., Goudriaan, A., Bechara, A., participants on the Iowa Gambling Task. Psychol Kornreich, C., Verbanck, P., and Noël, X. 2012. Decision Assessment, 25(1), 180–93. making under ambiguity but not under risk is related to https://doi.org/10.1037/a0029929 problem gambling severity. Psychiat Res, 200(2–3), 568– 74. https://doi.org/10.1016/j.psychres.2012.03.053 [18] Struglia, F., Tomassini, a., Stratta, P., Spaziani, D., & Rossi, A. (2009). P01-75 Cognitive functions, impulsivity [8] Caroselli, J. S., Hiscock, M., Scheibel, R. S., and Ingram, and personality traits in alcohol dependent subjects. Eur F. 2006. The simulated gambling paradigm applied to Psychiat, 24, S463. https://doi.org/10.1016/S0924- young adults: an examination of university students’ 9338(09)70696-0 performance. Appl Neuropsychol, 13(4), 203–12. https://doi.org/10.1207/s15324826an1304_1 [19] Wood, S., Busemeyer, J. R., Koling, A., Cox, C. R., & Davis, H. (2005). Older adults as adaptive decision makers: [9] Damasio, A. 1994. Descartes’ error: Emotion, reason, and evidence from the Iowa Gambling Task. Psychol Aging, the human brain. New York, NY: Putnam. 20(2), 220–5. https://doi.org/10.1037/0882-7974.20.2.220 [10] Damasio, A. 1999. The Feeling of What Happens. [11] Fernie, G., and Tunney, R. J. 2006. Some decks are better than others: the effect of reinforcer type and task instructions on learning in the Iowa Gambling Task. Brain Cognition, 60(1), 94–102. https://doi.org/10.1016/j.bandc.2005.09.011 81 Palaeolithic toolmaking and the evolution of cognition and language Petar Gabrić Marko Banda Ivor Karavanić Department of Linguistics Department of Archaeology Department of Archaeology Faculty of Humanities and Social Faculty of Humanities and Social Faculty of Humanities and Social Sciences Sciences Sciences University of Zagreb University of Zagreb University of Zagreb pgabric@ffzg.hr mabanda@ffzg.hr ikaravan@ffzg.hr ABSTRACT Research in this area has been in large part focused on inferring This paper reviews in short the current research on the hypothesis “modern” and “symbolic behavior” from archaeological data, of coevolution between Palaeolithic stone tool manufacture on such as ochre and ornament use, figurative art production, one side, and cognition and specifically language on the other. Of subsistence strategies etc. (e.g. d’Errico et al. 2003). The research particular interest are behavioral and neuroimaging studies. was led by the premise that such behaviors indicate the existence of a package of “higher” cognitive abilities. However, these Keywords discussions have been heavily criticized from a methodological language evolution, cognitive evolution, Palaeolithic, stone tools, point of view (e.g. Botha 2009, 2012), and they currently struggle Oldowan, Acheulean, Levallois, cultural transmission to find support in cognitive science (e.g. Garofoli 2014). 1. INTRODUCTION1 2. PALAEOLITHIC STONE TOOLS In the past two decades great interest has emerged for Recently, vast literature has appeared supporting the coevolution interdisciplinary discussion on the evolution of cognition and of cognition and language on one side, and the manufacture of specifically language (e.g. Janković and Šojer 2015). Recent work Palaeolithic stone tools on the other (see e.g. Stout and Hecht strongly emphasizes the role of cumulative culture, i.e. products, 2015 for a review). Archaeology has recognized several skills and knowledge created over generations of human lifetimes, phylogenetic phases of stone tool manufacture, and research in in the evolution of specific hominin cognitive abilities, including this topic has tried to correlate these phases with evolutionary language (e.g. Lotem et al. 2017). developments in cognition, most notably visuospatial processing, executive functioning, social cognition and language. Moreover, it remains a controversial issue whether evolutionary changes in hominin cognition should be viewed as structural 2.1 Lomekwian and/or functional novelties, or as exaptations of preexisting The earliest known hominin stone tools dated to ~3.3 mya come primate structures and/or functions (e.g. Stout and Chaminade from Lomekwi, Kenya, and consist of various rocks used for 2007: 1092). Nevertheless, ample literature now acknowledges pounding, stone anvils and cores from which flakes were struck that exaptation played a major role in hominin cognitive evolution using a hammerstone (Harmand et al. 2015). The site predates the (e.g. Kolodny and Edelman 2018), but also e.g. in the evolution of earliest known Homo specimens dated to ~2.8 mya (Villmoare et hominin musculature underlying bipedalism, tool use and speech al. 2015). Cognitive implications of the Lomekwi findings, (Diogo 2018). For language specifically, it has been e.g. argued regarding among others bimanual coordination and lateralization that both child first language and adult second language more generally, are discussed in Lewis and Harmand (2016). acquisition are served by general-purpose learning systems of declarative and procedural memory (Hamrick et al. 2018), and 2.2 Oldowan that linguistic semantic processing is highly dependent on Oldowan represents the next stage of hominin stone technology, modality-specific processing mechanisms of the sensorimotor ranging from ~2.6 (Semaw 2000) to ~1.42 mya (Toth and Schick system and not only on amodal or “abstract” semantic operations 2018: 7). It is generally considered to be characterized by flaked (e.g. Pulvermüller 2013). pebble tools, namely choppers and chopping tools, used for pounding or bone splitting. The knapping of these pebbles Still, empirical research in the evolution of cognition and produced smaller flakes which could have been utilized for meat language remains relatively scarce, mostly due to serious butchering (Toth 1987). Oldowan is most commonly associated methodological limitations (see §3). One approach has been to with H. habilis, but it is not excluded that australopithecines find behavioral correlates of specific cognitive functions in the engaged with Oldowan tools (Karavanić and Janković 2009: 107). Palaeolithic record, the archaeological period beginning with the Compared to Lomekwian Oldowan is described as entailing appearance of first known hominin stone tools ~3.3 mya greater abundance of flakes and smaller tool dimensions (Hovers (Harmand et al. 2015) and finishing with the end of the last glacial 2015). Toth and Schick (2018) argue that Oldowan findings are period (Karavanić and Janković 2009). suggestive of the incorporation of stone tools as a critical adaptive component which presumably led to more complex subsistence 1 Abbreviations: ESA=Early Stone Age (~Oldowan and strategies, social behavior and communication. Acheulean); IPS=intraparietal sulcus; SMA=supplementary motor area; SMG=supramarginal gyrus; vPrCG=ventral precentral gyrus; vPMC=ventral premotor cortex 82 2.3 Acheulean both teaching conditions improved performance compared to The oldest Acheulean finds are dated to ~1.7 mya (Diez-Martín et imitation/emulation and that the effects were most pronounced in al. 2015) and lasted in some parts of the world up to ~100 kya. the verbal teaching condition. Additionally, Cataldo et al. (2018) Acheulean is most commonly associated with H. erectus and found in their study of flaking that subjects instructed with Middle Pleistocene hominins, such as H. heidelbergensis (Torre gesture-free verbal teaching underperform relative to subjects 2016). The most commonly recognized Acheulean tool is the instructed by gestural or “full-language” teaching. Cataldo and handaxe and its iterations, a bifacially flaked stone tool usually of colleagues thus conclude that while gestural communication was amygdaloidal form (Torre 2016). Furthermore, Acheulean likely under selective pressures in the Oldowan populations, this assemblages also contain various flake tools. H. ergaster/erectus is not necessarily the case for (spoken) language. was the first hominin species to expand out of Africa settling vast As to post-Oldowan techniques, Putt et al. (2014) compared the areas of Asia and Europe. However, this expansion did not efficiency of imitation/emulation and verbal teaching in the necessarily entail the spread of Acheulean, because the earliest acquisition of Acheulean handaxe manufacture. They reported no known hominin sites in Europe exhibit a stone industry with an significant differences e.g. in shape and symmetry of the tools Oldowan tradition (Karavanić and Janković 2009: 120). ~500 kya, between the groups, but the non-verbal group produced more many European stone industries start to resemble the later efficient flakes as seen in higher ratios of platform width to Acheulean with its well-formed handaxes (Karavanić and platform thickness and size to mass compared to the verbal group. Janković 2009: 124). It has been argued that Acheulean tool Putt and colleagues concluded that (spoken) language wasn’t production is more cognitively demanding relative to Oldowan necessary for the transmission of handaxe manufacture and that its due to its supposed higher hierarchical and sequential complexity implementation in the earliest learning stages might hinder (e.g. Torre 2016: 8ff.). progress. Ohnuma et al. (1997) compared the efficiency of Due to space limitations and the fact that empirical studies almost “verbal” and “non-verbal demonstration” in the acquisition of exclusively focused on Oldowan and Acheulean, later hominin Levallois flaking. There were no significant differences between technologies will not be discussed here (but see §3). Additionally, the two conditions in the acquisition rates and mean times, and further insights into the cognitive implications of Palaeolithic flaking success from which Ohnuma and colleagues concluded tools have come from the recent discoveries of primate tool- that (spoken) language was not necessary for Levallois flaking. related behaviors (see Haslam et al. 2017 for a review). Nonetheless, Levallois has been linked to increased demands in sequential and hierarchical planning as well as self-monitoring 3. EMPIRICAL STUDIES (e.g. Schlanger 1996: 246ff.). Empirical studies concerned with the coevolution of cognition and It should, however, be noted that there exist considerable stone tool manufacture include (1) behavioral studies methodological variations in the observed studies as in the investigating the effects of different cultural transmission neuroimaging studies, e.g. in the selection, homogeneity and conditions on the acquisition of stone tool manufacture, and (2) preparatory modification of raw material, learning duration and neuroimaging studies of stone toolmaking or its observation. rates, presence of an experienced teacher, number of subjects etc. More detailed and critical analyses of the studies in question are, unfortunately, beyond the limitations of this paper. 3.2 Neuroimaging studies Some methodological constraints include (1) difficulties in Neuroimaging studies have been mainly trying to determine the neuroimaging of stone tool manufacture, e.g. due to the static cortical structures which would have been under selective nature of the conventional methods such as fMRI, or the possibly pressures, and then tying these structures to specific cognitive more harmful effects of FDG-PET on subjects due to longer and functions. Hecht et al. (2014) conducted a longitudinal DTI study more demanding activities (Stout and Chaminade 2007: 1096), (2) with participants who in a two-year program acquired basic small numbers of subjects and/or shorter learning periods due to Oldowan flaking, Acheulean handaxe manufacture and Levallois the need of collecting large amounts of raw material for tool flaking. They recorded fractional anisotropy changes in branches production and the known infrastructural limitations of of the superior longitudinal fasciculus leading into the left SMG neuroscientific research on the number or the mere existence of and vPrCG, and right pars triangularis (part of Broca’s area). They subjects, and (3) recruiting modern humans to study cognitive thus concluded that the acquisition of Palaeolithic stone tool abilities of extinct hominin species (e.g. Putt et al. 2017: 1). manufacture entails structural remodeling of inferior frontoparietal areas. Stout et al. (2008) further report from their 3.1 Cultural transmission studies PET study of Oldowan flaking and Acheulean handaxe To our knowledge only five such studies have been conducted manufacture by expert subjects that Oldowan and Acheulean while interpreting the results has been challenging. All have manufacture shared increased activation compared to the control included subjects in the earliest learning stages. Regarding condition in specific bilateral parietal clusters in the superior Oldowan, Morgan et al. (2015) compared transmission efficiency parietal lobule, IPS and SMG, and additionally in BA 17 and 18 in during flaking acquisition across five communication conditions. the occipital lobe. Stout et al. (2011) conducted an fMRI study in They found i.a. that the transmission improved with gestural and which subjects with different degrees of experience in Palaeolithic verbal teaching relative to imitation/eumlation as seen e.g. in flake toolmaking viewed 20-second clips of Oldowan flaking and quality and the number of viable flakes produced. Furthermore, Acheulean handaxe making. Compared to the control condition they found that verbal teaching improves performance relative to increased activation during observation of ESA toolmaking was gestural teaching. Morgan and colleagues argued that ESA yet again documented in the occipital and inferior parietal areas, technologies possibly generated selection for increasingly and the precentral gyri, and was further found in the inferior complex transmission modes. These results were in some measure temporal cortices and the right Broca’s area (BA 44 and 45). replicated by Lombao et al. (2017) who compared the efficiency These results suggest that early Palaeolithic toolmaking relies of imitation/emulation, and gestural and verbal teaching in the largely on motor and visuospatial processing, but two studies have acquisition of the alternating flaking method. They reported that 83 also recorded increased activity in the prefrontal cortex suggesting 4. CONCLUSION the involvement of hierarchical and sequential action processing. Behavioral studies tentatively demonstrate that language was not a Oldowan toolmaking has been described as involving mainly the prerequisite for early stone tool transmission. However, they frontoparietal sensorimotor areas, most notably the vPrCG, SMA suggest that non-linguistic gestures might have been under and IPS, and the cerebellum while it is not associated with selective pressure as early as Oldowan. Neuroimaging studies prefrontal activity (Stout 2000, 2007). Studies have concluded that show that Acheulean compared to Oldowan activates prefrontal Oldowan toolmaking relies mostly on motor and visuospatial areas suggesting that Acheulean possibly requires significantly processing with no apparent role of e.g. executive functioning, more executive control. While the prefrontal activity is mostly suggestive of more “ape-like” cognitive abilities (Putt et al. 2017: limited to BA 44 and 45 (Broca’s area), the nature of the 4). It is here notable that the parietal cortex has witnessed a coevolution of toolmaking and language remains largely significant enlargement during hominin evolution and it is unsettled. Future research should include later hominin suggested that the most pronounced changes occurred around the technologies and seek to determine the particular cognitive IPS as evidenced by an inferior displacement of the lower parietal functions associated with Acheulean- and Mousterian-related areas (Bruner 2010). Furthermore, changes in cerebellum size prefrontal activity. Further accumulation of research will have been noted as well in a computational study comparing hopefully reveal new methodological possibilities in language Neanderthal and modern human brains (Kochiyama et al. 2018). evolution research. Stout et al. (2008) found higher activation during Acheulean 5. REFERENCES relative to Oldowan toolmaking bilaterally in the vPMC, inferior [1] Botha, R. (2009). “Theoretical underpinnings of inferences parietal areas and the right Broca’s area. Furthermore, Uomini and about language evolution: the syntax used at Blombos Cave”. Meyer (2013) conducted an fTCD study with expert subjects In: Botha, R., Knight, C. (eds.). The Cradle of Language. engaged in the Acheulean handaxe production and silent Oxford [etc.]: Oxford University Press, 93–111. phonemic fluency. They noted high correlations between the [2] Botha, R. (2012). “Inferring modern language from ancient hemodynamic lateralization patterns during the two tasks. It has objects”. In: Tallerman, M., Gibson, K. R. (eds.). The Oxford been argued that there is a strong evolutionary connection Handbook of Language Evolution. Oxford [etc.]: Oxford between functional lateralization, as seen e.g. in bimanual University Press, 303–12. coordination, and language (e.g. Uomini 2015). However, Putt et al. (2017) reported from their fNIRS study that subjects acquiring [3] Bruner, E. (2010). “Morphological differences in the parietal Acheulean handaxe manufacture in a verbal teaching condition lobes within the human genus”. Curr Anthropol, 51, S77–88. had increased activation in the right pars triangularis compared to [4] Cataldo, D.M., Migliano, A.B., Vinicius, L. (2018). “Speech, the non-verbal condition. These results cast potential doubt on the stone tool-making and the evolution of language”. PLoS validity of previous results regarding Broca’s area. However, ONE, 13(1), e0191071. comparisons between the toolmaking and control conditions were not reported. Furthermore, the study included only subjects in [5] d’Errico, F., Henshilwood, C., Lawson, G., Vanhaeren, M., their early learning phases. Still and all, it has been proposed that Tillier, A.-M., Soressi, M., Bresson, F., Maureille, B., Broca’s area poses a possible connection between the evolution of Nowell, A., Lakarra, J., Backwell, L., Julien, M. (2003). toolmaking and language because of its prominent role in “Archaeological evidence for the emergence of language, schematic body representation as well as sequential and symbolism, and music – an alternative multidisciplinary hierarchical goal-directed action processing (e.g. Ruck 2014). perspective”. J World Prehist, 17(1), 1–70. Additionally, Kemmerer (2012) suggested that the cross- [6] Diez-Martín, F., Sánchez Yustos, P., Uribelarrea, D., linguistically most prevalent word orders (SOV and SVO) reflect Baquedano, E., Mark, D. F., Mabulla, A., Fraile, C., Duque, the ways Broca’s area processes actions and/or events. Moreover, J., Díaz, I., Pérez-Gonzáles, A., Yravedra, J., Egeland, C. P., Putt et al. (2017) found bilateral temporal Acheulean-related Organista, E., Domínguez-Rodrigo, M. (2015). “The origin activity which they associated with auditory working memory. of the Acheulean: the 1.7 Million-year-old site of FLK West, Finally, Putt and Wijeakumar (2018) conclude based on their Olduvai Gorge (Tanzania)”. Sci Rep, 5, 17839. earlier study that Acheulean handaxe manufacture and modern [7] Diogo, R. (2018). “First detailed anatomical study of language rely on different components of working memory, with bonobos reveals intra-specific variations and exposes just-so Acheulean recruiting visual and auditory working memory stories of human evolution, bipedalism, and tool use”. Front components not typically related with modern human linguistic Ecol Evol, 6, 53. processing. However, this auditory component might represent a precursor to modern verbal working memory. A further point of [8] Garofoli, D. (2014). “Do early body ornaments prove convergence might have been the vPMC (Putt and Wijeakumar cognitive modernity? A critical analysis from situated 2018: 282). cognition”. Phenomenol Cogn Sci, 14(4), 803–25. Miura et al. (2014) conducted an fMRI study comparing [9] Hamrick, P., Lum, J. A.G., Ullman, M. T. (2018). “Child observation of videos of Mousterian toolmaking and a man first language and adult second language are both tied to pronouncing Uzbek words, a language unfamiliar to the subjects. general-purpose learning systems”. Proc Natl Acad Sci U S Among others, increased activation during Mousterian A, 115, 1487–92. observation relative to the perception of Uzbek words was found [10] Harmand, S., Lewis, J.E., Feibel, C.S., Lepre, C.J., Prat, S., in the right pars opercularis (part of Broca’s area) and bilaterally Lenoble, A., Boës, X., Quinn, R.L., Brenet, M., Arroyo, A., in BA 6 (entailing the PMC and SMA). Taylor, N., Clément, S., Daver, G., Brugal, J.-P., Leakey, L., Mortlock, R.A., Wright, J.D., Lokorodi, S., Kirwa, C., Kent, D.V., Roche, H. (2015). “3.3-million-year-old stone tools 84 from Lomekwi 3, West Turkana, Kenya”. Nature, 521, 310– [24] Ohnuma, K., Aoki, K., Akazawa, T. (1997). “Transmission 5. of tool-making through verbal and non-verbal [11] Haslam, M., Hernandez-Aguilar, R.A., Proffitt, T., Arroyo, communication: preliminary experiments in Levallois flake A., Falótico, T., Fragaszy, D., Gumert, M., Harris, J.W.K., production”. Anthropol. Sci, 105(3), 159–68. Huffman, M.A., Kalan, A.K., Malaivijitnond, S., Matsuzawa, [25] Pulvermüller, F. (2013). “How neurons make meaning: brain T., McGrew, W., Ottoni, E.B., Pascual-Garrido, A., Piel, A., mechanisms for embodied and abstract-symbolic semantics”. Pruetz, J., Schuppli, C., Stewart, F., Tan, A., Visalberghi, E., Trends Cogn Sci, 17(9), 458–70. Luncz, L.V. (2017). “Primate archaeology evolves”. Nat [26] Putt, S.S., Wijeakumar, S. (2018). “Tracing the evolutionary Ecol Evol, 1, 1431–7. trajectory of verbal working memory with neuro- [12] Hecht, E.E., Gutman, D.A., Khreisheh, N., Taylor, S.V., archaeology”. Interact Stud, 19(1–2), 272–88. Kilner, J., Faisal, A.A., Bradley, B.A., Chaminade, T., Stout, [27] Putt, S.S., Wijeakumar, S., Franciscus, R.G., Spencer, J.P. D. (2014). “Acquisition of Paleolithic toolmaking abilities (2017). “The functional brain networks that underlie Early involves structural remodeling to inferior frontoparietal Stone Age tool manufacture”. Nat Hum Behav, 1, 0102. regions”. Brain Struct Funct, 220(4), 2315–31. [28] Putt, S.S., Woods, A.D., Franciscus, R.G. (2014). “The role [13] Hovers, E. (2015). “Archaeology: tools go back in time”. of verbal interaction during experimental bifacial stone tool Nature, 521, 294–5. manufacture”. Lithic Technology, 39(2), 96–112. [14] Janković, I., Šojer, T. (2015). “The evolution of speech and [29] Ruck, L. (2014). “Manual praxis in stone tool manufacture: language.” Opvscvla archaeologica, 37/38, 11–48. implications for language evolution”. Brain Lang, 139, 68– [15] Karavanić, I., Janković, I. (2009). Osvit čovječanstva: 83. početci našega biološkog i kulturnog razvoja. Zagreb: [30] Schlanger, N. (1996). “Understanding Levallois: lithic Školska knjiga. technology and cognitive archaeology”. Cambridge [16] Kemmerer, D. (2012). “The cross-linguistic prevalence of Archaeological Journal, 6(2), 231–54. SOV and SVO word order reflects the sequential and [31] Semaw, S. (2000). “The world’s oldest stone artefacts from hierarchical representation of action in Broca’s area”. Lang Gona, Ethiopia: their implications for understanding stone Linguist Compass, 6(1), 50–66. technology and patterns of human evolution between 2·6–1·5 [17] Kochiyama, T., Ogihara, N., Tanabe, H.C., Kondo, O., million years ago”. J Archaeol Sci, 27, 1197–214. Amano, H., Hasegawa, K., Suzuki, H., León, de M.S.P., [32] Stout, D., Chaminade, T. (2007). “The evolutionary Zollikofer, C.P.E., Bastir, M., Stringer, C., Sadato, N., neuroscience of tool making”. Neuropsychologia, 45, 1091– Akazawa, T. (2018). “Reconstructing the Neanderthal brain 100. using computational anatomy”. Sci Rep, 8, 6296. [33] Stout, D., Hecht, E. (2015). “Neuroarchaeology”. In: Bruner, [18] Kolodny, O., Edelman, S. (2018). “The evolution of the E. (ed.). Human Paleoneurology. Cham: Springer, 145–75. capacity for language: the ecological context and adaptive value of a process of cognitive hijacking”. Phil. Trans. R. [34] Stout, D., Passingham, R., Frith, C., Apel, J., Chaminade, T. Soc. B, 373, 20170052. (2011). “Technology, expertise and social cognition in human evolution”. Eur J Neurosci, 33, 1328–38. [19] Lewis, J.E., Harmand, S. (2016). “An earlier origin for stone tool making: implications for cognitive evolution and the [35] Stout, D., Toth, N., Schick, K., Chaminade, T. (2008). transition to Homo”. Phil. Trans. R. Soc. B, 371: 20150233. “Neural correlates of Early Stone Age toolmaking: technology, language and cognition in human evolution”. [20] Lombao, D., Guardiola, M., Mosquera, M. (2017). “Teaching Phil. Trans. R. Soc. B, 363, 1939–49. to make stone tools: new experimental evidence supporting a technological hypothesis for the origins of language”. Sci [36] Stout, D., Toth, N., Schick, K., Stout, J., Hutchins, G. (2000). Rep, 7, 14394. “Stone tool-making and brain activation: position emission tomography (PET) studies”. J Archaeol Sci, 27, 1215–23. [21] Lotem, A., Halpern, J.Y., Edelman, S., Kolodny, O. (2017). “The evolution of cognitive mechanisms in response to [37] Torre, de la I. (2016). “The origins of the Acheulean: past cultural innovations”. Proc Natl Acad Sci U S A, 114(30), and present perspectives on a major transition in human 7915–22. evolution”. Phil. Trans. R. Soc. B, 371, 20150245. [22] Miura, N., Nagai, K., Yamazaki, M., Yoshida, Y., Tanabe, [38] Toth, N. (1987). “The first technology”. Scientific American, H.C., Akazawa, T., Sadato, N. (2014). “Brain activation 256(4), 104–13. related to the imitative learning of bodily actions observed [39] Toth, N., Schick, K. (2018). “An overview of the cognitive during the construction of a Mousterian stone tool: a implications of the Oldowan Industrial Complex”. Azania, functional magnetic resonance imaging study”. In: Akazawa, 53(1), 3–39. T. et al. (eds.). Dynamics of Learning in Neanderthals and Modern Humans. Volume 2: Cognitive and Physical [40] Uomini, N.T. (2015). “Paleoneurology and behaviour”. In: Perspectives. Tokyo [etc.]: Springer, 221–32. Bruner, E. (ed.). Human Paleoneurology. Cham: Springer, 121–43. [23] Morgan, T.J.H., Uomini, N.T., Rendell, L.E., Chouinard- Thuly, L., Street, S.E., Lewis, H.M., Cross, C.P., Evans, C., [41] Uomini, N.T., Meyer, G.F. (2013). “Shared brain Kearney, R., Torre, I. de la, Whiten, A., Laland, K.N. (2015). lateralization patterns in language and Acheulean stone tool “Experimental evidence for the co-evolution of hominin tool- production: a functional transcranial doppler ultrasound making teaching and language”. Nat Commun, 6, 6029. study”. PLoS ONE, 8(8), e72693. 85 [42] Villmoare, B., Kimbel, W.H., Seyoum, C., Campisano, C.J., DiMaggio, E.N., Rowan, J., Braun, D.R., Arrowsmith, J.R., Reed, K.E. (2015). “Early Homo at 2.8 Ma from Ledi- Geraru, Afar, Ethiopia”. Science, 347(6228), 1352–5. 86 Psihonevrobiološka razvojna perspektiva razumevanja nasilja izr. prof. dr. Barbara Simonič Teološka fakulteta Katedra za zakonsko in družinsko terapijo ter psihologijo in sociologijo religije Poljanska c. 4, 1000 Ljubljana, Slovenija Tel: +386 1 4345810; fax: +386 1 4345854 e-mail: barbara.simonic@teof.uni-lj.si POVZETEK1 2. PROBLEMI DEFINICIJ AGRESIVNOSTI Kompleksnost nasilja se kaže v različnih pojavnih oblikah, vzročnosti, vzorcih, posledicah in kontekstih. Zaradi te kompleksnosti se IN NASILJA pojavljajo tudi težave pri preventivi in celoviti obravnavi nasilja. V Nasilje oz. agresivno vedenje se običajno smatra za nezaželeno, prispevku so prikazane nekatere dileme pri definiciji in razumevanju čeprav so pri tem nekatere izjeme glede na to, kakšno vedenje se v nasilja, s poudarkom na psihonevrobiološki perspektivi razvoja pa je določeni situaciji pojavlja. V določenih okvirih se namreč agresivno prikazana ena izmed možnosti razumevanja agresivnega vedenja oz. vedenje lahko smatra kot zaželeno (npr. samoobramba). Zaradi teh nasilja, ki lahko služi kot izhodišče za dolgoročno preprečevanje različnih okoliščin in vzrokov (motivacija, učinki, dejanja, psihološki, nasilja. socialni in politični pomen), v katerih se agresivno vedenje pojavlja, imamo o nasilju različne definicije. Ta različnost ustvarja tudi Ključne besede neenotnost pri razumevanju tega, kako se nasilje razume, kateri vzorci agresivnost, nasilje, splošni model agresivnosti, psihonevrobiologija, vedenja so prepoznani kot nasilni, kako so povezani faktorji tveganja razvoj, regulacija afekta ter tudi katere intervencije in politike so primerne pri preprečevanju nasilja. Pomanjkanje soglasja je ovira za usklajevanje raziskovanja, načrtovanja in oblikovanja učinkovitih ukrepov za preprečevanje in 1. UVOD obravnavo različnih oblik nasilja, posledice tega pa se odražajo v Agresivnost (nasilje)2 je kompleksen pojav, ki ga srečamo v različnih splošnem blagostanju družbe. Skoraj vsakdo lahko prepozna, ali je v pojavnih oblikah in v različnih dimenzijah družbene in osebne določeni situaciji neko dejanje ali situacija nasilna. Težje pa je v stvarnosti. Eden od razlogov, zakaj v določenih primerih ostaja ta splošnem določiti, kaj je nasilno in kaj ni [4]. S tega stališča je kljub pojav prezrt, je pomanjkanje jasne definicije, ki ni strogo znanstvena, občutku jasnosti, kaj nasilje je, to vseeno dvoumen pojav, kar se ampak je velikokrat tudi stvar presoje, ki je pogojena tudi s kulturo, pokaže pri poskusu splošnega označevanja in kategorizacije njegovih vrednotami in družbenimi normami trenutnega obstoja [1], vse to pa pojavov, logike pojavljanja in stopnjevanja, iskanju vzročnih razlag in predstavlja tudi težavo pri načrtovanju ukrepov preventive nasilja. njegovega vrednotenja, pri tem pa lahko nevarno zaidemo na področje Najbolj osnovne definicije nasilje opredeljujejo kot vedenje, ki je zagovarjanja pravičnosti nasilja [5]. namerno usmerjeno k povzročitvi škode in ki dejansko povzroči fizično ali psihično škodo. Svetovna zdravstvena organizacija [2] Kljub temu, da obstaja splošno soglasje o tem, da nasilje deluje opredeljuje nasilje kot namerno uporabo fizične sile ali moči, ki uničujoče in da za sabo vedno pušča žrtve, pa vseeno ostajajo vrzeli obstaja v obliki grožnje ali pa v dejanjih zoper samega sebe, drugo nedoločenosti, saj splošne definicije ne povedo, kdo ali kaj je bilo osebo ali proti skupini ali skupnosti, za posledico pa ima visoko poškodovano in kako resna je ta poškodba (npr. ali je lahko neko verjetnost, da povzroči telesno poškodbo, smrt, psihično škodo, zastoj agresivno vedenje zaradi težke psihiatrične diagnoze smatrano kot v rasti (razvoju) ali prikrajšanje (odvzem za razvoj potrebnih stvari). nasilje?). Učinkov nasilja, glede na kontekst, ni mogoče vedno jasno Zaradi škode, ki jo povzroča nasilje, je seveda primarnega pomena označiti kot pozitivne ali negativne [6]. Zdi se, da vsak poskus razmišljati o preventivi nasilja, pri čemer pa je primarnega pomena dokončnega določanja, kaj je nasilje, vedno nekomu dela krivico (ali razumevanje vzročnosti agresivnega vedenja, česar pa definicije osebi, ki je agresivna, ali pa žrtvi). Kar pa lahko z gotovostjo rečemo večinoma ne opredeljujejo. Pri tem so lahko v pomoč številni pogledi, pri vseh poskusih definicije nasilja pa je, da se pri vseh pojavlja teorije in modeli, ki na različne, a sorodne načine razlagajo etiologijo osnovno načelo prekoračitve meja – določiti, kaj pa so meje, pa se v nasilja [3]. dobi, ko se pojavlja opuščanje in relativizacija moralnih, spolnih, izobraževalnih, pravnih standardov, zdi spet zelo težka naloga [5]. 1 Doseženi rezultati so delno nastali v okviru projekta št. J5-9349, ki ga je pojasnjena. V prispevku tako uporabljamo izraza 'nasilje' in 'agresivno financirala Javna agencija za raziskovalno dejavnost Republike Slovenije iz vedenje' kot sinonima, ki sta odraz zunanje manifestacije državnega proračuna. 'agresivnosti', ki pa označuje bolj notranje stanje oz. notranjo napetost 2 V pričujočem prispevku so izmenično uporabljeni različni izrazi, ki so v posamezniku in se nujno ne manifestira v zunanjem 'agresivnem velikokrat (po navedbah različnih avtorjev), razumljeni kot sinonimi: vedenju' oz. 'nasilju'. 'nasilje', 'agresivnost', 'agresivno vedenje'. V definicijah se pojma 'nasilje' in 'agresivnost' praviloma istočasno pojavljata in nista vedno razmejena ali 87 3. MODELI AGRESIVNOSTI IN zniževanje možnosti agresivnosti in agresivnega vedenja, saj vplivajo AGRESIVNEGA VEDENJA na kognicije, afekte in stopnjo telesnega vznemirjenja. V splošnem je tako, da več kot je dejavnikov tveganja za agresivnost, večja je Zaradi omenjene problematike se kot izhodišče razumevanja in možnost agresivnosti in agresivnega vedenja [10]. načrtovanja ukrepov za preprečevanja nasilja zdi bolj smiselno izhajati iz celostnih perspektiv razumevanja, kakršne predstavljajo V drugi fazi se torej odvija vpliv vnosov (osebnih in situacijskih različni modeli in ki vključujejo različne vzročne dejavnike tveganja dejavnikov) na poti, ki vodijo v procese ocenjevanja in odločanja za nasilje. Svetovna zdravstvena organizacija je npr. za razumevanje (agresivne ali neagresivne rezultate). Osebni in situacijski dejavniki kompleksnosti narave nasilja izoblikovala t.i. ekološki model (angl. lahko spremenijo afektivno stanje osebe, kognicije in stopnjo ecological) model, ki je dinamičen in odprt nenehnim dopolnitvam, telesnega vzburjenja. Vsi ti trije dejavniki oblikujejo trenutno notranje saj upošteva interakcijo in spreminjanje različnih dejavnikov, ki stanje osebe in so med seboj prepleteni. Npr. afektivno stanje lahko vplivajo na povečanje tveganja za izvajanje nasilja, in dejavnikov, ki vpliva na spoznavne procese in stopnjo fizične vznemirjenosti [11]. rizično vplivajo na to, da je nekdo žrtev nasilja. Ti dejavniki so Jeza npr. lahko spodbudi nasilno razmišljanje in poveča telesno razvrščeni na štiri ravni. Prva raven obravnava zgodovinske biološke vznemirjenost. Prav tako lahko spoznavni procesi in vznemirjenje in osebne dejavnike, ki vplivajo na ravnanje oseb in povečujejo vplivajo na afektivno stanje (npr. če interpretiramo situacijo kot možnost, da oseba postane bodisi nasilnež bodisi žrtev nasilja. Druga nasilno, se lahko poveča jeza, kar vodi v višjo stopnjo telesnega raven se omejuje na obravnavo bližnjih odnosov (npr. odnosi v vznemirjenja) [8]. družini, s prijatelji, intimnimi partnerji in ljudmi istega stanu) in raziskuje, kako ti odnosi povečujejo možnost, da oseba postane bodisi nasilnež bodisi žrtev nasilja. Tretja raven se osredotoča na organizirane skupnosti, v katerih potekajo družbene interakcije (npr. šole, delovna mesta, soseske) in išče značilnosti, ki povečujejo tveganje za izbruh nasilja. Četrta raven pa proučuje širše družbene dejavnike, ki oblikujejo klimo, ki nasilje bodisi spodbuja bodisi ga zavira [2]. Ekološki model razjasnjuje vzroke nasilja kakor tudi njihovo kompleksno vzajemno delovanje, zato predlaga, kaj je treba na različnih ravneh simultano storiti, da bi se nasilje preprečilo. Temu podoben je splošni model agresivnosti (angl. General Aggression Model – GAM), ki predstavlja celovit in integrativen model za razumevanje agresivnosti in agresivnega vedenja, nudi pa tudi razmislek o ključnih vidikih intervencij, ki bi lahko vodile k preprečevanju nasilnih in uničujočih odzivov [7]. Splošni model agresivnosti predpostavlja, da na človekovo agresivnost močno vplivajo zaznavne strukture, ki zaznamujejo veliko socialno-kognitivnih fenomenov, kot so percepcija, odločanje in vedenje. Nekatere teh najpomembnejših struktur vključujejo tudi prepričanja, sheme dojemanja, sheme pričakovanj in vedenjske skripte. Gre za strukture, ki se razvijajo preko izkušenj in lahko vplivajo na dojemanja na različnih nivojih, od preprostega dojemanja predmetov do kompleksnega dojemanja socialnih dogodkov [8]. Slika 1: Splošni model agresivnosti: bližnji in oddaljeni vzroki in procesi [12] . Splošni model agresivnosti (slika 1) v razlaganju agresivnosti združuje dve skupini procesov: 1. bližnji procesi (angl. proximate Skupaj te poti ustvarijo notranje stanje, ki poveča verjetnost izida processes) so tisti, ki so neposredno povezani z agresivnim vedenjem agresivnega vedenja [13]. To je tretja faza, kjer govorimo o rezultatih v danem trenutku; 2. oddaljeni procesi (angl. distal processes) pa so (procesi vrednotenja oz. ocene, ki so lahko tudi avtomatski, procesi tisti, ki vplivajo na bližnje procese preko globljih tendenc in struktur odločitve in agresivno ali neagresivno vedenje). V tej fazi torej oseba (gre za aspekt, kako biološki in okoljski dejavniki sodelujejo in ovrednoti situacijo in se odloči, kako se bo odzvala. Aktivnost, za vplivajo na osebnost osebe) [9]. V teh procesih pa splošni model katero se odloči, vpliva na socialne stike, ki povratno vplivajo na agresivnosti izpostavlja tri glavne faze oz. elemente interakcij med osebne in situacijske dejavnike, kar vodi v nov začetek cikla oz. osebo in situacijo, ki se pojavljajo kot 1. vnosi (angl. inputs), 2. poti epizode agresivnosti. Vsako ponavljanje epizode agresivnosti (ali (angl. routes) in rezultati (angl. outcomes) [8]. Prva faza so vnosi, kjer neagresivnosti) služi kot učni proces, ki lahko vpliva na razvoj gre za dejavnike dveh vrst. Prvi so povezani z osebo ( osebni dejavniki, agresivnih zaznavnih struktur (in tako osebnosti) skozi čas [8]. kot so spol, prepričanja, osebne lastnosti, vrednote, dolgoročni cilji, razpoloženje...) in predstavljajo stanje, v katerem se poveča možnost za agresivnost. Drugi dejavniki so povezani s situacijo ( situacijski 4. PSIHONEVROBIOLOGIJA RAZVOJA dejavniki, kot so provokacija, socialni stres, socialno zavračanje, KOT DEJAVNIK AGRESIVNOSTI frustracija, alkoholiziranost, nasilnost medijev, bolečina ali neudobje, V zgoraj omenjenih dimenzijah agresivnosti in nasilja, ki jih visoka temperatura, hrup, prisotnost orožja, ogrožajoči ali strah opredeljujejo različni modeli in definicije, se zdi, da so nekateri vzbujajoči dražljaji...), v kateri se poveča možnost za agresivne dejavniki tveganja za nasilno vedenje bolj dostopni posameznikovi reakcije [9]. Posamezni dejavniki ali kombinacija osebnih in kontroli, drugi pa manj. Predvsem zaskrbljujoči so slednji, saj le na te situacijskih dejavnikov predstavlja izhodišče za povečevanje ali posameznik težko neposredno vpliva, bodisi ker so vkoreninjeni v 88 splošne sheme prepričanja na družbeni ravni, ali pa ker predstavljajo na sam razvojni proces in tudi strukturo osebnosti posameznika [19]. nezavedne sfere njegove osebnosti. Glede na splošni model Travmatični odnosi in čustveno zanemarjanje negativno vplivajo na agresivnosti gre pri tem za oddaljene procese, kamor se uvrščajo strukturo in funkcijo desne hemisfere, subkortične strukture in biološki dejavniki (npr. ADHD, hormonsko neravnovesje, frontalni korteks [18]. Desna možganska hemisfera in subkortične hipersenzitivnost itd.) in okoljski dejavniki (npr. kulturne norme, ki strukture so temeljnega pomena v čustvenem procesiranju, podpirajo nasilje, nefunkcionalne družine in starševstvo, težki prefrontalni korteks pa igra pomembno vlogo pri regulaciji afekta življenjski pogoji, nasilne soseske in vrstniške skupine, nasilni mediji [20]. Tako ni čudno, da ti ljudje tudi kot odrasli trpijo posledice, ki itd.), ki vplivajo na osebnost posameznika [14]. Predvsem pri uničujoče delujejo v njihovem življenju in v življenjih njihovih okoljskih dejavnikih, kjer so vključeni odnosi z bližnjimi in bližnjih. V poznejših življenjskih obdobjih so otroci z varno predstavljajo okolje, v katerem se posameznik razvija, se zdi smiselno navezanostjo v večji meri sposobni razviti in vzdrževati dobre bližnje iskati tudi možnosti interveniranja, ki bi dolgoročno pripomoglo k odnose, zlasti s starši, vrstniki in partnerji kot ne-varno navezani preprečevanju kasnejšega agresivnega vedenja. otroci. Pri varno navezanih se v otroštvu, najstništvu in v odraslosti razvijejo različne bolj zaželene osebnostne lastnosti, pri njih je večja Zgodnje izkušnje s starši vplivajo na razvoj organske podlage naše verjetnost prisotnosti konstruktivnih oblik čustvovanja in čustvene osebnosti, in sicer se zapišejo v možgane, ki so dogovorni za samoregulacije, izražajo bolj pozitivno samospoštovanje [21]. procesiranje zavednih in nezavednih informacij skozi življenje. Med Različne motnje v odraslosti imajo svoje korenine v otroštvu, npr. nosečnostjo in v prvih dveh letih življenja so vzpostavljeni temelji motnje hranjenja, zloraba substanc, preveč impulzivno vedenje, čustvenega delovanja in čustvenih virov odraslega človeka. Takrat se pomanjkanje empatije, depresivnost, tesnobnost, agresivnost itd., kar oblikujejo »socialni možgani«, del možganov, ki se uči, kako je vse posledica tega, da zaradi slabe razvitosti čustvene regulacije obvladovati občutke v odnosu z drugimi ljudmi in v različnih posamezniki niso sposobni primernega upravljanja s svojimi občutki situacijah. Primarni odnos otroka, zlasti z materjo (ali drugim [16]. V povezavi s tem je pomembno poudariti, da so posamezniki, ki primarnim skrbnikom), tako zagotavlja osnovo za to, kako se bodo niso sposobni primerne regulacije afektov, kot je jeza in druga vzpostavili procesi regulacije afekta v otrokovih možganih. To bo tudi afektivna stanja, bolj nagnjena k agresivnemu vedenju, da bi s tem določilo posameznikovo večjo ali manjšo sposobnost za vstop v spremenili, odstranili ali se izognili neprijetnim čustvenim stanjem čustvena in socialna razmerja in situacije pozneje v življenju. Odnos (vzniku agresivnega občutja) [22]. Gre za stanje t.i. afektivne med otrokom in odraslim tako zagotavlja prostor za razvoj otroka [15]. disregulacije, o kateri govorimo, ko oseba ni sposobna primerno občutiti čustev, ko jo čustva prevzamejo ali ko ostanejo nerazrešena Ob rojstvu je otrok še »nedokončan«. Čeprav ima vse anatomske dele, [23]. pa je kot »surov material« z genetskim načrtom in edinstveno paleto možnosti, ki se morajo na določen način še razvijati skozi izkušnje, zlasti socialne [16]. Izkušnje v zgodnjem življenju so lahko še posebej Z vidika psihonevrobiološke perspektive razvoja kakovost odnosov s pomembne pri organiziranju načina razvijanja osnovnih regulativnih starši v zgodnjem otroštvu vpliva na razvoj možganskih področij, ki struktur v možganih. Vsakodnevne izkušnje tako oblikujejo strukturo so odgovorna za različne vrste regulativnih strategij. Šibko razvit možganov, ker aktivirajo določene poti v možganih, krepijo obstoječe prefrontalni korteks je značilen pri številnih psihopatoloških in povezave in ustvarjajo nove. Psihonevrobiološka perspektiva razvoja negativnih vedenjskih strategijah. Brez močnega prefrontalnega poudarja, da je zgodnje otroštvo obdobje, ko je določena osnovna korteksa mehanizmi samokontrole, pomirjanja sebe in občutka »arhitektura« v možganih [17]. povezanosti z drugimi, ne dozorijo [16]. V nezmožnosti uravnavanja močnih čustev zaradi nerazvitih procesov regulacije je tako mogoče Zgodnje socialno okolje (odnosi z primarnimi skrbniki) tako prepoznati vir nasilja [24]. Do tega pride zato, ker nedozoreli neposredno vplivajo na končno vzpostavljenost krogotokov v orbitofrontalni sistem ni sposoben regulirati impulzov (tudi možganih otroka, ki so odgovorni za prihodnje družbeno in čustveno agresivnih) iz nižjih limbičnih struktur, kar povečuje možnost življenje. Odnosi navezanosti neposredno oblikujejo zorenje otrokove agresivnega vedenja, ki predstavlja neprimerno regulacijo afektivnih desne hemisfere, posebej sisteme prefrontalnega dela desne hemisfere, stanj, ki vzniknejo v določeni situaciji (jeza, sram, strah...) [18]. ki so odgovorni za regulacijo čustev, vključno s pozitivnimi čustvi (npr. veselje in zanimanje), pa tudi negativnih čustev, kot sta strah in 5. SKLEP agresija [18]. Vzorci varne navezanosti vodijo tako v organiziranost Sposobnost regulacije čustev (tudi regulacija agresivnih impulzov) od primernega regulatornega sistema v prefrontalnih območjih desne otroštva naprej služi kot zaščitni faktor pred agresivnim vedenjem hemisfere, učinkovito delovanje v tem delu možganov pa je posameznika. Čeprav se ta sposobnost lahko razvija in dopolnjuje še osrednjega pomena za otrokovo naraščajočo sposobnost kasneje, se v prvih dveh letih življenja oblikujejo osnovni vzorci za samoregulacije, sposobnosti fleksibilnega uravnavanja stresnih soočanje z negativnimi in težavnimi čustvenimi situacijami. Človeška čustvenih stanj zaradi interakcij z drugimi ljudmi in samoregulacije v bitja so odprti sistemi, ki jih oblikujejo drugi ljudje. Človeški avtonomnih kontekstih. Gre za način, kako varni odnosi navezanosti psihološki in duševni sistemi se razvijajo v odnosu z drugimi ljudmi - (epizode nežnosti, uglašenosti, sinhronije) spodbujajo razvoj desne in to se najbolj intenzivno dogaja v otroštvu. Lahko rečemo, da je možganske hemisfere, pospešujejo učinkovito regulacijo afekta in človeški otrok najbolj družbeno vplivano bitje (ali biološki sistem) na gradijo temelje otrokovega psihičnega zdravja. Skrbnikova zemlji [16]. Biološki in socialni dejavniki so v človekovem razvoju senzitivnost je tako zunanji organizator otrokove biološke podlage zelo prepleteni in povezani, še posebej v obdobju zgodnjega razvoja. vedenjske regulacije. Na podlagi tega se v določenem trenutku razvije Seveda imajo genetika in okoljski dejavniki vpliv na dojenčke, vendar tudi samoregulacija otroka, ko se v relacijskem okolju dogradijo so socialni dejavniki, zlasti senzitivna nega in skrb skrbnika tisti, ki notranji regulatorni mehanizmi [18]. vplivajo in dejansko spremenijo biološke elemente (kot sta kemija možganov in možgansko tkivo). Zelo pomembno je, da ne V primeru slabih relacijskih okvirov razvoja lahko govorimo o zanemarjamo razmerja med biologijo, izkušnjami in vedenjem, saj je relacijski travmi, ki v nevrobiološkem delovanju otroka pušča povsem to lahko škodljivo in uničujoče za našo celotno družbo [24]. Zato je v drugačne vzorce, saj negativno vpliva na od izkušenj odvisne razvojne povezavi s tem bistveno razmisliti in razviti ustrezne programe za procese v delovanju možganov. To pušča dolgoročen negativen vpliv zgodnje ukrepanje, ki bi lahko delovali tudi kot zaščitni dejavnik proti 89 tveganju agresivnega vedenja in nasilja. Tovrstno ukrepanje, ki bi [14] Anderson, C. A. in Carnagey, N. L. (2004). Violent evil and the spodbujalo zdrave odnose med primarnimi skrbniki in otrokom General Aggression Model. V: A. G. Miller (ur.), The Social (preprečevanje relacijske travme in spodbujanje učinkovitega in Psychology of Good and Evil (str. 168-192). New York: Guilford senzitivnega starševstva), se mora začeti še pred otrokovim rojstvom Press. in ga je treba podaljšati skozi perinatalno in postnatalno obdobje. [15] Simonič, B. in Poljanec, A. (2014). Building Motherhood in the Programi za učinkovito starševstvo (na katerem koli področju - Young Mothers' Group. Child Care in Practice, 20(3), 270-285. pediatrija, psihiatrija, psihologija, socialno delo, izobraževanje, pastoralna oskrba ...) lahko ustvarijo razvojni kontekst za [16] Gerhard, S. (2004). Why Love matters. London: Routledge. preoblikovanje ne-varne navezanosti v varno in na ta način spodbudijo [17] Siegel, D. J. (2012). The Developing Mind. New York: Norton. od izkušenj odvisno nevrobiološko zorenje desne hemisfere, ki je ključnega pomena pri primerni regulaciji različnih motivacijskih [18] Schore, A. N. (2003). Early relational trauma, disorganized (afektivnih) stanj, vključno z agresivnimi [18]. Dejstva, ki jih je attachment, and the development of a predisposition to violence. razkrila (nevro)znanost v preteklih desetletjih, so jasna in jih ni V: M. F. Solomon in D. J. Siegel (ur.), Healing trauma (str. 112- mogoče zanikati: obstaja velika priložnost za nekaj, kar bi 113). New York: W. W. Norton & Company. posamezniku omogočilo optimalen začetek, da bo bil kar najbolje [19] Repič Slavič, T. in Gostečnik, C. (2017). Relational Family čustveno opremljen za soočanje z izzivi življenja. Therapy as an Aid Toward Resolving the Trauma of Sexual Abuse in Childhood in the Process of Separation in the Couple 6. REFERENCE Relationship. Journal of Marital and Family Therapy, 43(3), 422-434. [1] Cerar, J. (2017). Nasilje žensk v partnerskem odnosu skozi prizmo relacijske družinske paradigme. Doktorska disertacija. [20] Fosha, D. (2003). Dyadic regulation and experiental work with Teološka fakulteta Univerze v Ljubljani. emotion and relatedness in trauma and disorganized attachment. V: M. F. Solomon in D. J. Siegel (ur.), Healing trauma (str. 221- [2] Svetovna zdravstvena organizacija. (2002). World report on 281). New York: W. W. Norton & Company. violence and health: summary. Geneva: World Health Organization. [21] Thompson, R. A. (2008). Early Attachment and Later Development Familiar Questions, New Answers. V: J. Cassidy [3] Hampton, R. L. (1999). Family Violence. London: Sage. in P. R. Shaver (ur.), Handbook of attachment (str. 348-365). [4] Tolan, P. H. (2007). Understanding Violence. V: D. J. Flannery, New York: Guilford. A. T. Vazsonyi in I. D. Waldman (ur.), The Cambridge [22] Roberton, T., Daffern, M. in Bucks, R. S. (2012). Emotion Handbook of Violent Behavior and Aggression (str. 5-18). regulation and aggression. Aggression and Violent Behavior, Cambridge: Cambridge University Press. 17(1), 72-82. [5] Heitmeyer, W. in Hagan, J. (2003). Violence: The Difficulties of [23] Heller, L. in LaPierre, A. (2012). Healing devlopmental trauma. a Systematic International Review. V: W. Heitmeyer in J. Hagan Berkley: North Atlantic Books. (ur.), International Handbook of violence research (str. 3-11). New York: Kluwer Academic Publishers. [24] Karr-Morse, R. in Wiley, M. S. (2013). Ghosts from the Nursery. New York: Atlantic Monthly Press. [6] Imbusch, P. (2003). The Concept of Violence. V: W. Heitmeyer in J. Hagan (ur.), International Handbook of violence research (str. 13-39). New York: Kluwer Academic Publishers. [7] Baron, R. A. in Byrne, D. (1995). Social psychology: Understanding Human Interaction. New Delhi: Prentice Hall. [8] Allen, J. J, Anderson, C. A. in Bushman, B. J. (2018). The General Aggression Model. Current Opinion in Psychology, 19, 75-80. [9] Anderson, C. A. in Groves, C. (2013). General aggression model. V: M. S. Eastin (ur.), Encyclopedia of Media Violence (str. 182- 187). Los Angeles: Sage. [10] Gentile, D. A. in Bushman, B. J. (2012). Reassessing media violence effects using a risk and resilience approach to understanding aggression. Psychology of Popular Media Culture, 1(3), 138-151. [11] Bower, G. H. (1981). Mood and memory. American Psychologist, 36(2), 129-148. [12] Allen, J. J. in Anderson, C. A. (2017). General Aggression Model. V: P. Roessler, C. A. Hoffner in L. van Zoonen (ur.), International Encyclopedia of Media Effects. Hoboken: Wiley- Blackwell. [13] Bishmi, P. D. (2017). A comparative study on level and direction of aggression in high and low rejection sensitive rejection. Indian Journal of Health and Wellbeing, 8(7), 729-733. 90 Emergence of Visual Consciousness in ADHD Children Katja Zupanič Rawan Y. Fakhreddine Zoltán Nádasdy University of Ljubljana University of Texas at Austin Eötvös Loránd University Maribor Austin, Texas Budapest katja.zupanich@gmail.com rawan-f10@utexas.edu zoltan@utexas.edu ABSTRACT neural correlates of consciousness. While some were occupied with The nature of consciousness has excited the imagination of the search of a common center of the brain responsible for scientists and researches for years. In the past, some were of the conscious awareness, others search for multiple regions or systems opinion that a single part of the brain is important fort he emergence that play an important role in the production of consciousness. of consciousness, while some disagreed, one of the first being the William James was one of the first advocates of this position. He psychologist William James, who claimed that what is important claimed that consciousness is a result of the common effort of the for its emergence is the function of the brain as a whole and that whole brain (Principles of Psychology, 1890/1950, as cited in [3]). parallel function of different parts is key. In our study we focus However, an increasing quantity of empirical evidence shows that mainly on the research of visual consciousness. We want to find out some parts of the brain are much more important than others when whether there is a difference in the emergence of visual it comes to the production and expression of conscious awareness. consciousness between healthy participant (children ages 6-16) and These are mainly the thalamocortical system and the reticular children diagnosed with ADHD. We attempt to measure the system of the brain stem [3]. emergence of visual consciousness using an experimental paradigm designed by dr. Zoltán Nádasdy and his team. In the search for brain regions involved in the creation of consciousness there is a structure that has gathered a lot of attention Keywords – the thalamus. The thalamus is a structure in the midbrain found at visual consciousness, ADHD, visual attention, visual integration the top of the brain stem. In the mid-1800 the idea already appeared that the thalamus is the sensory center of the brain. 1. INTRODUCTION Neuroanatomical examinations of patients showed that thalamic The nature of consciousness has puzzled mankind since the regions in an otherwise unaffected brain caused major sensory beginning of time. What is consciousness, how can our physical dysfunction. These neuroanatomical findings showed that the brain interpret subjective experience, what are qualia? The question thalamus plays a key role in the processing of sensory information also presents itself as in where in the brain does consciousness lie. [4]. Later neuroanatomical studies of the thalamus and its All these are questions that scientists have been posing for years connections to other parts of the brain highlighted its function in and still today. sensory processing and integration. It is important to note that it was discovered that all sensory information travels through the What is consciousness, where is consciousness formed, which thalamus before reaching the corresponding parts of the cortex [3]. neural correlates are responsible for consciousness, all these are questions that scientists and philosophers are occupied with today Francis Crick [1] contributed much to the literature on and were occupied with in the past. The search for rigorously consciousness by suggesting that the TRN (thalamic reticular defined neural correlates has been going on for decades.Some nucleus) and the dorsal thalamus play a key role in consciousness recent studies these days identify the thalamus as a central structure by means of controlling and expressing an internal attentional implicated in conscious awareness [4]. The thalamus works as a searchlight as first suggested by Treisman and colleagues (1991, as relay station or a passageway of sensory information into certain cited in [1]). Crick suggested the TRN produces rapid firing in parts of the cerebral cortex [4], which means that all information thalamic relay neurons to control the brain’s attentional searchlight, we are consciously aware of (except olfactory) meets the thalamus acting on Malsburg synapses to create conjunctions of neurons, at some point. expressed by cell assemblies, which Crick believes to be the neural expression of the attentional searchlight. The study we present in the paper, tests visual awareness using an By researching attentional capacity of the brain at tasks comprising experimental paradigm developed by dr. Nádasdy and colleagues. different types of search, researchers noted that the brain searches The task comprised visual integration of image fragments presented consecutively, focusing on one object at a time, before moving on on a computer screen. The project connects the neuroscientific and to the next [1]. psychological aspects of visual consciousness. In his attempt to explain how the thalamocortical system controls We assume a critical point in time when visual consciousness (of attention, Crick suggested the following mechanism: when sensory recognition of an image as a whole or non-whole) emerges and we signals pass through the TRN on their way to the cortex, this causes posit that this time of emergence is delayed with certain excitation. Crick suggested that the TRN and dorsal thalamus work developmental or neurological (neuropsychological) disorders, together to increase activity at the most active thalamocortical namely in ADHD children. pathways and at the same time reduce activity in less active parts 2. THEORETICAL BASIS of the thalamocortical network. He described this mechanism as “… the heating up of the warmer parts of the thalamus and the As the body of neurological knowledge has dramatically grown in cooling down of the colder parts” [1] (page 4587). the past few decades, so too has grown the interest in and search for 91 3. HYPOTHESIS 4.3 Procedure Our hypothesis is that there is a certain amount of information The test is composed of 100 trials, presented in series on a computer needed to be able to put together image fragments with confidence screen. In every trial, 4 similar images were shown on the screen, (confidence here not referring to the reported level of confidence divided by black borders slightly covering the images (in the shape by subjects). Hence, the decision point to reach a certain level of of a cross). Every trial included eight phases where the borders confidence for a given image is consistent across healthy subjects. between images diminish in 7 steps from thick to very narrow. This point can be standardized for a selected set of pictures and the Lastly, in the 8th phase, it is revealed whether the four fragments test can be validated. Having the image set standardized, the test join into a whole or remain different. should be sensitive to disorders implicated in consciousness, By pressing the “S” key it is meant that the subject believed that the attention, mental focus, and visual integration processes such as presented fragments are part of a continuous picture, pressing “R”, functions compromised in ADHD. We also wish to find out the person believed that the images are part of four different whether our test is appropriate for the detection of such disorders. pictures and pressing “N”, the subject was unsure about the The main research goal is to test the hypothesis and based on this continuity of the presented image fragments. construct a theoretical framework describing the differences at the An exit command was also added to the program in case any emergence of visual consciousness in healthy children and children participant wanted to stop before finishing the task. In this case, the with ADHD. participant had the option of pressing the “escape” key on the 4. THE EXPERIMENT keyboard, which would save their data and immediately leave the program. 4.1 Method Participants sit down in front of a computer screen on which four 4.4 Expected Results very similar pictures appear, divided by black lines. The participant decides whether the images compose one joint picture or whether We call the moment during the task when subjects suddenly see and they are different. They record their decision by pressing a key on decide that the picture fragments are merging an “aha” experience. the computer keyboard. At every keystroke the lines between the We define this moment as the emergence of visual consciousness. images thin until the entire picture is revealed and it becomes clear The moment when different subjects reach this level of visual whether the images form a whole. The pictures include themes from consciousness, we predict, should depend on how effectively they everyday life (houses, cars, landscape, furniture etc.), various kinds process and analyze image fragments. Children diagnosed with of animals and abstract pictures, like pictures of the universe, ADHD may not be able to integrate the fragments of visual microscopic cells etc. information as effectively as children with normal cognitive control do. Hence, with this task we aim to quantify the execution of visual consciousness. In our study, visual consciousness is experimentally tested using a between-subjects design. The independent variable is the 5. ACKNOWLEDGMENTS manipulation of trials. During solving time, the task has seven We would like to thank Osnovna šola Kidričevo and Zavod levels equivalent to every phase of testing. The dependent variable Mavrični bojevnik for their help and support recruiting our is the judgment of participants about the continuity of the images participants, and of course all of the children and parents who were presented in every phase (by pressing a computer key). willing to work with us. 4.2 Progress 6. REFERENCES So far 48 children have taken part in the study. The control group [1] F. Crick. Function of the thalamic reticular complex: the consists of 20 students of the Kidričevo elementary school, (10 searchlight hypothesis. Proceedings of the National Academy female, 10 male), aged between 8 and 9 years. The participants of Sciences, 81(14):4586-4590, 1984. were primary school children that took part in the research study [2] F. Crick, C. Koch. Towards a neurobiological theory of voluntarily with parental or legal guardian consents. There are 28 consciousness. Seminar in the Neurosciences, 2: 263-275, children (27 male, 1 female) in the target group, 8 children age 6- 1990. 8, 15 age 11-14 and 5 age 10-12. [3] G. Edelman. Consciousness: The remembered present. The criterion for inclusion was an age of above 5 years and under Annals of the New York Academy of Sciences, 929(1): 111- 15 years. The final sample of the group of all participants was 122, 2006. planned to encompass at least 50 children (25 tested children in the control group and 25 tested children with a psychiatric ADHD [4] E. G. Jones. The Thalamus. Plenum Press. New York, NY diagnosis). 1985 92 European Legal Regulation of Self-learning Systems Tjaša Zapušek University of Copenhagen Faculty of Law - LL.M. Njalsgade 76, København, Denmark tjasa.zapusek@gmail.com ABSTRACT nature. [Deangelis 2014] On 20 February 1947, Alan Turing had a Modern technology often gives us an impression of a better life, yet lecture at the London Mathematical Society, where he enlightened simultaneously raises new risks that have not been present ever the academic public with the statement that “what we want is a before. This paper will present differences between three different machine that learns from experience”.[Press 2017, 1] Alan Turing ‘decision-making processes’ that occur as a consequence of three concluded his lecture with these words: “Putting the same point different types of systems, namely, AI can be equipped with a differently, the machine must be allowed to have contact with deterministic algorithm, a supervised, or an unsupervised learning human beings in order that it may adapt itself to their standards.” algorithm. Through this systematization, concerns regarding [Turing 1947, 14] foreseeability and certainty of the system’s actions will be In these few decades, since Alan Turing first introduced machine examined. The scope of the thesis is limited to foreseeability learning and algorithms, the latter have become a big and a regarding harm caused by physical machines or products, equipped ubiquitous part of our lives. They are used for data processing, with advanced algorithms. This paradox raises questions regarding calculations and automated reasoning. [Deangelis 2014] the role of scientists in situations when they are facing uncertain risks, and legal scholars when regulating advanced technology. “To make a computer do anything, you have to write a computer program. To write a computer program, you have to tell the Keywords computer step by step, exactly what you want it to do. The self-learning systems, algorithms, strict liability, development risk computer then ‘executes’ the program, following each step defence, product liability, product safety, machine learning mechanically, to accomplish the end goal. When you are telling the computer what to do, you also get to choose how it’s going to do it. 1. INTRODUCTION That’s where computer algorithms come in. The algorithm is the Presence of artificial intelligence (AI) in our everyday life will call basic technique used to get the job done”. [Deangelis 2014, 1] into question the efficiency of existing legal rules. Yet before we Pursuant to Deangelis, the point where this explanation could be focus on regulating liability concerning the harm caused by a slightly adjusted is where the text indicates that ‘you have to tell machine equipped with advanced algorithms we should focus on the computer step by step what you want it to do’. [Deangelis 2014] the question ‘is risk assessment is even possible, can we even Instead of following pre-defined and programmed instructions, predict (with existing scientific knowledge) potential consequences some computer algorithms are designed in a way that enable a of these machines?’ computer to learn on their own, in fact they facilitate machine learning which may result in a creation of unique corrections When policy makers and judicial authorities are confronted with an between obtained data that sometimes ‘produce’ unforeseeable uncertain situation they turn to experts for definite answers and outcomes [Deangelis 2014] Legal concerns that have been conclusive evidences, even though uncertainty and absence of emphasized in this thesis can be summarized with the statement by foreseeability clearly precludes definitiveness and conclusiveness. Dr. Panos Parpas, who is a lecturer at Imperial College London, The following research questions were developed in order to define that reservations of academics and people around the world are not the research problem and further, to address the gaps in existing directed towards the algorithms per se, but towards the structure of legislation. a society and its ability/inability to cope with algorithms (data use). [Hickman 2013] At the moment, we are witnessing an awkward (i) What is the nature of algorithmic uncertainty? marriage between algorithms and data. [Deangelis 2014] Together (ii) How does the European legislation address the concept of with technological progress there will be mistakes, but it should be uncertainty and un-foreseeability? taken into account that machines are just a tool and tools should not be blamed. [Deangelis 2014] Learning as such can present either 2. ALGORITHMS acquiring or enhancing existing knowledge. As stated by Herbert The algorithmic breakthrough is often connected with Alan Turing, Simon, machine learning refers to adaptive changes in a particular whose one of the most ground-breaking achievements was breaking system that enable the system to perform the same task (or tasks the German Enigma code during the Second World War. [Roberts which have been drawn from the same population) more precisely 2016] In 1952, he published a set of mathematical equations and effectively next time. [Sathya 2013] The text below will through which he wanted to explain the pattern we see in nature, introduce different types of algorithms and their main features. such as zebra stripes and leopard spots. [Serna 2017] These Through this systematization, concerns regarding certainty and algorithms are still in use when describing patterns that emerge in foreseeability of their actions and consequences will be presented. 93 2.1. Deterministic algorithms 2017] A machine that is in the process of unsupervised learning This algorithm is constructed to prevent a machine from making finds input data and classifies it (according to its own criteria) in autonomous decisions, creating new patterns, and consequently different categories. [Ploj 2013] We use unsupervised learning creating unpredictable outcomes. [Zapušek 2017] Robots equipped algorithms to identify hidden patterns and unlabelled input data, with deterministic algorithms follow predefined paths. [Sathya 2013] since they are capable of mining new data and [Hildebrandt 2016] The major significance that distinguishes the creating novel, unexpected patterns. [Ploj 2013] The system is first group of algorithms from the others is that the final action, blinded when it goes into an operation. [Van Loon 2018] It carries machine’s response or steps have already been predicted by a out its own immense and faultless logical operations that serve as human being. [Hildebrandt 2016] In the case of a deterministic guidance, but still due to the lack of exact input and output data (the algorithm, we will always find someone liable for a robot’s actions system has no reference data at all) the transparency of the process – as it will be presented in the text bellow, existing legal rules and steps is not clear. [Van Loon 2018] Despite the lack of concerning the liability of the producer will suffice. transparency this algorithm has the powerful ability to interpret limitless amount of data, through its binary logic. [Van Loon 2018] 2.2. Supervised learning algorithms 3. UNCERTAINTY More advanced forms of algorithms can enable machine learning and supervised learning algorithm is one of them. [Hildebrandt Uncertainty, as a concept, has been around for a long time, Its roots 2016] In the case of supervised learning algorithms, a machine is stretch back to Plato and Socrates, who exposed their doubt whether fed with a so-called training set by human supervisors, which scientific knowledge, no matter how elaborated and consists of particular data and predefined patterns that provide comprehensive, reflects reality. [Tannert 2007] Kant and Prauss some sort of a background on what counts as a desired and extended the idea about uncertainties with the statement that the satisfactory solution or outcome. [Ploj 2013] Hence, in the case of more detail we have about the mysteries of nature, the more we supervised learning, the machine already knows the output of the become aware of limitations of our knowledge about what things algorithm, before it even starts learning. [Van Loon 2018] Since the as such are, how do they work, etc. [Kant 2013] Limitations to our outcome is already known, all that a system needs to do is to understanding are the main reason for our inability to foresee future execute steps and processes that are needed to get from input to the events and their effects. [Tannert 2007] When policy makers and desired output. In a situation, when algorithms produce completely judicial authorities are confronted with an uncertain situation they different and unexpected results, training data serves as a guidance turn to experts for defined answers and conclusive evidences, even to steer the algorithm back towards the right path. [Van Loon 2018] though uncertainty clearly precludes definitiveness and For this reason, a supervised model can also be called error back- conclusiveness. [Fox 2009] The paradox causes questions propagation algorithm. [Van Loon 2018] Such error correction- regarding the role of scientist and scholars in situations when they learning algorithms have been thought to train the network based are facing uncertain risks when regulating advanced/modern on the input and output samples and to find error signals, which technology. In the analysis of cases of EU risk regulations it has denote the difference of the desired output and output calculated. been observed that the uncertainty paradox leads to non-effective [Van Loon 2018] and unintelligible policy-making processes. [Fox 2007] As emphasized by Mourik [Mourik 2004], without sufficiently 2.3. Unsupervised learning algorithms designed infrastructure, not only will questions concerning This one is, compared to a supervised learning algorithm, not (yet) responsibility eventually emerge, the non-sufficient regulation may as widespread and frequently. [Brummette 2017] Although the also lead to a state called ‘organized irresponsibility’. [Fox 2009] unsupervised learning algorithm has not been used on a wider scale Ulrich Beck uses this term when society is ill-prepared and is yet, its capabilities and methodology represent the future of consequently unable to effectively respond to the “inevitable machine learning. [Van Loon 2018] A good illustration of this type surprises, negative consequences and/or long-term impacts of machine learning would be imagining a child who has just started associated with uncertain risks”. [Fox 2009, 1] The idea behind discovering the world. [Brummette 2017] After his first interaction ‘risk society’ suggests that a world has become more hazardous, yet with a four-legged animal, he might hear someone call out the word this is not entirely true. [Giddens 1999] [Beck 2009] The society “dog”. [Brummette 2017] After a while, when the child sees other has become increasingly preoccupied with the future and safety that four-legged animals, such as a cow, a cat, or a horse, he thinks they generate the notion of risk. [Giddens 1999] With the term ‘risk’ we are all dogs. [Brummette 2017] “This is because the natural are describing potential events with particular consequences which classification methods installed in a human brain informed him that are evaluated as negative. [Fox 2009] The economist Knight the trait ‘four legs’ is associated with a specific animal type. As the emphasizes that risk and uncertainty are not synonyms per se. He child grows and sees more four-legged animals, additional detailed views a risk as calculable, hence controllable island in the ocean of classifications emerge. Dogs, cows, and horses are all discovered uncertainty. [Van Asselt 2006] Risk will usually be applied to the to have distinct traits and become a subset of four-legged animals situations where all potential future outcomes can be specified, in the child’s mind.” [Brummette 2017, 1] The idea of comparing uncertainty as a bare term can be applied in the situations where all machine learning process with a child’s brain was also presented potential future outcomes cannot be defined or specified before by Alan Turing in his paper ‘Computing machinery and their occurrence. [Faber 1992] It is natural that whenever we make intelligence’. [Turing 1950] He compared a child’s brain with a a decision that contains unpredictable factors, we need to weight notebook. According to him there are “rather little mechanisms, not only potential outcomes, but also their benefits and risks. The and lots of blank sheets”. [Turing 1950, 19] Hence, heading back scale of potential consequences of decisions that are carried out on to scientific conclusions about the unsupervised machine learning a national, European or even international level have a much greater technique, which makes correlations between obtained data without influence on people in comparison with individual decisions. previously hypothesizing them. [Zapušek 2017] [Hildebrandt Uncertainty as such has no ethical quality, it is inherent to an 2016] Without the provision of a training set, consisting of data and individual situation. [Tannert 2007] But, if there are concerns predefined patterns, the machine cannot familiarize itself with regarding dangerousness of a situation, uncertainty itself may information and what counts as a desirable solution. [Zapušek trigger ethically adjusted behavior, the main purpose of which is 94 the avoidance of danger and diminution of existing risks – which regulation on the European level follows the same pattern. The EC might be the case when the society is confronted with products, Directive on Liability for Defective Products (hereinafter: Liability equipped with self-learning algorithms that lead product’s actions. Directive) [Council Defective Products], adopted in 1985 ensures [Tannert 2007] “When it comes to decisions that affect people’s that producers take responsibility for their products vis-a-vis lives and health [...] carrying out research to diminish uncertainty consumers. It was one of the first pieces of European legislation and, consequentially, risks can become an ethical duty.” [Tannert that explicitly focused on the consumers’ protection and which 2007, 892] introduced the concept of strict liability, where producers are held liable regardless of whether the defect is their fault. In its article 4. FORESEEABILITY IN TORT LAW 7(e) the producer can escape liability for harm caused by the Cosmologists suggest that the Universe is comprised largely of defective product by showing that the state of scientific and ‘dark matter’, for ninety per cent of matter in the Universe does not technical knowledge at the time when they put the product into glow, but is dark. [Baker 2010] Even though we cannot directly see circulation was not such as to enable the discovery of the existence dark matter, we can detect its mass through its gravitational pull on of the defect. [Council Defective Products] The Liability Directive other astronomical objects in the Universe. [Baker 2010] This introduced a new common scheme of strict liability that in invisible stuff is powerful enough to bind all things together. comparison with the law of negligence does not require a According to professor David Owen, foreseeability is the “dark producer’s negligent behaviour. [Alaimo 2014] In negligent cases matter” of tort which connects its components, and “gives moral the claimant has to prove that the defendant knew/ought to have content to the law of negligence, controlling how each element fits known about the present risk. [Van Dam 2013] However, the new together and, ultimately, whether one person is bound to pay scheme presented by the Liability Directive sets the requirement in another for harm” [Owen 2009, 1277]. From a legal perspective its Article 4 regarding proof of the existence of a product’s defect, the test of foreseeability can be described with a sentence “whether harm, and a causal link between these two. [Council Defective one can see a systematic relationship between the type of accident Products] The creators of the Liability Directive wrote the that the plaintiff suffered and...the defendant’s (by provisions that allow national legislations to limit and dilute strict someone/something defendant is responsible for) wrongdoing.” liability. [Alaimo 2014] One of these provisions is also the [De Villiers 2015, 344] Considering the aspect of foreseeability a “development risk defence” which is still considered one of the defendant may escape liability if scientists could not predict (ex most controversial parts of this Directive. [Elliott 2011] The latter ante) a systematic relationship between wrongdoing and harm. [De is very similar to the ‘state of art’ defence that has been developed Villiers 2015] For many years, in civil law systems of most in negligence cases. [Arbour 2014] This so-called development risk European countries, tort liability was based on a broadly defined defence is enshrined in previously mentioned Article 7(e) of the general clause of fault liability. [Werro 2004] However, in the 19th Liability Directive under which the producer can escape liability century, Europe was confronted with an increase in technical and for harm caused due to lack of knowledge. According to Stapleton, industrial risks. [Werro 2004] For this reason, the majority of the above mentioned Article does not refer to risks of a product, European legal systems established liability rules which provide rather it is directed at the discoverability of the defect itself. some form of strict liability. [Werro 2004] A distinction between [Stapleton 1994] Consumer associations’ opinions believed that it fault liability and strict liability lies in their conceptual levels. “The is crucial to protect consumers from unknown and unforeseeable distinctive factor is whether or not liability rests on the judgment risks, and that the adoption of the mentioned defence (and that the defendant should have behaved otherwise than he did”. consequently the exclusion of liability for such risks) would create [Werro 2004, 7] In order to decide whether a defendant acted with a gap [Petitpierre 1974] in a general protection of consumers. fault, we need to determine all relevant standards of conduct and [Alaimo 2014] The majority of these associations claimed that the then establish that a defendant did not meet these standards. [Werro development risk defence weakens the principle of strict liability, 2004] The determination regarding the standard of care is based on and due to un-foreseeability it puts an unreasonable burden on objective criteria. [Werro 2004] Legal systems of other European consumers. [Alaimo 2014] Producer associations’ hold the opposite countries on the other hand focus primarily on the so-called ‘what position and argue that the exclusion of such a defence would could reasonably be expected of the defendant’ standard. [Werro significantly discourage scientific and technical research and 2004] Therefore, we may conclude that the limitation of negligence prevent marketing of new high tech products. [Alaimo 2014] The liability to so-called foreseeable harm prevents us to hold a idea behind the protection of producers against development risks defendant liable for not taking precautions with respect to harm is that despite taking into account all available scientific and which was not reasonably foreseeable. Fault liability can be technical knowledge it is impossible for the producer of the product explained in one sentence as a liability for reasonably foreseeable to foresee the risk in question. [Van Dam 2013] To sum up, and avoidable harm. [Werro 2004] Compared to fault liability, producers cannot be considered negligent if they have (within strict liability is “concerned with the precautionary consequences reasonable limits) analyzed and gathered all available scientific and of harm not attributable to a lack of precaution on the part of the technical knowledge. [Alaimo 2014] Making a final decision defendant, or with what may be referred to as ‘accidental harm’.” concerning liability and compensation after the occurrence of harm [Werro 2004, 9] Besides liability for reasonably foreseeable and is important, yet not enough. In order to provide comprehensive avoidable harm, it can also extend to unforeseeable and regulation of modern technologies, filling the liability gaps will unavoidable harm. [Werro 2004] [Viney 1998] Even though the simply not be enough. Deciding on liability questions must be concept of reasonably foreseeable and avoidable harm is primarily carried out simultaneously with controlling such risks before they used in negligence cases, it would be incorrect if we claimed that cause harm. However, the prohibition of the development risk probability and foreseeability issues are irrelevant to strict liability. defence, meaning the extension of a producer’s liability to development risks would almost certainly cause more harm than 5. DEVELOPMENT RISK DEFENCE benefit [Alaimo 2014], because: (1) complete liability regarding an Existing legislation of the majority of European member states unforeseeable event would most probably lead producers to regulates strict liability in the way that excludes liability for harm irrational decisions in research and development, as well as if the latter lies beyond certain limits of foreseeability. Legal innovation. [Alaimo 2014] The adoption of the development risk 95 defence suggests that too much liability might chill innovation; will be held responsible for the harm caused by modern technology, [Arbour 2014] (2) holding producers completely liable could especially if we have the capacity (knowledge) to identify the risks become an incentive for excessive litigation because plaintiffs will and through thorough research predict potential consequences of get compensation as soon as they prove the existence of a defect their operation. According to Arbour “(...) it may not be in a and a causal link between the cause and harm; [Alaimo 2014] (3) it producer’s best interest to know too much (...) as too much will cause financial problems, since it will be difficult to provide knowledge could lead to the exclusion of the DRD (development enough finances to compensate victims.[Alaimo 2014] risk defence), and, consequently, expose producers to liability.” The Liability Directive complements European product safety [Arbour 2014, 932] However, the pile of iron and cables will (in the majority of cases – except ‘vis major’) not be the main reason legislation and this is known as the ‘New Approach’ to product for unforeseeable consequences, the major issue is the operation of safety. The latter focuses on the prevention of accidents by setting the machine, its program and its self-learning capabilities. This is common safety rules which allow single market for goods to reduce why the provisions and the regulatory scope of not only the Safety administrative burden and to function as smoothly as possible. [Report 2018] In theory, as stated in Section 36 of the preamble of Directive, but also the Liability Directive needs to pay attention to software and algorithms – How does it work, which factors are the Safety Directive, [Council Product Safety] “this Directive (the relevant for the final outcome, and is the power to monitor these Safety Directive) should not affect victims’ rights within the factors in our hands? meaning on Council Directive 85/374/EEC of 25 July 1985... concerning liability for defective products (the Liability 6. EU RULES ON PRODUCT SAFETY Directive).” [Council Defective Products] However, in legal For a long time, product safety as a sector has been regulated at a practice the provisions and the scope of matters regulated by the national level as a part of public law (in Germany Safety Directive affect consumer’s liability. A requirement for ‘Gerätesicherheitsgesetz’ [GESETZ]). [Cafaggi 2009] “These deeper research results in greater awareness, not only of producers norms provided a set of rules for the producers regarding product and programmers, but also consumers. It can be claimed that safety, supplemented by technical standards for specific products removal of a development risk defence could lead producers set by hybrid state-private bodies”. [Cafaggi 2009, 245] However, towards stricter internal policies on safety. [Alaimo 2014] the European Commission quickly realized that differences However, keeping the development risk defence as it is – as a safe between technical regulations and standards among member states legal bubble that allows producers to escape liability for of the European Union presented an obstacle for trade. [Cafaggi unforeseeable and unpredictable consequences is a risky option as 2009] For this reason, in order to improve free movement of goods, well. In Rosselli’s recommendations for the European Commission the European Community started to follow the concept of ‘full it has been emphasised that researchers need to focus on the harmonization’ of technical requirements concerning a specific analysis of mutual impact between the development risk defence type of product. [Cafaggi 2009] Despite the effort, the European and general product safety legislation, which has not yet been Commission soon realized that this approach will not be successful carried out. [Alaimo 2014] In Europe, we must achieve greater due to the inflexibility of the European legislation. The latter was coherence between the Liability Directive and the Safety Directive not able to keep pace with fast technological progress and this led (and their transpositions in national legislation). In order to provide to the abolishment of the idea about full harmonization. [Cafaggi a comprehensive and an efficient legal response to modern 2009] However, the European Commission found a solution technologies it is necessary to place involve independent scientists through the judgment of the European Court of Justice in the and researchers to evaluate product safety. [Alaimo 2014] The ‘Cassis the Dijon’ case where the Commission developed a well- latter would help producers and the society avoid the so-called known principle of mutual recognition that resulted in a ‘New ‘junk-science’ getting a place in the courtroom. [Arbour 2014] Approach’ [ECJ 120/78], which should soon become the main Even though the boundaries of the development risk defence have concept for product safety regulations. [Cafaggi 2009] The latter is been set by the judiciary, the courtroom is not an appropriate place based on the so-called hybrid regulation, meaning that for scientific guesswork, Justice Posner concluded in the case main/essential requirements regarding product safety, which are in Rosen vs Ciba-Geigy Corp. [Rosen 1996] the public interest, are harmonized in EU directives, yet Requirements in the Safety Directive regarding detailed and special/unique requirements on the other hand are listed in technical thorough evaluation referring to potential consequences (evaluators standards (harmonized standards), developed by private European must foresee the unforeseeable and pass this knowledge to the standards organizations (such as CEN, CENELEC, ETSI) and are producers or manufacturers) are very important for further officially mandated by the Commission. [39] The benefit of evaluation regarding the liability for harm caused by a machine or keeping manufactured products in conformity with harmonized a product. Pursuant to the Report of European Commission “2018 standards is the presumption that products are in line with the is not 1985 (when The Liability Directive was adopted). The EU corresponding essential requirements. In other words, conformity and its rules on product safety have evolved, as have the economy with technical standards will relieve the producer of the burden of and technologies. Many products available today have proof that products meet all the requirements concerning product characteristics that were considered science fiction in the 1980s. safety. [Cafaggi 2009] On this point it is important to mention that The challenges we are facing now and even more acutely in the only the essential safety requirements are mandatory and future — to name but a few — relate to digitalization, the Internet consequently legally binding in comparison with harmonized of Things, artificial intelligence and cybersecurity.“ [Report 2018, standards, the application of which remains optional. [Cafaggi 1] Absence of detailed safety requirements will leave a lot of 2009] Compared to harmonized standards, which are set by private potential knowledge about the operation of products undiscovered, organizations, most national non-harmonized regulations (national and in such situations Article 7(e) of the Liability Directive will sector) rely on technical standards set by a mixture of industry protect producers and allow them to exculpate its liability if state (private body) and state representatives. [Cafaggi 2009] Even of scientific and technical knowledge at the time when he put the though a process of standardization is left to private bodies states product on the market was not such as to enable the existence of the want to have influence in the standardization process. [Cafaggi defect to be discovered. We want to avoid a situation when no one 2009] Despite the fact that the process of standardization is 96 organized in private forms they present a guideline for regarding safety in industry; (3) provide a system to restrict the administrative decisions, take into account interests of the general potentially harmful impact of development risks; and (4) provide public, want to satisfy common safety expectations and concretize sufficient compensation for victims.” [Cafaggi 2009] What seems due diligence in civil legislation. [Cafaggi 2009] However, to be the main problem regarding machines, equipped with governmental influence and surveillance regarding product programs that due to their algorithms enable self-learning, risk standardization is very important from European point of view. The assessment of such programs is not required by the Safety need derives from ‘the obligation to reduce restrictions or measures Directive, even though a machine (equipped with such a program) having equivalent effect on free movement of goods. [Cafaggi may cause harmful consequences as a result of to the program. 2009] Governments are sending their state representatives to Without sufficient and regular evaluation of product’s safety more standardization boards in order to provide further suggestions consequences that will occur in the future will, from a legal concerning alternative approaches to certain product areas for perspective, be recognized as unforeseeable and this will prevent standardization. [Cafaggi 2009] Private standards (product safety) the judicial branch to provide satisfaction for harm suffered by de facto observe double rules; first, they denote the required level victim. of safety, and second they denote the required level of care. [Cafaggi 2009] The interaction between product safety (outlined in 8. REFERENCES standards) and product liability can be described in the following way: (1) Public law sets a minimum level of safety and formulates 1. ALAIMO, A., et al. 2014. Analysis of the Economic Impact basic requirements. Moreover, the administrative procedures are of the Development Risk Clause as provided by Directive limited and formulated in ways that provide protection to the 85/374/EEC on Liability for Defective Products. Fondazione essential public and private goods like health and life; (2) Technical Rosselli, appointed by the Directorate for the Internal standards provided by private agencies specify a minimum level of Market of the European Commission (Contract No. safety that is provided by the public law; (3) Tort (civil) law focuses ETD/2002/85), 19-22, 59-66. on individual circumstances and tries to define and attach the 2. ARBOUR, M.E. 2014. Portrait of Development Risk as a obligation on the producer. In addition, it fills the gaps left by Young Defence. Technological Innovation and Civil technical standards and public law. [Cafaggi 2009] At first sight, it Responsibility, 59(4), 915-917, 932. seems that this briefly described structure regarding product safety and liability provides the protection against all types of dangers. 3. BAKER, J. 2010. 50 ideas you really need to know Universe. However, it fails to cover some crucial products/parts of the Quercus Publishing Plc, 72. product, such as software, especially software equipped with 4. BECK, U. 2009. Critical Theory of World Risk Society: A advanced machine learning algorithms; devices connected with the Cosmopolitan Vision. Constellations 16(1), 7. internet, etc. [Cafaggi 2009] Here, product liability rules fail to cover gaps that were left by public law. This is the repercussion of 5. BRUMMETTE, W. 2017. Supervised vs Unsupervised our mistaken perception under which we understand product safety Machine Learning, ClickUp. https:// in the traditional sense. If we look at European regulation we will clickup.com/blog/supervised-vs-unsupervised-machine- not find any specific legal provisions, even though such embedded learning/. systems (ES) are put into robotic machines (such as medicinal 6. CAFAGGI, F., MUIR WATT, H. 2009. The Regulatory products). [Cafaggi 2009] Not only at the European level, the Function of European Private Law. Edward Elgar Publishing situation at the national level is very similar. There are hardly any Limited, 245-253. safety regulations concerning ES. [Cafaggi 2009] The directive 7. DEANGELIS, S. F. 2014. Artificial Intelligence: How 2001/95/EC on general product safety does not encompass ES and Algorithms Make Systems Smart, Wired. consequently producers/programmers of ES are not obliged by https://www.wired.com/insights/2014/09/artificial- specific product safety regulations, since they do not exist. [Cafaggi intelligence-algorithms-2/. 2009] As mentioned before, the gap left by public law (such as lack of safety regulation about ES) cannot be filled exclusively by 8. DE VILLIERS, M. 2015. Foreseeability Decoded. Minnesota reorganization or updating of civil liability rules. Journal of Law, Science & Technology 16(1), 344-345. 7. CONCLUSION 9. ELLIOTT, C., QUINN, F. 2011. Tort Law. Longman, Pearson, 191. When policy makers and judicial authorities are confronted with an uncertain situation they turn to experts for definite answers and 10. FABER, M., MANSTETTEN, R., PROOPS, J.L.R. 1992. conclusive evidences, even though uncertainty clearly precludes Humankind and the Environment: An Anatomy of Surprise definitiveness and conclusiveness. The uncertainty paradox leads and Ignorance. Environmental Values, Cambridge, 1(3), 224. to an ineffective policy-making processes. If authorities want to 11. FOX, T. 2009. The role of uncertainty intolerance in provide a comprehensive and an efficient legal response to modern European Risk Governance. King’s College Risk Research technologies and liability for their actions it is necessary to involve Symposium, 1-3. sound, independent scientists and researchers to evaluate product safety. Authorities’ decisions about liability questions must be 12. GESETZ UBER TECHNISCHE ARBEITSMITTEL OF 24 carried out simultaneously with the control of such risks before they JUNE. 1968. Bundesgesetzblatt I, 717. cause harm. Such a defence must exist alongside measures (the 13. GIDDENS, A. 1999. Risk and Responsibility. The Modern majority of which must focus on safety regulation) that may help to Law Review 62(1), 3. achieve the following four objectives: (1) “provide industry with 14. HICKMAN, L. 2013. How algorithms rule the world, The unambiguous rules in order to prevent situations of jeopardization Guardian, Section: Mathematics. https:// of products’ safety due to insufficient knowledge. Industry and www.theguardian.com/science/2013/jul/01/how-algorithms- sectors will have to accept that such rules will result in a slower rule-world-nsa pace of innovation; (2) promote continuous transfer of knowledge 97 15. HILDEBRANDT, M. 2016. Smart Technologies and the 34. ZAPUŠEK, T. 2017. Artificial intelligence in medicine, The End(s) of Law, Edward Elgar Publishing Limited, 23-25. Asia Pacific Journal of Health Law & Ethics, 11, 117. 16. KANT, I. 2013. Kritik der reinen Vernunft. Felix Meiner 35. WERRO, F., PALMER, V.V. 2004. The Boundaries of Strict Verlag, Hamburg, 49. Liability in European Tort Law. Carolina Academic Press, 17. MOURIK, R. M. 2004. Did Water Kill the Cows. Pallas Stämpfli Publishers Ltd, Bruylant, 3-10. Publications. 36. Art. 30 of the European Economic Community - EEC Treaty, 18. OWEN, D. G. 2009. Figuring Foreseeability. now Art. 28 EC Treaty Wake Forest Law Review 44, 1277. 37. Council Directive 2001/95/EC of the European Parliament 19. PETITPIERRE, G. 1974. La responsabilité du fait des and of the Council of 3 December 2001 on general product produits : les bases d'une responsabilité spéciale en droit safety [2002] OJ L 11, Preamble. suisse, à la lumière de l'expérience des Etats-Unis. Genève, 38. Council Directive 2001/29/EC on the approximation of the Georg, 165. laws, regulations and administrative provisions of the 20. PLOJ, B. 2013. Metoda mejnih parov za učenje umetnih Member States concerning liability for defective products nevronskih mrež, [1985] OJ L 210. Doktorska dizertacija, Fakulteta za elektrotehniko, računalništvo in informatiko Maribor, 7. 39. ECJ, Case 120/78, Rewe-Zentral AG vs. 21. PRESS, G. 2017. Alan Turing Predicts Machine Learning Bundesmonopolverwaltung für Branntwein (1979), p. 649. And The Impact Of Artificial Intelligence On Jobs, Forbes. 40. REPORT FROM THE COMMISSION TO THE https://www.forbes.com/sites/gilpress/2017/02/19/alan- EUROPEAN PARLIAMENT, THE COUNCIL AND THE turing-predicts-machine-learning-and- the-impact-of- EUROPEAN ECONOMIC AND SOCIAL COMMITTEE on artificial-intelligence-on-jobs/#651077b01c2b. the Application of the Council Directive on the 22. ROBERTS, E. 2016. The Enigma Machine, Stanford approximation of the laws, regulations, and administrative University class (1). http://stanford.edu/class/ provisions of the Member States concerning liability for archive/cs/cs106a/cs106a.1164/handouts/29 defective products (85/374/EEC), 7.5.2018. TheEnigmaMachine.pdf. 41. Rosen v Ciba-Geigy Corp, 78 F (3d) 316 at 319, 64 USLW 23. Rosen v Ciba-Geigy Corp, 78 F (3d) 316 at 319, 64 USLW 2612 (7th Cir 1996). 2612 (7th Cir 1996). 42. The European Consumer Association (BEUC). 1999, 24. SATHYA, R. 2013. Comparison of Supervised and Response to the Commission Green Paper on Liability for Unsupervised Learning Algorithms for Pattern Classification. defective products, COM (99) 396 final, 28 July 1999, 8. (IJARAI) International Journal of Advanced Research in Artificial Intelligence, 2 (2), 34-35. 25. SERNA, H., BARRAGÁN, D. 2017. Patterns in nature: more than an inspiring design. Revista de la Academia Colombiana de Ciencias Exactas, Físicas y Naturales, Nat. 41, 352. 26. STAPLETON, J. 1994. Product Liability. Butterworths, London, 236. 27. TANNERT, C., ELVERIS, H.D., JANDRIG, B. 2007. The ethics of uncertainty. EMBO reports: Science and Society, 8(10), 892-893. 28. TURING, A. M. 1950. Computing Machinery and Intelligence. Research paper, 19. 29. TURING, A. M. 1947. Lecture to the London Mathematical Society on 20 February 1947, Seminar text, 13. 30. VAN ASSELT, M.B.A., VOS, E. 2006. The Precautionary Principle and the Uncertainty Paradox. Journal of Risk Research 9(4), 2. 31. VAN DAM, C. 2013. European Tort Law: Second edition. Oxford University Press, 434-435. 32. VAN LOON, R. 2018. Machine learning explained: Understanding supervised, unsupervised and reinforcement learning, Big Data made simple. http://bigdata- madesimple.com/machine-learning-explained- understanding-supervised-unsupervised-and-reinforcement- learning/. 33. VINEY, G., JOURDAIN, P. 1998. Les conditions de la responsabilité. L.G.D.J., 163. 98 Legal Practitioners’ Views on Neuroscientific Evidence in Court Proceedings Cristina Siserman-Gray Faculty of Law, University of Vienna Juridicum, Schottenbastei 10-16 1010, Vienna, Austria +436802300653 cristinasiserman@gmail.com ABSTRACT with significantly reduced sentence length and increased number This paper presents the results of an empirical study conducted of mitigating factors listed” [2]. Though this correlation has seen on a number of legal practitioners in Austria, Romania and corroborated by the US president’s Council on Bioethics, which Slovenia to measure the relevance of the neuroscientific claimed that “judges and juries have recognized the persuasive evidence when determining the legal responsibility of a allure of brain scans” [3] and Munro’s 2014 study, which defendant during court proceedings. The legal practitioners were concluded that the public is more likely to trust diagnosis when presented five case scenarios in which they were asked to brain imaging evidence is presented [4], our study will step analyze the admissibility of neuroscientific evidence. The results beyond these conclusions to understand how neuroscientific revealed certain important challenges when interpreting evidence affects legal practitioners in legal proceedings. neuroscientific evidence for use in legal proceedings. This study summarizes those challenges and further concludes that the 3. DESIGN OF THE EXPERIMENT application of neuroscientific data is not a straightforward 3.1 Participants matter. As such, it also argues that neuroscientific experts must The participants selected for this study were legal practitioners play a greater role in court proceedings in order to provide better (judges, lawyers and attorneys) or graduates of legal studies who guidance to judges and other legal practitioners. were already working in the legal profession. The participants’ views were assessed via an online questionnaire that was Keywords completed by 91 participants (N=91). Legal practitioners, neuroscience, law, courtroom, criminal responsibility etc. The participants were categorized in three main groups, a group from Slovenia (=Slovenian group comprising 41 practitioners with an average of 28.2 years old); a second group from 1. INTRODUCTION Romania (=Romanian group comprising 22 practitioners with an Neuroimaging scans have been used in a variety of cases in average age of 34.1 years old); and a third group (=Mixed group order to establish a defendant’s mental state. In civil cases, comprising 28 legal practitioners with an average age of 28.6 neuroimaging has been requested in personal injury, disability years old) representing legal practitioners residing in Austria, belief and contract cases, while in criminal cases it has been but coming from other countries in Europe and North America. mainly invoked in order to negate the mens rea of a crime and to avoid convictions. For instance, statistics show that in some 3.2 Instruments countries such as the USA over 1500 judicial opinions issued during 2005-2012 discuss the use of neuroscience by criminal The legal practitioners were asked to analyze five case scenarios defendants [1]. However, despite the wide use of scans in courts, that were designed in such a way as to determine a) the types of there is concern among scientists and legal practitioners neuroscientific evidence that would be acceptable in courts and regarding the practical consequences of neuroscientific b) how the legal practitioners’ assessment of the individual’s developments. The aim of this empirical study seeks to responsibility (i.e. guiltiness, type and length of punishment) understand how European legal practitioners relate to “neurolaw would be influenced by this type of evidence. The case scenarios cases” and the degree to which they admit neuroscientific were also designed to assess how legal practitioners view “new evidence in court in order to determine the moral and legal science”, such as advancements in neuroscience. responsibility of an individual. Before briefly introducing each case scenario and the results, a few remarks are required to fully understand why each case was 2. PREVIOUS RESEARCH formulated as it was. Sloboghin identified five categories of To our knowledge this is one of the first studies of its kind to neuroscientific evidence based on the cases presented in assess the views of legal practitioners regarding neuroscientific American courts [5], which we adopted as the basis for our evidence and the impact of those views on their decisions when scenarios. The 5 types of evidence are: (a) evidence of assessing the responsibility of an individual. While a number of abnormality; (b) cause-of-an-effect evidence; (c) effect-of-a- studies have concluded that neuroscientific evidence affects cause evidence; (d) individualized neuro-psychological findings legal practitioners, few have attempted to qualify that effect. compared against known performance baselines; (e) Therefore, in order to test the degree to which legal practitioners individualized neuroscience findings compared against known use neuroscientific evidence in the assessment of responsibility, legal baselines. we based our study on a hypothesis tested by Aspinwall et al. in 2012 who argued that there is a “correlation between the inclusion of the biological explanation of neurological disorder 99 4. RESULTS AND DISCUSSION which, of course, is not necessarily true. In their view, the In the following we will discuss the questionnaire results of each evidence submitted does not indicate the probability with which scenario, as well as the main problems that were raised by the people with strokes would also commit rape, which is the central respondents. Each case scenario presents an offence and the type question that a judge would want to have answered and which of neuroscientific evidence that the defendant invoked in order was not done in a satisfactory manner here. to either excuse his/her behavior or diminish his/her legal Consequently, legal practitioners noted that neuroscientific responsibility. The legal practitioners were asked to perform studies presented in courts can rarely be presumed to be fully three tasks in order to determine: (i) whether the individual in conclusive, despite the fact that some researchers often invoke the scenario should be held responsible for the act; (ii) the kind assumptions of cause and effect. In their view, the notion of of punishment or treatment that should be applied to an “cause” presents differences in meaning between the social individual; and (iii) the length of punishment (if any). We will sciences and the criminal law. It might be that because of this additionally highlight some of the challenges identified by the reason the two fields frequently clash. According to many legal legal practitioners when analyzing the type of neuroscientific practitioners, among which Denno, as well, scientists do not evidence presented in each case scenarios. often use terms such as “cause-and-effect”, but instead use concepts such as “laws of change”, “paradigms”, “models and 4.1 Difficulties demonstrating causality theories”, “hypothesis testing” and “falsification” [6]. According As reported by legal practitioners, demonstrating causality is to them, these various “jargons” make their work more among the most important problems that arise in practice when complicated. using neuroscientific evidence. The problem of causality was specifically included in one of this first case scenarios, in which 4.2 The effect of methodological weaknesses the legal practitioners were presented with a “cause-of-an- There is an awareness among legal practitioners that many effect” evidence, and asked to determine whether the evidence scientific studies have methodological weaknesses, which have showing a link between the damage to the frontal lobe and an unknown degree of impact on the overall results. Because of abnormal sexual behavior is acceptable. They were presented the these methodological weaknesses, many Courts have showed following scenario: certain resistance in considering certain statistical data in their “Mr. Jones, aged 35, was arrested for having raped three young women. decision-making. Therefore, when deciding to accept In the pre-trial investigation, a scan of Mr. Jones' brain using PET probabilistic results as evidence, legal practitioners generally (positron emission tomography) revealed serious damage to his frontal discard causal links that are assessed as not being fully lobe, apparently as a result of a stroke. The medical expert showed that during the stroke, the frontal lobe (which is involved in judgment, impulse convincible, regardless of the type of evidence, neuroscientific control and sexual behavior) has been irreversibly damaged and this or otherwise. explains Mr. Jones’ abnormal sexual behavior”. This was the solution reached by a majority of legal In principle, a “cause-of-an-effect” evidence could be highly practitioners when presented the following scenario which exculpatory in order to decide on a volitional impairment. contained a type of evidence that can be categorized as an However, the results indicated that our participants did not individualized neuro-psychological finding: generally agree in this case. In fact, some Slovenian legal practitioners reported that this type of evidence is not very Mr. Johnson, aged 27, was brought in front of the court for having sold useful for them as a basis to decide the legal responsibility of an methamphetamines. The defense lawyers informed the judges that the defendant found out 8 years before that he had an unusually large individual. Though this argument was expressed by only some pituitary tumor, which caused irreversible brain tumors. The medical Slovenian respondents, it could generally explain the answers of expert showed that pituitary tumors may affect thyroid production, causing the Slovenian Group, which agreed by a large majority (80.2%) mood disorder and damage to the frontal, temporal and thalamic regions, which may cause problems in decision-making, mental flexibility and that the accused in this case should be held responsible for rape. overall intellectual capacity. The medical expert supported the defense The same view was expressed in a proportion of 64.3% by the who claimed that because of the tumor, the defendant was more Mixed Group. susceptible to be influenced and manipulated by the drugs dealers and that is why he accepted to sell the drugs”. Interestingly, the Romanian judges were of a different opinion, In general, individualized neuro-psychological findings with only 41% of them agreeing that the accused should be held compared against known performance baselines (such as those responsible. The Romanian legal practitioners believed that the in this scenario) are accepted in courts because they provide evidence was acceptable to diminish the responsibility of the insight into the particular defendant’s biological functioning. defendant, but not sufficient to remove it completely. The Despite this fact, a great majority of respondents across all three Romanian legal practitioners did not challenge the evidence per groups (average of 87.9%) rejected the evidence presented and se in as much detail as the Slovenians did, but the Romanian agreed that the accused should be held responsible for selling legal practitioners that did agree that the accused should be held drugs. An argument invoked by the legal practitioners for this responsible also reported that they were not convinced beyond conclusion was that it is generally very difficult for scientists to any reasonable doubt about the causal link between the stroke come up with convincing and compelling data relevant for all and the abnormal sexual behavior. necessary demographic groups in order to make a useful Additionally, some Slovenian legal practitioners noted comparison with the defendant. That is because findings from disparities in the way scientific conclusions could be interpreted. neurological testing can vary significantly based on a series of In their view, the conclusion presented in this case, which variables, including gender, age, and education, among others. argues for a causal link between a stroke and abnormal behavior, In the view of the legal practitioners, interpreting specific results could also be interpreted as accepting that a high proportion of can be very difficult without this baseline information. people that commit rape had a frontal lobe stroke at some point, 100 Another concern noted by legal practitioners was that even if statistically weak and believe that more carefully performed these baseline measurements would be obtained, a correct studies are needed before they can use this type of evidence in determination about relative impulsivity of the defendant at one the courtroom. point in time does not prove anything about the impulsivity of Nonetheless, the legal practitioners also argued this type of data the defendant at the time the crime occurred (in our case when must be put in context. In their view, without knowing the the accused decided to sell the drugs). Consequently, our general base rate for violence in people with normal levels of respondents confirmed conclusions reached in other studies, testosterone, crime prevalence in groups of people with high such as that of Sloboghin who noted “that science cannot levels of testosterone is not of much help to legal practitioners. currently answer the normative question of how far below the average a defendant would have to register on a particular 4.4 Too much focus on incrimination rather performance task to be considered legally impaired” [5]. than exoneration 4.3 Limits regarding potential simultaneous Connected to the conclusion above, an interesting view of many effects and interrelationships legal practitioners was the acknowledgement that current neuroscience evidence is mostly focused on culpability issues Another challenge of neuroscientific evidence noted by legal (whether an individual is morally and legally responsible for practitioners was that they were seldom presented studies that their acts) rather than providing a basis for exoneration. Because examined all the possible variables. By ignoring potential of that, in deciding on the relevance of such evidence, legal simultaneous effects and interrelationships that may exist among practitioners and experts are aware that they have to pay the numerous other biological, social, and economic factors, attention both to the precise nature of the evidence in question they concluded that very few of these studies were complete, and the specific legal doctrine to be addressed. Though some leaving room for interpretation. Therefore, a majority of judges may argue that neuroscience evidence alone could be sufficient in our study continue to believe that most biological and to remove the responsibility of an individual, in reality, in most environmental studies of crime remain isolated in their cases, it is not. This was what the legal practitioners decided in particular disciplines, which often makes them unacceptable as the case of the following case scenario: evidence in the court. “Ms. Black, aged 47, accountant, was filed for conducting fraudulent This was one of the main conclusions reached based on the insurance practices for a period of 8 years. She was diagnosed 10 years results obtain for the following case scenario: before with anoxic encephalopathy caused by a myocardial infarction. A SPECT (single-photon emission computed tomography) was performed, “Mr. Green, aged 47 is brought in front of the criminal court for assault which indicated a reduction in blood flow in the temporal and frontal and injuries on a pedestrian. He assaulted the pedestrian on the reason lobes (which are associated with executive functioning and memory). The that he almost generated an accident while walking on the lane destined medical expert showed that her fraudulent behavior could be explained by only for bikers. Mr. Green’s lawyer pleaded that the defendant is a her impairment”. peaceful person never having been involved in fights and that his violent behavior was the result of the defendant’s level of testosterone which For this scenario, the majority of the legal practitioners changed its normal level because of the ingestion of some steroids that Mr. considered that the SPECT evidence that was included to Green took during sport training. The medical expert showed that the demonstrate a reduction in blood flow in the temporal and saliva samples collected from the defendant exhibited abnormal level of testosterone and this generated his aggressive behavior. He also added frontal lobe should not be received as an exculpatory evidence. that there is no unanimity within the scientific community regarding the Their main argument was that neuro-abnormalities have minimal positive correlation between testosterone level and aggression in general, relevance for cause in this case, particularly when having to but that in his opinion, in this particular case, the correlation is evident”. explain a causal link over an extended period of time (a stroke This case scenario contains “effect-of-a-cause” evidence, which that occurred 10 years before the trial). The causal link between basically compares the prevalence of criminal behavior among the stroke and the multiple fraudulent acts was impossible to individuals presenting a neurological impairment to those that establish. Therefore, the majority discarded this evidence, with do not have such an impairment. More precisely, the evidence 95.2% of the Slovenian, 81.9% of the Romanian respondents refers to research that indicates a higher prevalence rate of and 85.7% agreeing that the defendant should be held violence among people with high level of testosterone. When responsible for fraudulent practices. asked to evaluate whether the defendant should be held As noted by the legal practitioners, a reason why they believed responsible for assault and injuries on a pedestrian, the great prison punishment was appropriate was due of the concern that, majority (94.5%) of respondents from all three groups agreed in this case, improperly used neuroscience evidence could be that he should be held responsible. Compared to the previous used to mitigate punishment and thereby present a greater risk of two case scenarios, this one contained less divergent views. re-offense. It seems thus that judges, when confronted with the As mentioned by the respondents in the comments sections, and public pressure to ensure to that there is no risk of re-offense confirmed by other studies, evidence of clear association (like in the scenario presented above), may be more inclined to between androgen levels and aggression in human males is admit the neuroscientific evidence in order to evaluate one’s risk currently inconclusive. Legal practitioners took note of studies of re-offending. The “double-edge sword” nature of that noted that behavioral “traits of dominance and aggression in neuroscience evidence is important and was not noted by the the human male have been associated with higher levels of respondents prior to this case scenario. Therefore, based on testosterone” [6]. However, they were also aware that, as Archer these results, it is important to be aware of the possibility that concluded, “direct associations between androgen levels, neuroscientific evidence could be misused in the future to primarily testosterone, and criminality show somewhat evaluate or predict an individual’s potential level of conflicting results, possibly because of the different types of dangerousness. hormone measures used” [7]. As such, it seems that legal practitioners view research on some of these factors as 101 4.5 Cultural differences in weighing the for assigning responsibility, by minimizing arbitrary strength of the evidence incrimination or full exonerations based only on the premise that “my brain made me do it”. Finally, it is important to note that we believe that some of discrepancies included in our results are attributable to cultural We are also hopeful that the conclusions obtained in this study differences. In fact, these cultural differences seemed are sufficient to provide at least a more accurate picture of how particularly important in relation to the severity of punishment. various legal practitioners witness, from their positions, the new This is seen most clearly in the following scenario: developments in neuroscience. However, based on the results obtained in this study it is difficult to predict whether “Mr. Smith, aged 37, is a teacher and he is being tried for having made sexual advances on his young stepdaughter. He was found to have a tumor neuroscientific evidence will play a more important role in in the right frontal lobe of his brain. His medical results showed that when courtrooms in Europe in the future. There are many optimistic the tumor was removed, his pedophilic behavior stopped. When the tumor voices that see a future for neuroscientific tools, as they may recurred, the behavior also resumed”. provide a more objective assessment of an individual’s When asked whether the defendant should be held responsible responsibility, particularly when compared to legal rules, which for the sexual advances committed on his stepdaughter, a are generally insufficient and represent only a translation of the majority of the Slovenian and the Mixed groups, with 83% and social beliefs in legal terms. Of course, some concerned voices 60.7%, respectively, agreed that the accused should be held support the view that neuroscientific techniques are still not responsible for sexual advances. However, the majority of ready to provide sufficient accuracy, and hence they are not Romanians (59%) disagreed, believing that in this case, the fully reliable in the court. In our view, caution is certainly individual should not be held responsible for sexual advances. warranted in regard to neuroscientific evidence. However, with We believe that the difference in the way this defendant was further education and better understanding of neuroscientific evaluated by the three groups has to do with the type of evidence by judges and other legal practitioners, we are certain neuroscientific evidence presented to the participants in the that it holds great potential for the future. study. In this case, the legal practitioners were asked to evaluate evidence of a brain abnormality. The Slovenian and Mixed 6. ACKNOWLEDGMENTS Group argued that since hypothetical cases like these are very Our thanks to University of Ljubljana where an important part of rare in practice, very few defendants can present such a the study was performed, particularly to Prof. Dr. Olga Markic straightforward connection between neurology and behavior. As and Assist. Dr. Toma Strle who coordinated this project. such, the majority of the respondents from those two groups believed that the behavior of the defendant could not be excused 7. REFERENCES because this neurological abnormality provides an insufficient [1] Presidential Commission for the Study of Bioethical basis for making the necessary link to the unlawful behavior. Issues. 2015. Gray Matters: Topics at the Intersection of That was not the case for the Romanian Group, who in a Neuroscience, Ethics and Society, Vol. 2, Washington proportion of 59.1% believed that the accused should not be DC. held responsible for sexual advances. In this situation, Romanian [2] Aspinwall, L., Brown, T., and Tabery, J. 2012. The legal practitioners admitted the neuroscientific evidence on the Double-Edge Sword: Does Biomechanism Increase or presumption that the medical conclusions were correct. Decrease Judges’ Sentencing of Psychopaths? Science. Compared to the Slovenian and Mixed Groups, the Romanian 337, 845-849. DOI= 10.1126/science.1219569. Group did not challenge the accuracy of the medical results and/ [3] President’s Council on Bioethics. 2004. Staff Working or of the neuroscientific evidence that was brought to be Paper: An Overview of the Impact of Neuroscience examined in the case. The reason for this discrepancy is Evidence in Criminal Law. (September 2004). Retrieved unknown and could warrant further study. September 25, 2018 from 5. CONCLUSION AND FURTHER https://bioethicsarchive.georgetown.edu/pcbe/background/ DIRECTIONS neuroscience_evidence.html. The results of this experiment are addressed to both scientists [4] Munro, C. and Munro, C. 2000. “Soft” versus “hard” and legal practitioners, who, we hope, will be able to draw some psychological evidence: Biased evaluations of scientific practical applications for their work. Our first recommendation evidence that threatens or supports a strong held political for the scientists is to make sure that information they release identity. Basic and Applied Social Psychology. 36, 6, 533- publicly is valid and clearly states the scientific limitations. 543. DOI= https://doi.org/10.1080/01973533.2014.960080 Experts in the field of neuroscience should keep in mind that as [5] Sloboghin, C. 2017. Neuroscience Nuance: Dissecting the much as legal practitioners would sometimes want to accept a Relevance of Neuroscience in Adjudicating Criminal certain type of neuroscientific evidence, specific constraints of Responsibility. Journal of Law and the Biosciences. 4, 3, the legal doctrine in which they work prevent them. A second 577-593. DOI= 10.1093/jlb/lsx033. message is addressed to legal practitioners, who should begin [6] Denno, D. 2016. The Place for Neuroscience in Criminal considering the best ways to integrate new scientific discoveries. Law, In Philosophical Foundations of Law and Neuroscientific evidence can provide a more nuanced Neuroscience, New York, Oxford University Press, 69-84. understanding of neurological impairments in a way designed to humanize the legal system rather than creating motives for [7] Archer, J. 1991. The influence of testosterone on human discriminating against individuals. This more nuanced aggression, British Journal of Psychology. 82. 1-28. DOI= understanding can additionally provide a more graded approach https://doi.org/10.1111/j.2044-8295.1991.tb02379.x. 102 FROM AI TO OUR FUTURE I AI and the Drake equation Matjaž Gams Jozef Stefan Institute Jamova 39, 1000 Ljubljana, Slovenia matjaz.gams@ijs.si ABSTRACT prehiteti ljudi na vseh področjih? V tem prispevku je najprej This paper deals with a question, can we predict the future of the predstavljena največja svetovna konferenca s področja umetne human civilization based on the analysis of the current AI inteligence, sledi pa še analiza Drakove enačbe z novimi research, and computational models of the Drake equation. The računskimi pristopi. Na osnovi teh analiz skušamo oceniti, kaj perspective of human civilization is analyzed through two items: nas čaka v bližnji prihodnosti, tj. skušamo napovedati prihodnji report from the AI mega-conference, and the Drake razvoj človeške civilizacije. equation/Fermi paradox. The mega-AI conference in Stockholm Letos poleti je bila v Stockholmu največja svetovna konferenca represents the state of the art in the AI field, demonstrating that IJCAI, združena z evropsko ECAI [4]. Skupno je bilo preko major world powers are fast increasing AI funding. The 6.000 udeležencev. Približno polovica vseh prispevkov je bila computational models such as recompilations of the Drake kitajskih, pol manj je bilo evropejskih in ameriških. Velesile se equation are gaining attention as one of the means to estimate zavedajo, da je področje umetne inteligence eno izmed ključnih, the longevity of the human civilization. kjer se odloča, kdo bo dominiral svetu. Tako Putin kot Trump in POVZETEK Ši Džinping intenzivno povečujejo sredstva za umetno inteligenco, Evropa jih bo v nekaj letih nekajkrat povečala. To je prvi prispevek od dveh, ki opisujeta isti koncept: ali lahko napovemo bodočnost človeštva na osnovi analize sedanjih Dnevno naredi umetna inteligenca neverjetnih 10 bilijonov raziskav umetne inteligence (AI), računskih modelov na osnovi odločitev. Dosežkov umetne inteligence samo v lanskem letu je Drakove enačbe in novi socialni črni scenarij, ki bi lahko bilo toliko, da jih lahko omenimo le minimalni delež. Recimo na zaustavil razvoj človeštva. področju varnosti marsikje po svetu uporabljajo sistem, ki vsak dan izdela nov urnik obhodov varnostnikov po letališčih, po V tem prispevku je analizirana bodočnost človeške civilizacije pristaniščih in podobno. Kjer so bili uporabljeni, so izmerili skozi dve temi: poročilo z največje svetovne AI konference in bistveno večjo učinkovitost. V skrbi za okolje so raziskovalci analiza Drakove enačbe oz. Fermijevega paradoksa. Mega pod vodstvom prof. Tambeja (tam je bil tudi naš doktorand dr. konferenca letos v Stockholmu je predstavila trenutne raziskave Kaluža) tovrstne programe podarili šestdesetim rezervatom po AI v svetu in pokazala, da svetovne velesile izredno povečujejo svetu, da se bodo uspešneje upirali krivolovcem. Leta 2015 so financiranje AI področja. Po drugi strani pa narašča število programi globokih nevronskih mrež začeli premagovati ljudi pri analiz in objav s področja bodočnosti človeštva. Računski prepoznavanju vidnih nalog in danes nas prekašajo na mnogo modeli med drugim dajejo ocene trajanja človeške civilizacije. področjih, npr. pri prepoznavanju malignih tkiv. Pri nekaterih nalogah, recimo pri ostrenju slike (dež, megla, sneg itd.) so Keywords / ključne besede programi osemkrat boljši kot ljudje. Seveda se takoj pojavi Progress of human civilization, Drake equation, Fermi paradox, strah, ampak če čakate nekaj tednov na diagnozo, ali imate raka artificial intelligence ali ne, v Ameriki pa to naredi umetna inteligenca v nekaj minutah bolje kot katerikoli zdravnik – kaj pravite, ali bi jo Razvoj človeške civilizacije, Drakova enačba, Fermijev uvedli tudi pri nas? Pri nekaterih sistemih kot zamenjavi paradoks, umetna inteligenca organov so programi že desetletja v uporabi in so rešila na tisoče življenj. Nekateri so novejši, recimo letos so vpeljali prvi 1. UVOD inteligentni program, ki ugotavlja diabetes iz pregleda oči, prav tako prvi program za ugotavljanje abnormalnosti prsnega koša pri slikanju. Neverjeten razvoj je najbolj znan pri avtonomni Večina projektov v odseku E9 na Institutu »Jožef Stefan« se vožnji – danes imajo povprečni avtomobili kar nekaj ukvarja z izboljšavami življenja in zdravja ljudi, recimo s skrbjo avtonomnih inteligentnih funkcij, modernejši kot Tesla pa vozijo za starejše preko inteligentnega pomočnika na pametni uri, ali praktično sami in jih samo še nadziramo v nenavadnih pa za srčne bolnike. Del raziskav pa je bolj visoko letečih. Nekaj situacijah. Nesreč je približno stokrat manj, čeprav tistih nekaj časa že razmišljamo, kako zgraditi inteligentne asistente iz mediji toliko bolj napihnejo. biblije, korana, slovenske ustave. Asistenti ne bi znali samo odgovarjati na vprašanja s primernim besedilom, ampak bi bili In smo pri »moralnem stroju« - vprašanju, koga naj povozi sposobni tudi tolmačiti določen del vsebine, semantike, vrednot. avtonomno vozilo, če npr. izbira med otrokom ali starejšim: Bi tako ustvarili kopijo npr. krščanskega boga? Ti programi bi http://moralmachine.mit.edu/. V Nemčiji je prepovedano bili sposobni demonstrirati svoje avtonomno razmišljanje na upoštevati leta (prepovedana starostna diskriminacija), v testih osnovi podanih besedil, torej bi bili v nekem smislu »živi«. Za pa se izkaže, da precej ljudi raje ponudi možnost preživetja dosego tega pa morajo imeti tudi, ali predvsem kognitivne otrokom. Najpomembnejše je število, torej naj bi avto zavil med lastnosti. dva namesto med tri ljudi. Dobršen del anketirancev pa raje izbere brezdomca kot otroka itd. Preko spleta so zbrali že okoli Če so trenutne raziskave oz. analize na tem nivoju, ali so v svetu petdeset milijonov odločitev in izkaže se, da imajo določene raziskave že blizu superinteligence [1,2,3], ki bi morebiti znala države podobne sisteme preferenc, koga povoziti in koga ne. 103 Tako so ugotovili značilne vzorce vrednosti pri zahodnih Slika 1: AI strategija EU temelji na treh elementih: državah, spet druge v vzhodni Evropi itd. Anglofilske države kot znanost/tehnologija, socio-ekonomske spremembe in socialno Anglija, Amerika, Avstralija in Nova Zelandija so svoj skupek s okolje. podobnimi vrednotami, čeprav so geografsko precej narazen. Ta grupiranja torej pokažejo, katere države imajo podoben sistem vrednot in kako se razlikujejo od drugih. Na Sliki 1 je prikazana strateška usmeritev EU glede umetne inteligence. Med drugim bo EU podprla gradnjo nove odprte V praksi je taka situacija izredno redka, saj imajo ljudje možnost platforme za AI, malce podobno kot Muskov OpenAI (Slika 2). odskočiti, avto se zaleti v oviro in zaradi varnostnega pasu Na sliki 2 je prikazana usmeritev bodočega financiranja: najprej potniki skoraj gotovo preživijo, medtem ko pešec kakšne 70% povečanja in nato 100% in nato še 100%. posebne zaščite nima. Študija, ena izmed mnogih, pa predvsem pove nekaj o nas samih – kakšni smo kot ljudje in kako cenimo Ameriški DoD je ustanovil Joint AI Center (JAIC), ki bo življenje drugih. Sistemi umetne inteligence zato ne rešujejo financiral 600 AI projektov v vrednosti $1.7 milijarde. samo pereča inženirska in socialna vprašanja, ampak tudi iz nas delajo boljše ljudi. Marsikdo med nami je prepričan, da je ravno umetna inteligenca tista ključna znanstvena veda, ki bo dvignila človeštvo v novo civilizacijsko ero [11]. 2. IJCAI 2018 – KAKO ZGRADITI UM Julija 2018 v Stockholmu so se združili ICML, AAMAS, ICCBR in SoCS z IJCAI and ECAI v prvo mega-konferenco. Namen dogodka je bil ponovno integrirati področje in se namesto s čedalje bolj specifičnimi problemi ukvarjati z ključnimi splošnimi. Opis v tej sekciji je namenjen skupni IJCAI-ECCAI konferenci [4]. Okoli 7000 udeležencev je 14 dni sodelovalo v mega- konferenci. IJCAI-ECCAI konferenca je doživela ponoven rekord s 3470 poslanimi prispevki: 37 % več kot v letu 2017. Slika 2: Tudi EU bo signifikantno povečala financiranje AI. Leta 2017 [5] je bilo 37% vseh prispevkov kitajskih, leta 2018 pa 46%. Ameriških in evropskih prispevkov je bilo vsakih po Najpomembnejša strateška usmeritev mega-konference IJCAI je 20%. bila, kako zgraditi um. Ker umetnega uma odraslih očitno še ne Spremembe niso prišle same po sebi, ampak so rezultat znamo zgraditi, so raziskave usmerjene predvsem v študije uma ključnega vlaganja kitajskega državnega vodstva v znanost in otrok od rojstva dalje. Čeprav se je vpliv AI na vsakodnevno zlasti v umetno inteligenco. Odgovor Amerike in Evrope je že življenje ljudi izjemno povečal, saj AI vsakodnevno naredi 100 tu. bilijonov odločitev, so koncepti kot razumevanje, semantika, um, kognicija, čustva, »duša«, kvalia itd. ostali nedosegljivi za AI, podobno kot rešitev Turingoveg testa. Vseeno je razvoj izredno hiter. Nekatere nove aplikacije so vizualno fascinantne – recimo konja znajo nevronske mreže prebarvati v zebro in se na zaslonu premika konj v obliki zebre. Iz seznama obrazov slavnih osebnosti znajo ti programi zgraditi nove obraze neobstoječih ljudi. Sistemi govorijo praktično kot ljudje in tudi razpoznavajo govor na tem nivoju, veliko vidnih nalog pa opravljajo precej bolje kot ljudje. Google je začel uporabljati globoke nevronske mreže za prepoznavanje vprašanj in občasno je na čudna vprašanja generiral vizije, tudi preroške vizije črne prihodnosti človeštva. Vraževerni so začeli govoriti o prebujanju prave inteligence, a so razvijalci Googla pojasnili, da so nevronske mreže prav čudna vprašanja povezala s prav čudnimi odgovori. Obstaja plitva, tj. sedanja uporabna in raziskovalna AI, globoka AI, tj. globoke nevronske mreže, za katero velja mnenje raziskovalcev umetne inteligence, da je v resnici plitva, prava AI in lažna AI. Lažna AI je, ko vskočijo ljudje, tipično v neko komunikacijo med virtualnim asistentom in človekom, in tega ne povedo. Prava AI pa je AI, sposobna rešiti Turingov test, tista, 104 ki da nam ljudem vtis resnične inteligence. V Turingovem testu preračunal verjetnosti in ugotovil, da bi že morali vzpostaviti ljudje še vedno hitro prepoznajo računalnike, čeprav se stik z drugimi civilizacijami. Sedaj vemo precej več: število pretvarjajo, da so ljudje, ker jim manjka prava AI. planetov v znanem vesolju je reda velikosti 10 na 22, pregledali smo ogromno več vesolja in nismo našli nobenih znakov S čedalje več raziskavami postaja tudi čedalje bolj jasno, kje se civilizacije! Pri tem je minilo 68 let od Fermijevega vzklika in AI razlikuje od prave AI. Recimo pri učenju se otrok (Slika 3) 64 let od njegove smrti. Umrl je istega leta kot Alan Turing, nauči enega koncepta, npr. kako sestaviti kocke, nato pa ga računalniški Einstein in oče umetne inteligence (in se je rodil praktično takoj ali z minimalnim učenjem prenese na druge avtor tega prispevka). domene, recimo sestavljanje figuric. Pri tem se uči univerzalno, medtem ko mora AI uporabljati specialne programe za Eno izmed najbolj poznanih ocen števila civilizacij je postavil posamezne naloge, jih ne uspe posplošiti in ne zna prilagoditi Drake s svojo enačbo leta 1961 [6]. Ocena števila razvitih algoritmov podobnim nalogam. zunajzemeljskih civilizacij v naši Galaksiji je (x je znak za množenje) N = R* x fs x fp x ne x fl x fi x fc x L, kjer velja: R* je razmerje zvezdnih skupin v naši Galaksiji, tj. hitrost nastajanja novih zvezd, število reda velikosti 1. fs je delež Soncu podobnih zvezd z lastnimi planeti. fp je delež zvezd z lastnimi planeti. ne je povprečno število planetov, ki omogočajo življenje v razmerju z zvezdo, ki ima planete. fl je delež planetov, kjer se domnevno lahko razvije življenje. fi je delež planetov, kjer se dejansko razvije življenje. fc je delež tistih, ki bi želeli ali se bili sposobni sporazumevati. L je pričakovana življenjska doba take civilizacije. V bistvu ta enačba temelji na oceni števila planetov; pomnoženi z verjetnostjo, da so na planetu ugodni pogoji za razvoj življenja; pomnoženi z verjetnostjo nastanka življenja; Figure 3: Otroci se učijo bistveno drugače kot sistemi AI. Zakaj pomnoženo z verjetnostjo tehnološko napredne civilizacije; se ne bi zgledovali po njih? skalirano s časom trajanja civilizacije, saj nas zanima število civilizacij sedaj. V tej enačbi se trajanje naše galaksije pokrajša, Drug pomemben poskus v smeri prave AI je t.i. splošna AI, ki ko se pojavi pri R kot produkt in pri L kot imenovalec (glej skuša razvijati široko uporabne algoritme namesto ozko enačbo). specializiranih. Tako npr. naj bi obstajal en algoritem za igranje vseh iger, ki bi znal izvajati vse algoritme za različne igre, in Ker je izračun odvisen od privzetih vrednosti in ker imajo prenašal znanje med njimi. Večinsko mnenje med strokovnjaki različni strokovnjaki razne ocene, Wikipedija predlaga tri umetne inteligence je, da bomo to vrsto inteligence razvili čez možnosti za konkretne številke: kakšnih 10 let. Splošno mnenje je, da nam ne manjka denarja ali • Ena civilizacija v naši galaksiji – komunikacija oz. procesorskih zmogljivosti – iščemo nove, drzne ideje, kako zaznavanje drugih civilizacij s sedanjimi sredstvi ni razviti splošno, pravo in super-inteligenco. Bolj ali manj je tudi mogoče. konsenz, da bo AI pomagala reševati nakopičene socialne težave • 100 civilizacij naši galaksiji, v povprečju oddaljenih (Slika 1). Konferenca je ostala strogo znanstvena, vendar je med 5000 svetlobnih let med dvema civilizacijama, ki znanstveniki določena zaskrbljenost zaradi čedalje pogostejših lahko med seboj komunicirata. napadov na znanost in prodiranje ideologij med znanstvenike. • Štiri milijone civilizacij, komunikacija preprosta. 3. FERMIJEV PARADOKS - ČRNI Zvezd in planetov je glede na naše znanje ogromno: galaksij je okoli 10 na 11 do 12 in podobno število je zvezd v galaksiji, SCENARIJ ČLOVEŠTVA torej je skupno število zvezd okoli 10 na 23 do 24. Trenutna ocena planetov je okoli 10 na 22. Če je med temi planeti le na Katere nevarnosti prežijo človeštvu? Skoraj zagotovo so vsakem milijardnem civilizacija, to pomeni 10 na 13 civilizacij, največje in najbližje nevarnosti povezane s hitrim vzponom to je deset tisoč milijard. V naši galaksiji je okoli 10 na 11, tj. človeške civilizacije. Prav tako med najbolj perečimi ni sto milijard planetov, kar po podobnem merilu prinese sto nekaterih, ki se pogosto pojavljajo v medijih, npr. globalnega civilizacij s povprečno razdaljo med njimi pet tisoč svetlobnih segrevanja. Čeprav se planet Zemlja segreva in čeprav je let. Na taki razdalji je možno prepoznati močne energijske najverjetnejši krivec za to človek, je ta nevarnost obvladljiva, če signale. se le uspemo dogovoriti. Bolj problematične so neobvladljive, nerazumljene in potuhnjene nevarnosti. V kakšnem iz naslednjih Verjetnost, da je življenje edino na Zemlji, je praktično nič, pa prispevkov bomo predstavili izvirno teorijo, temelječo na če pregledate Googla, Wikipedijo ali sami malce preračunate. socialnem propadu, tu pa bodo na kratko opisane analize Samo – sosednjih naprednih civilizacij gotovo ni, ker bi jih že Drakove enačbe. opazili! Prav daljnih civilizacij trenutno najverjetneje ne moremo prepoznati, niti srednje oddaljenih, če se nočejo Veliki fizik Enrico Fermi je že leta 1950 med atomskimi razkriti. raziskavami v Los Alamosu vprašal: »Pa kje so?«, ko je 105 Precej vprašanj ostaja odprtih, vendar nekaj lahko vseeno teleskopi. Čedalje bolj napredne analize in ocene števila sklepamo: Če so civilizacije prav redko posejane po vesolju, civilizacij v vesolju nakazujejo, da je civilizacij malo. potem so lahko trajale dalj časa, pa se niso opazile med seboj. Če pa so gosteje posejane, tj. če se življenje seli s planeta na Obstajata dve teoriji: po eni je civilizacij dejansko malo in smo planet v obliki trosov ali medzvezdnih poletov, morajo propasti ena redkih civilizacij v tem delu vesolja, vendar hkrati ni relativno kmalu, drugače bi jih že opazili. Kaj je to »kmalu«, ne posebne nevarnosti, da bomo kmalu propadli. Po drugi teoriji je vemo, od reda velikosti 100 do 10.000 let. V primeru redke bilo pred nami že ogromno civilizacij relativno blizu nas, vendar poseljenosti imamo opravka s pojavom, da življenje nastane so vse po vrsti propadle. Ta usoda čaka tudi nas, najverjetneje, samo od sebe v primernih pogojih, kar smo v laboratorijih že če se ne bomo pravočasno zavedli problema in potegnili pogosto postorili, vendar življenje ne uspe poseliti bližnjih primernih potez. planetov. Dovolj daleč narazen pa lahko civilizacije obstajajo Zlasti v primeru druge možnosti se je potrebno zavedati, da je dalj časa, tudi stotine milijonov let, a se ne uspejo zaznati zaradi čim hitrejše potovanje vsaj na bližnje lune in planete nujnost ogromnih razdalj. preživetja, kot večkrat poudarja Elon Musk. Naslednje vprašanje je, zakaj ni starejša in napredna civilizacija LITERATURA poselila vsega vesolja. V grobem ostajajo tri možnosti: [1] Bostrom, N. 2014. Superintelligence – Paths, Dangers, - Potovanje na daljše razdalje je preveč zahtevno, ker ni Strategies. Oxford University Press, Oxford, UK. mogoče potovati hitreje kot s hitrostjo svetlobe - Trajanje civilizacije je omejeno / Nekaj žre civilizacije [7] [2] Yampolskiy, R.V. 2016. Artificial Superintelligence. CRC - Napredne civilizacije pustijo razvijajoče se, kot npr. našo, Press. in se nočejo vpletati. [3] http://www.tromba.si/matjaz-gams-scenariji- superinteligence-in-cloveske-civilizacije/ Trije raziskovalci s Future of Humanity Institute (FHI) [8], Superinteligenca Oxfordska univerza, med njimi Anders Sandberg, (s katerim je avtor prispevka intenzivno debatiral med konferenčnim izletom [4] IJCAI 2018 conference (https://www.ijcai-18.org/). na kitajski zid) so objavili drugačno razlago Fermijevega [5] Gams, M. 2017. Editor-in-chiefʼs introduction to the paradoksa [9]. Za vsak parameter v enačbi so vstavili special issue on "Superinteligence", AI and overview of verjetnostni interval in izračunali končno verjetnostno IJCAI 2017. Informatica : an international journal of distribucijo. V članku avtorji trdijo, da je verjetnost, da smo computing and informatics, ISSN 0350-5596, vol. 41, no. 4, edina civilizacija v galaksiji 53 do 99,6%, da smo edini v str. 383-386 vesolju pa 39 do 58 %. Torej drugih civilizacij ne zaznamo zato, [6] https://sl.wikipedia.org/wiki/Drakova_enačba, ker so tako redke. Dokaj verjetno smo edini v galaksiji in morda https://en.wikipedia.org/wiki/Drake_equation, tudi v vesolju. Wikipedija Drake equation Pomemben zaključek njihove raziskave je, da ni posebne [7] http://beta.finance-on.net/files/2016-04-10/Matjaz- nevarnosti, da bo naša civilizacija kmalu propadla. Izračun Gams.pdf temelji na verjetnostni porazdelitvi in predvsem pri verjetnosti Nekaj »žre« civilizacije življenja da nizko oceno – kot da življenje le redko uspe, oz. le na prav posebnih planetih. Tak posebni primer naj bi bil naš [8] Future of Life Institute, https://futureoflife.org/ planet, ki ima luno, ki je nastal s trkom s planetom velikosti [9] http://www.jodrellbank.manchester.ac.uk/media/eps/jodrell Marsa itd. Tako trdi tudi Gribbin [10]. -bank-centre-for-astrophysics/news-and-events/2017/uksrn- slides/Anders-Sandberg---Dissolving-Fermi-Paradox- Naši izračuni še potekajo. Prve ugotovitve bodo predstavljene na UKSRN.pdf konferenci Slovenian Conference On Artificial Intelligence v Analiza Drakove enačbe okviru multikonference Informacijska družba. [10] Gribbin, J. 2011. Alone in the Universe: Why Our Planet Is Unique, ISBN-13: 978-1118147979, ISBN- 4. ZAKLJUČEK 10: 1118147979, John Wiley and sons. [11] Kurzweil, R. 2006. The Singularity Is Near: When Humans Transcend Biology, Sep 26, Penguin Books. Umetna inteligenca se razvija in se ukvarja z vprašanji reproduciranja osnovnih kognitivnih sposobnosti, ki se [12] Gams, M. 2001. Weak intelligence: through the principle pojavljajo pri otrocih. Na ta način pričakujejo, da bodo dosegli and paradox of multiple knowledge. Nova Science. izboljšavo sedanjih formalnih metod AI, ki nimajo praktično nobenih najbolj pomembnih človeških lastnosti. Hkrati AI čedalje bolj vpliva na vsakodnevno življenje v pozitivnem smislu – očitno je eden izmed ključnih generatorjev napredka [11]. V zadnjih desetletjih smo odkrili, da ima praktično vsaka zvezda najverjetneje nekaj planetov, torej je planetov ogromno. To je napovedoval že princip mnogoterega znanja [12], ki je tudi napovedal precejšnje število civilizacij. Hkrati pa ne opazimo nobene napredne civilizacije v delu vesolja, ki ga zaznajo naši 106 The Quest for Understanding: Helping People with PIMD to Communicate with their Caregivers. Matej Cigale Mitja Luštrek Matjaž Gams Jožef Stefan Institute Jožef Stefan Institute Jožef Stefan Institute Jamova cesta 39 Jamova cesta 39 Jamova cesta 39 Ljubljana, Slovenija Ljubljana, Slovenija Ljubljana, Slovenija matej.cigale@ijs.si mitja.lustrek@ijs.si matjaz.gams@ijs.si Torsten Krämer Meike Engelhardt Peter Zentel Pädagogische Hochschule Pädagogische Hochschule Pädagogik und Didaktik im Heidelberg Heidelberg Förderschwerpunkt Institut für Sonderpädagogik Institut für Sonderpädagogik Geistige Entwicklung Keplerstraße 87D Keplerstraße 87D Keplerstraße 87 Heidelberg, Germany Heidelberg, Germany Heidelberg, Germany kraemer@ph-heidelberg.de engelhardt@ph- zentel@ph-heidelberg.de heidelberg.de ABSTRACT make further attempts more distinct. The problem arises from the People with Profound Intellectual and Multiple Disabilities (PIMD) fact that the communication attempts of the people with PIMD are stand for a broad and very heterogeneous spectrum of people that indistinct, mostly unique to each individual, and hard to distinguish are characterised by some common aspects like a severe intellectual or interpret for people who are not close to them. The differences disability usually in combination with a lack of conventional and stem from different abilities of each person, different reception of symbolic communication abilities, coupled with the need for high the gestures from their caregivers and other external factors. levels of support due to comorbidities or other possible disabilities At present, only close caregivers are able to interpret the desires of (i.e., motor or sensorial impairments). Supporting these individuals people with PIMD in the right way, nevertheless often combined is extremely challenging, as their communication signals are with feelings of insecurity. This makes it difficult to easily expand atypical and idiosyncratic. Therefore, a plethora of these the circle of communication partners. The INSENSION project behaviours are not or not easily readable for the caregivers. Without faces this issue by aiming to use the advances in computer vision, background information on a specific person with PIMD, it is hard specifically new ways to extract posture and facial expressions for a caregiver, even a trained professional, to interpret the desires from video to codify them. In the second stage of the processing, and mental state of the person they are interacting with, which leads the context of the expression will be extracted, i.e. what is the state to a stressful interaction for both. With advances in computer vision of the environment around the person. This way, the expression can (CV), speech recognition technology (SRT) and artificial be coupled with the interaction and the intent of the communication intelligence (AI), we are making the first steps in codifying these can be interpreted. This would hopefully provide improvements for behaviours and attempting to mechanically extract the meaning of both sides: for the people in contact with people with PIMD to have the communication. The INSENSION project aims to use these a window into their internal states and see their attempts and, of advancements to catalogue the actions of persons with PIMD and course, for the people with PIMD themselves. However due to their the environment and thus provide feedback to caregivers and enable disabilities, the communication attempts of people with PIMD individuals to control their surroundings. A similar system could be seem simpler and their internal working models are often assumed used to analyse the behaviour of healthy individuals so that the to be limited to temporally and spatially neighbouring desires that generalised and personalised expressions of body language could makes the interpretation context smaller. be codified and compared across cultures and individuals. This research has broader implications as a similar, but more complex, system could be used to interpret communications of Keywords individuals so that their behaviour could be objectively determined communication, gesture recognition, PILD, AI enabling a more rigid research into the communication of people and their internal state. 1. INTRODUCTION The rest of the paper is organised as follows: In Section 2 we look People with PIMD experience a lot of trouble when they are at the state of the people with PIMD and their communication attempting to communicate to the outside world. Generally, a attempts. In Section 3 we take a brief look at the underlying profound intellectual disability, which complicates the learning of technical advancements that can facilitate the extraction of posture new skills for them, is combined with other sensory or physical and facial expressions and the vocalisations of the person. In impairments, which lead to an unusual communication in Section 4 we present the annotations that will be the input for the comparison to people without disabilities. Machine Learning (ML) system. In Section 5 we discuss the On this occasion, the task of their caregivers (parents, therapists, implications of this system and present some caveats to the system. etc.) is to interpret their communication attempts and teach them to In section 6 we look at broader implications of the system as it 107 could be used on a more general population to systematically codify or Simplified Signalling that draws the inspiration form natural the interactions as well as further fields of inquiry that could be gestures that are taught to the people with PIMD so that their developed based on this system. communication is in line with general public [6]. There is also research in using Brain Computer Interface for communication [7] that uses Electroencephalography (EEG) to map the activity of the 2. PEOPLE WITH PIMD brains to interpret the desires. The system requires adaptation based People with PIMD, as the name implies, have multiple disabilities, on the individual and training of the individual. which makes it even harder for them to participate in the large number of non-barrier-free parts of our society. Generally, PIMD means a profound intellectual disability combined with other 3. ANALYSING THE HUMAN sensory (blindness, deafness) or physical impairments (lack of mobility, problems with fine hand movements, etc.). These factors Enabling computers to interpret the desires of its users and their severely influence the person’s ability to live without any care, state of mind is a longstanding goal of computer science. The most support or therapy of others [1]. Individuals with PIMD have an developed systems focus on speech recognition[8], [9], but other above average risk to get additional diseases, and frequently require systems are also explored. Eye tracking [10] is becoming more and regular medication that also implies administrative aid. Individuals more robust, moving from specialised hardware to simple web are assumed to attempt to communicate but are often not able to do cameras [11]. so successfully, because of the inherent and external limitations [2]. More advanced systems enable facial features extraction [12], and A common denominator in the population we are dealing with is from this psychological state of the individual can be extrapolated the limited ability to communicate coherently with their caregivers [13]. These systems work extremely well on typical individuals in or the other way around. They usually communicate on a pre- good conditions, for example in good lighting and direct camera symbolic level and their understanding of speech is severely position [14]. In more dynamic conditions, such as unstable lighting limited. Some individuals have the ability to form joined attention these results are less certain but still reach acceptable levels of with their communication partner [3] but this is not universal. accuracy. Additional information, such as voice inflection or While they are capable of learning, the acquisition of new skills contact sensor provides additional information and greater takes significantly more time and requires frequent repetition. accuracy. Research has been conducted in extracting stress of People with PIMD tend to exhibit not or not easily readable students using only the smartphone, carried by the person[15]. behaviours in order to communicate their (dis-)pleasure or to get Another advancement in computer vision is the possibility of attention. Examples of these are pushing unwanted objects away, mapping the body parts of one or more persons from video [16]. loud vocalisations or banging to gain attention. In general, the This enables extracting of the limb and torso position, the position communication attempts of people with PIMD are relying on of fingers and facial markers. This enables researchers to qualify caregivers who have been trained to understand their the position of a person and increases the robustness and ease of communications by interpreting their whole body behaviour or analysing behaviours of humans. An example of this can be seen in specific personal expressions [4]. Figure 1. Communication attempts are very multifaceted based on the The position of a person is only part of the puzzle of finding out specific individual. Some persons are capable of vocalising simple what the person wants. In order to determine the context of the words, such as saying “Hi”, but do not consistently use them in a interaction there is another piece of the puzzle. The environment of correct manner or they grab towards toys and individuals they want the person must be taken into account. There are several systems to interact with. Making eye contact is possible for some that can take a video and return the objects present in the scene. individuals, which can be an orientation towards desired objects as One of the fastest open source solutions is YOLO [17]. Other communication attempt. Others do not have any coherent commercial systems also exist, such as Google or Amazon Object vocalisations, lack motor skills and require help holding items [3], recognition API [18]. [5]. Further complicating their behaviour or the interpretation of is stereotypy. These are actions that do not contain communication attempts but can be considered “ticks” that do not carry meaning. There is a correlation between stereotypical behaviours and low level of social interaction and stimulation. This behaviour can escalate over time to aggressive behaviour and sometimes even self-injuring. The level of these problematic behaviours seems to be correlated with communication problems [5] and would presumably point to this being an expression of frustration. These behaviours can range from hand wringing, to hitting legs, head or nearby objects, from purposeful breath holding to screaming etc. Several attempts were made to bridge the gap of communication of people with PIMD. Some individuals have access to switches that produce specific sounds enabling an easier communication for less experienced communication partners [4], [6]. These switches can take several forms form simple push buttons to systems that attach to the individual muscles. Other systems, such as Picture Exchange Communication where the individuals with PIMD are expected to Figure 1. An example of the image processed by provide a picture, usually on a card, for the desired communication OpenPose. The Fingers, arms and facial characteristics are extracted and returned as points. 108 4. THE EXPRESSIONS OF THE PEOPLE 5. OVERVIEW OF THE PROPOSED WITH PIMD SYSTEM In our research we are working with six people with PIMD in order Our system will take the information collected from the camera, to provide a robust data set that will be used to train the system and microphone and other sensors, extract the objects in the scene and extract the state of the individual and the communication they are the information about the communication attempts of the users and, trying to accomplish. Based on this we hope to extract some based on this, provide a guess on the presumable mood and rudimentary information, such as their psychological state communicated content. (pleasure, displeasure or neutral) and the mode of communication In the first round, communication classes will be extracted using they are exhibiting such as protest, demand or comment. Based on unsupervised learning that will return common actions performed this information and the context – the activity that was happening by the individual, as annotated or provided by the movement before, the objects that are available for interaction and the analysing system. This will return meaningful communication estimation of internal needs based on models such as hunger, thirst clusters. These will be simple like moving the hand from position etc. – the system will propose the action the caregiver should take. A (on head) to position B (pointing), interacting with the desk, The first step for this is collecting the data. In our case this video is vocalisation or a combination of these, such as pointing and manually annotated in order to provide information for the system. producing a distinct sound. We use several cameras, an infrared camera, a wristband to collect All these communication clusters can then be mapped onto the state physiological parameters and microphones that collect sounds. The and communication system. At this stage supervised learning will videos are collected in several interactions ranging from meal time, be used. Some actions will for instance indicate that the user is playing (Figure 2) and physical therapy and even some life-skills displeased - providing the information on the internal state of the training. individual. While others will indicate communication attempts such as demand for something. Together they can indicate to the caregivers what action they should take to provide care for their charges. The first problem we have to solve is to remove the behaviour that does not carry any communication information. In general, it is expected that action that happen regardless of the state of the individual will be filtered out. This could become problematic as some actions can have several meanings. As such the efficacy of this is considered ongoing research. In addition, the context, such as presence of strangers or objects in the vicinity, will be used to provide information on the desires and aversions of the individual, creating an internal database of likes and dislikes that can be taken into consideration. Figure 1. An image of video recording of a play session with a caregiver. 6. CONCLUSIONS The videos are annotated to indicate the position of the individuals As the system is not yet operational, there are several problems, that so that the system can be trained to return the desired information are still part of research. For instance, it is unknown which role such as arm position, facial expressions and actions such as rubbing stereotypy will play, or how accurately we can extract the parts of the body, interacting with objects or people, presence of information if the user is interacting with a specific object or person. The information that the user is interacting with one object may not be enough to infer the actual desire of the user. Once this, and other problems are resolved the system can be extended to general public. Enabling researchers and others to analyse the behaviour of the individuals and extract the communication desires and psychological state of the individual to further their understanding of the motivations and desires of people, providing standardised analysis of their movements and philological states. This will in turn enable greater rigidity of inquiry providing a faster and reproducible way of analysing behaviour of people. However, the system could be misused. Organisations and individuals could use the system to determine the state of the Figure 1. An image of video recording of a play session individual and use this to manipulate her or him for their gain. A with a caregiver. system that can extract the information could be used to associate some products and people with this feeling, thus steering the people people in the scene and any external disruptions, such as loud noises to the competition. Furthermore, body language could be important etc. In Figure 3 we can see an example of the resulting annotation. for determining not the information but relationships of people, creating an internal map that could result in an interaction map that 109 could be used to influence certain groups of people or simply to Disabil. , vol. 4, pp. 133–145, 2016. determine the connections between them and their reactions to [8] M. Benzeghiba et al. , “Automatic speech recognition and certain events. speech variability: A review,” Speech Commun. , vol. 49, no. 10–11, pp. 763–786, 2007. 7. ACKNOWLEDGMENTS [9] M. Anusuya and S. Katti, “Speech recognition by Funding: This work was supported by the European Union’s machine: A review,” Int. J. Comput. Sci. Inf. Secur. , vol. 6, no. 3, pp. 181–205, 2009. Horizon 2020 research and innovation program [grant agreement No 780819 (INSENSION)]. [10] C. H. Morimoto and M. R. M. Mimica, “Eye gaze tracking techniques for interactive applications,” Comput. Vis. Image Underst. , vol. 98, no. 1, pp. 4–24, 2005. [11] J. San Agustin et al. , “Evaluation of a low-cost open- 8. REFERENCES source gaze tracker,” Proc. 2010 Symp. Eye-Tracking Res. Appl. - ETRA ’10, p. 77, 2010. [1] H. Nakken and C. Vlaskamp, “A Need for a Taxonomy for Profound Intellectual and Multiple Disabilities,” J. [12] W. S. Chu, F. De La Torre, and J. F. Cohn, “Selective Policy Pract. Intellect. Disabil. , vol. 4, no. 2, pp. 83–87, transfer machine for personalized facial expression Jun. 2007. analysis,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 39, no. 3, pp. 529–545, 2017. [2] K. Atkin and M. P. Lorch, “An ecological method for the sampling of nonverbal signalling behaviours of young [13] Z. Yin, M. Zhao, Y. Wang, J. Yang, and J. Zhang, children with profound and multiple learning disabilities “Recognition of emotions using multimodal physiological (PMLD),” Dev. Neurorehabil. , vol. 19, no. 4, pp. 211– signals and an ensemble deep learning model,” Comput. 225, 2016. Methods Programs Biomed. , vol. 140, pp. 93–110, 2017. [3] P. J. Yoder and S. F. Warren, “Relative treatment effects [14] H. Gunes and H. Hung, “Is automatic facial expression of two prelinguistic communication interventions on recognition of emotions coming to a dead end? The rise of language development in toddlers with developmental the new kinds on the block,” Image Vis. Comput. , vol. 55, delays vary by maternal characteristics.,” J. Speech. Lang. no. 1, pp. 6–8, 2016. Hear. Res. , vol. 44, no. 1, pp. 224–237, 2001. [15] M. Gjoreski, H. Gjoreski, M. Lutrek, and M. Gams, [4] C. Harding, G. Lindsay, A. O’Brien, L. Dipper, and J. “Automatic Detection of Perceived Stress in Campus Wright, “Implementing AAC with children with profound Students Using Smartphones,” 2015 Int. Conf. Intell. and multiple learning disabilities: A study in rationale Environ. , pp. 132–135, 2015. underpinning intervention,” J. Res. Spec. Educ. Needs, [16] S.-E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh, vol. 11, no. 2, pp. 120–129, 2011. “Convolutional Pose Machines,” in Conference on [5] Me. Laura Roche, “Evaluating and Enhancing Computer Vision and Pattern Recognition, 2016. Communication Skills in Four Adolescents with Profound [17] J. S. D. R. G. A. F. Redmon, “(YOLO) You Only Look and Multiple Disabilities,” Victoria University, 2017. Once,” Cvpr, 2016. [6] J. Goldbart and S. Caton, “Communication and People [18] R. Buyya, C. S. Yeo, and S. Venugopal, “Market-oriented with the most complex needs: what works and why this is cloud computing: Vision, hype, and reality for delivering essential,” What is communication and what are complex IT services as computing utilities,” Proc. - 10th IEEE Int. communication needs? , no. July. pp. 3–4, 2010. Conf. High Perform. Comput. Commun. HPCC 2008, pp. [7] H. Adeli, P. Rahimian, and N. Tabrizi, “Communicating 5–13, 2008. with People with Profound Intellectual Disabilities Using Brain Computer Interface,” J. Technol. Pers. with 110 Kognitivna raven kot del relacijske družinske terapije pri predelovanju spolnega nasilja Tanja Repič Slavič Teološka fakulteta, Poljanska 4, Ljubljana, katedra za Zakonsko in družinsko terapijo ter psihologijo in sociologijo religije Frančiškanski družinski inštitut, Prešernov trg 4, Ljubljana E-mail: tanja.repic@teof.uni-lj.si POVZETEK Take izkušnje imajo velik vpliv tudi na telesno zdravje [6, Posamezniki, ki so v otroštvu doživeli spolno nasilje ter se 7]. v odraslosti soočajo s posledicami, potrebujejo za Samo nasilje je za žrtev lahko tako travmatično, da če se le globinsko preoblikovanje vzorcev, odnosov in notranje ta ne zdravi, lahko posledice postanejo in ostanejo drže, strokovno pomoč. Brez dela na sebi se pogosto lahko doživljenjske [8]. Pomembno je poudariti, da otrok, ki je bil znajdejo nazaj v začaranih krogih nasilja, ali kot žrtve ali žrtev spolnega nasilja, v obdobju mladostnika ali odraslega kot storilci, in doživljajo neprijetne posledice nasilja, ki jih posameznika predstavlja veliko tveganje, da sam postane celo nezavedno prenašajo naprej na otroke. Predno pa je povzročitelj spolnega nasilja nad otroci [9]. Raziskave so posameznik sposoben iti v globino čustev in nezavednega pokazale, da je spolna zloraba v otroštvu zelo pogosta med spomina, je v relacijski družinski terapiji (RDT) s strani spolnimi prestopniki, saj ponavljajo priučeno vedenje, ki so terapevta nujno poskrbeti za varen prostor, ki ga med ga bili sami deležni, torej jih je v otroštvu tako zelo drugim omogoča prav vrednotenje travme na kognitivni travmatiziral in zaznamoval. Spolni prestopnik se kot otrok ravni, o čemer bomo pisali v prispevku. V tem kontekstu nauči pozitivnega odnosa do spolnega vedenja med bomo najprej predstavili nekatera teoretična izhodišča v otrokom in odraslim človekom [10]. Za žrtve posilstva povezavi z RDT in spolnim nasiljem, potem pa bomo bolj obstaja velika verjetnost, da bodo ponovno žrtve, za natančno osvetlili stopnje procesa RDT na kognitivni ravni. deklice, ki so bile telesno ali spolno zlorabljene, pa je velika verjetnost, da bodo ponovno zlorabljene, kot odrasle Ključne besede ali da bodo postale prostitutke [11]. Spolno nasilje, relacijska družinska teorija in terapija, kognitivna raven, razreševanje travme. 2. RELACIJSKA DRUŽINSKA TEORIJA IN SPOLNO NASILJE 1. SPOLNO NASILJE RDT poudarja, da zgodnje travme, kamor sodi tudi spolno nasilje, lahko povzročijo najpogosteje nezmožnost pri Spolno nasilje je vsako vedenje ali dejanje s spolno reguliranju agresivnih impulzov, strahu, sramu in gnusa [12]. Ti vsebino, ki si ga žrtev ne želi in pri njej vzbuja občutek, da afekti ostanejo zelo globoko ukoreninjeni v psihobiološki je njeno telo le inštrument za zadovoljitev potreb [1]. Pri strukturi otroka in še v odrasli dobi lahko silovito zaznamujejo spolnem nasilju gre za vsa spolna dejanja povezana s posameznika, ki je bil žrtev zlorabe, zlasti pa se prebudijo, ko spolnostjo, ki jih oseba čuti kot prisilo, ob njih doživlja pride v odraslem odnosu do intime. Zaznamuje pa tudi njegovo sram in gnus ter da so bile prekoračene telesne meje in okolico, ki se večkrat z velikim nerazumevanjem do njegove meje osebnega dostojanstva. Na drugi strani pa storilec to bolečine odvrača od njega, saj ne razume, zakaj se vedno znova stori z namenom spolnega vzburjenja [2, 3]. Nasilje se zapleta v odnose, kjer je spet zlorabljena [13, 2]. Gostečnik [12, pojavlja na različnih nivojih, še posebej boleče pa je, kadar 14] meni, da se med spolnim nasiljem vzpostavi izredno močna nastopa kot relacijska travma, kar pomeni da se izvaja s afektivna povezava med rabljem in žrtvijo. Zato lahko žrtev strani žrtvi bližnje osebe, npr. družinskega člana. Še zlasti zlorabe ravno na osnovi tega odnosa, ki je bil kruto vsiljen, v težki obliki nasilja, ki se izvajata v družini ali pa novih situacijah nezavedno išče zlorabo oziroma v zlorabi išče neučinkovito razrešujeta, sta fizično in spolno nasilje. podobne situacije, vzdušja in ljudi, ki jo bodo ponovno zlorabili. Žrtve nasilja so zaznamovane z dolgotrajnimi posledicami. Pri tem posameznik podoživlja afekte zlorabe, kot bi se zloraba Raziskave med najpogostejšimi posledicami omenjajo dogajala tukaj in sedaj, ne pa kot del preteklosti. Nezavedno posttravmatsko stresno motnjo, neprimerno regulacijo čutijo, da morajo ostati zvesti tem občutjem, saj bi sicer čutili afekta, depresijo, anksiozne motnje, zlorabo drog in obupno praznino, močno hrepenenje, ki celo organsko boli, alkohola, kompulzivno vedenjsko ponavljanje travmatičnih dokler ne pride do ponovitve začaranega kroga in pomiritve. To scenarijev, občutke sramu in krivde, destruktivnost do sebe praznino vsekakor lahko pomirijo tudi z raznimi oblikami in drugih, vedenjsko impulzivnost, agresivne izpade, zasvojenosti, z avtodestruktivnimi vedenji, kot so avtoagresijo, spremenjeno podobo o sebi in podobno [4, 5]. samopoškodovanje in motnje hranjenja, ... [15]. Zloraba pomeni odnos, in odpovedati se odnosu za zlorabljenega pomeni izgubiti 111 vse, zato pogosto vztrajajo v še tako bolečih odnosih, v upanju stopnje, ki pa se pri vsakem posamezniku odvijajo v na razrešitev [16]. drugačnem vrstnem redu. Prav tako ni značilno, da bi se neka stopnja zaključila in začela druga, ampak se lahko iste S področja pomoči pri reševanju posledic spolnega nasilja je stopnje večkrat ponovijo, vendar vsakič na drugačen način, sicer veliko različnih tehnik, svetovalnih metod in pristopov, kar na drugem nivoju. V tem oziru proces okrevanja spominja pa omogoča le razreševanje na bolj površinski ravni, saj ni na spiralo, kjer se ponovijo isti nivoji, a vsakič raven višje. dovolj možnosti za naslavljanje vseh dinamik in čustvenih stanj, Pomembno je vedeti, da ima vsak posameznik svojo pot ki jih pusti travma nasilja. Sodobna relacijska psihoterapevtska okrevanja. Ni boljših in hitrejših poti, so le različne poti, ki paradigma, kamor se uvršča tudi model relacijske družinske vodijo k istemu cilju. V relacijski družinski terapiji je terapije (RDT), poudarja prav to: nerazrešene psihične vsebine, zagotovo cilj, da se spolno nasilje razrešuje na kognitivni ki so zaznane tudi na nevrobiološki ravni, prihajajo na dan v ravni, na ravni čutenj in afektov ter na telesni ravni [2, 21]. različnih simptomatičnih in velikokrat nefunkcionalnih oblikah V nadaljevanju bomo predstavili kognitivno raven, ki je na vedenja. RTD preko mehanizmov projekcijsko introjekcijske nek način najbolj na površini, a brez nje do ostalih dveh identifikacije (transferja in kontratransferja) prodira v nezavedni naprej ni mogoče prodreti. spomin in tako z ozaveščanjem le tega pomaga regulirati posameznikova psihobiološka stanja in afekte, ki izhajajo iz teh stanj [12, 17]. Ker je razreševanje travme zelo kompleksen 4 KOGNITIVNA RAVEN PRI proces, saj potlačene psihosomatske vsebine prihajajo postopno na dan, je nujno v terapiji zagotoviti varen prostor. PREDELAVI SPOLNEGA NASILJA Klinične izkušnje [2] kažejo, da zlorabljeni začnejo predelovati spolno zlorabo najprej na kognitivni ravni . Ko 3. PROCES RELACIJSKE DRUŽINSKE govorimo o kognitivni ravni, imamo v mislih racionalno TERAPIJE PRI RAZREŠEVANJU dojemanje, miselne procese posameznika. V tem okviru se SPOLNEGA NASILJA prepletajo različne teme in stopnje procesa, ki pa se ne odvijajo pri vseh v istem vrstnem redu. Relacijska družinska terapija postavlja v središče a.) ODLOČITEV: Najprej se mora zlorabljeni odločiti, da terapevtskega dela odnos, ki omogoča spremembo osnovnih želi nekaj spremeniti. Do te odločitve jih največkrat relacijskih struktur. Velik pomen daje ozaveščanju pripeljejo posledice spolne zlorabe, ki jih doživljajo na potlačenih vsebin in zagotovitvi manjkajočih zgodnjih različnih področjih. V večini primerov nihče moči teh doživetij. Vendar pa ne gre zgolj za drugačna doživetja in posledic ne pripisuje spolni zlorabi ampak sebi, kot tistemu, popravljanje starih, nedokončanih procesov razvoja, temveč s katerim je nekaj zelo narobe, saj sicer ne bi doživljal za razumevanje temeljnega vzorca odnosov, ki ga živi toliko stisk in kriz. Slutijo pa, da tu in tam travmatičen posameznik, in za spremembo teh temeljnih vzorcev dogodek prispeva del bolečin, vendar nikakor v takem odnosov. Klient zagotovo potrebuje profesionalno pomoč, obsegu in s tako intenzivnostjo, kot se kasneje v procesu zlasti pri premagovanju in soočanju s strahom, terapije izkaže. samoobtoževanjem, krivdo, z jezo in drugimi močnimi čutenji ter afekti, ki jih pusti spolna zloraba [12, 18]. S b.) KOGNITIVNO DOJEMANJE POSLEDIC SPOLNEGA pomočjo novih izkušenj, ki se zgodijo v terapevtskem NASILJA: ko terapevt začne na srečanjih vrednotiti te odnosu, bo prihajalo pri klientu do sprememb. Te stiske in iskati izvor sedanjih doživljanj, ki se le prebujajo spremembe ne bodo povezane samo z vsakdanjim in prihajajo od drugod, začnejo zlorabljeni postopno življenjem, ampak tudi s spremembami v možganih [19], kognitivno dojemati resnične posledice spolne zlorabe, kar posledično vpliva tudi na hormonski sistem ter samo dinamiko, povezave s preteklostjo in sedanjostjo … preusmeritev privlačnosti, kar v praksi pomeni, da Čutenja, ki se jim ob tem pojavljajo, in telesne senzacije, ki zlorabljenemu niso nezavedno več privlačni ljudje, ob jih doživljajo, najprej le racionalno povezujejo z zlorabo, in katerih bo ponovno podoživljal vzdušje in afekte originalne normalno je, da občasno celo podvomijo in zanikajo, da bi travme [12, 17]. lahko zaradi te travme doživljali toliko groz. Rothschildova [16, 20] meni, da mora vsako terapijo, c.) GLASNO PRIZNANJE, DA SE JE SPOLNO NASILJE povezano s travmatičnimi doživetji, spremljati deset ZARES ZGODILO: velik korak naprej je, ko lahko temeljnih načel, ki so: posameznik glasno pred vsemi v skupini (če je terapija skupinska) ali le pred terapevtom (če je terapija 1. vzpostavitev varnosti za posameznika, tako na terapiji, inividualna) po nekaj srečanjih izgovori in prizna, da je bil kot izven nje; 2. razviti dober in učinkovit terapevtski spolno zlorabljen. Običajno na začetku besedne zveze odnos; 3. sposobnost in zmožnost ustavitve terapevtskega spolna zloraba sploh še ne zmorejo izreči, če pa že, jo procesa; 4. identificirati in graditi posameznikove notranje izrečejo zelo disociirano – brez čutenj, bolj racionalno, kot in zunanje vire; 5. graditi mehanizme za soočenje s travmo; da to ni nič tako zelo hudega. 6. vedno znižati pritisk; 7. prilagoditi se posameznikovim potrebam; 8. poznati širok spekter psihofizioloških teorij o Zaradi krutosti dogodka je povsem normalno, da se občasno travmi in PTSD; 9. upoštevati posameznikove svojske pri posameznikih pojavljajo dvomi, da si morda vse samo razlike in posebnosti ter 10. terapevt mora biti pripravljen domišljajo, da so kaj takega sanjali in da se jim to sploh ni opustiti določene ali vse terapevtske tehnike. res zgodilo. Vendar se v času terapije prej ali slej pokaže, da grozne sanje pustijo drugačen pečat in ne vplivajo tako V tem kontekstu temeljnih načel dela s travmo, se močno in tako intenzivno na človekovo doživljanje, prepletajo v samem procesu razreševanja različne teme in vsakdanje življenje. Prav tako se tudi izkaže, da so dvomi 112 le posledica bolečine in stisk ter da obstajajo tudi resnična V tem kontekstu lahko rečemo, da v prvih tednih ali dejstva, ki ovržejo dvom in potrdijo resničnost doživljanj. mesecih na terapijah čustva in afekte dojemajo kognitivno, Terapevt mora pri tem verjeti, da se bo v času terapije kar pomeni, da že vedo, da niso bili oni krivi, vendar se izkazalo in da je možno prepoznati, začutiti, kdaj bi bil nek krivde še ne morejo znebiti. Prav tako šele postopno lahko spomin implantiran ali izmišljen. priznajo, da se jim je zgodila krivica, vendar jeze na Č.) RAZKRITJE SPOLNEGA NASILJA TUDI DRUGIM: začetku sploh še ne zmorejo začutiti, če pa že, jo usmerjajo občutek olajšanja, da zlorabljeni ne nosi več sam svoje na »napačne« osebe in izražajo na neprimerne načine. S temne skrivnosti, nekatere vzpodbudi, da tvegajo povedati strani terapevta korak za korakom v začetnih fazah o spolni zlorabi tudi komu drugemu izven terapije – morda dojemajo, da je v takem dejanju s strani storilca čutiti enemu od družinskih članov (izkušnje kažejo, da najlažje ogromno sramu, gnusa, prezira, vendar večinoma ostajajo sestri ali bratu, šele potem mami, očetu redkokdo), ali še indiferentni, včasih prav depresivni in v svojem svetu – nekomu od prijateljev. Pomembno pri tem je, da žrtev ne da bi telesno začutili omenjena čutenja. Običajno ta zlorabe čuti, da je ta oseba vredna zaupanja in da bo čutenja začnejo doživljati veliko prej v vsakdanjem razumela zlorabo – da ne bo obsojala, minimalizirala življenju, ob osebah, ki niso bile originalen storilec, dogodka, opravičevala storilca … Dogaja se tudi ravno postopno celo ob terapevtu. V tem se jasno vidi transfer, obratno – da posamezniki, preden pridejo na terapijo, vsem prenos čutenj prek projekcijsko-introjekcijske identifikacije brez razmejitev razlagajo, kaj se jim je zgodilo, saj na ta na neko drugo osebo, kar se za začetek izkaže kot bolj način regulirajo bolečino in sproščajo napetost. V času varno, manj tvegano, kot če bi to čutili ob storilcu. Na to terapij pa kmalu vidijo, da so izbirali predvsem take osebe, kaže tudi najpogostejši odgovor na vprašanje, koliko so ki njihovih klicev na pomoč niso slišali. Tako začnejo jezni in besni na tistega, ki jih je spolno zlorabljal. Ob tem postopno postavljati meje in svoje zgodbe ne pripovedujejo večina samo skomigne z rameni, da jeze ne čutijo, čeprav bi več vsakemu, ki jih bolj ko ne iz vljudnosti vpraša, kako so, se jim zdelo prav, da bi jo čutili – razumsko torej vedo, da saj se s tem samo še bolj ranijo in razvrednotijo. bi »morali« čutiti jezo, gnus, bes, vendar čutenjsko tega še niso »sposobni« oziroma še ne smejo začutiti. Z drugimi d.) OZAVEŠČANJE TRAVME SPOLNEGA NASILJA IN besedami, ni še dovolj varno in je v terapiji potrebno prej DVOJNO PREPOZNAVANJE – SPOMINJANJE: naravna odkriti še druge plasti, obrambne mehanizme, afektivne pot procesa je, da začne v času terapij postopno prihajati na psihične konstrukte in simptome kompulzivne dan vedno več potlačenih čutenj, afektov, telesnih reakcij retravmatizacije, da lahko pridejo do samega jedra … Pri nekaterih se te vsebine prebujajo v odnosih z bolečine. drugimi ljudmi, pri drugih prek sanj, branja neke knjige, lahko prek močnih telesnih odzivov, gledanja kakega filma, e.) DOŽIVLJANJE KRIZ: spomini in čutenja, ki se začnejo vedenj, ki si jih ne znajo razložiti … Vse te reakcije mora intenzivno prebujati, drugačno gledanje na okolico, ljudi terapevt nenehno vrednotiti in skupaj z udeležencem okrog sebe, izvirno družino ter nove in nove teme, ki se še terapije prek povezav iskati, kam to spada, kje se je že tako odpirajo, sprožajo obdobja kriz. Posamezniki bodo v takih počutili, od kod to prihaja … Prve tedne ali celo mesece trenutkih povedali, da se jim zdi, da je sedaj še slabše, da je zlorabljeni te povezave razumejo le na kognitivni ravni – bilo lažje živeti prej, ko niso ničesar vedeli, da sedaj verjamejo, da je to res tako, lahko najdejo povezave, si to nenehno razmišljajo samo o zlorabah, da nimajo nobene ponavljajo, vendar jim to še ne prinaša nekega olajšanja, saj moči, da bi se ukvarjali s čim drugim, da se jim zdi, da se še niso zmogli tudi čustveno umiriti, telesno sprostiti. Pri bodo zapravili vse dobre priložnosti, ki se jim ponujajo, da tem je bistveno in nujno potrebno vztrajanje, nenehno niso sposobni videti drugega, razen svojih stisk in težav, da ponavljanje in utrjevanje teh povezav dvojnega nimajo energije, da bi si sploh kaj lepega zase privoščili, da prepoznavanja tega, kar se je zgodilo v preteklosti in se se jim zdi, da so nori, da bi se najraje zapili, za vedno sedaj samo prebuja, se ne dogaja več. Spolno zlorabljeni zaspali, pustili službo, študij … V takih krizah klinične vsa čutenja in afekte, ki so povezani s travmo spolne izkušnje kažejo, da najbolj pomaga, če imajo izven terapije zlorabe in se v neki sedanji situaciji samo prebujajo, ob sebi vsaj še eno osebo, na katero se lahko obrnejo, doživljajo na način, kot da se travma dogaja ravno sedaj včasih tudi to, da si jasno strukturirajo dan, predvsem [17, 22]. Njihovo telo, čustva, mimika, ton glasu, vedenje, takrat, ko jim je najtežje in bi samo spali, ter da obvezno vse se odzove na podoben način, kot da jih prav sedaj naredijo zase vsaj nekaj pozitivnega [23]. Prav tako nekdo spolno zlorablja. Ko se jim podobna občutja nekaterim pomaga, če lahko ubesedijo svoje stiske in krize pojavljajo v okolici, pogosto povedo, da jih drugi na list papirja in napišejo čisto vse, kar čutijo, brez označujejo kot »preobčutljive«, »zakomplicirane«, »cenzure«, samo da dajo ven iz sebe. Taki zapisi se »neuravnovešene«, »čudne« … Prav ponavljanje in izkažejo za zelo pozitivne tudi kasneje, ko pride morda utrjevanje teh povezav pa postopno začne spreminjati tudi novo obdobje krize in ko z branjem podoživljajo, kaj vse so čustvene in telesne odzive. V pomoč pri tem je lahko tudi že prestali, preživeli, premagali ... Mnogi povedo, da so se obrazec dvojnega prepoznavanja [2, 20], da se prek njega najbolj umirili, ko so težka čutenja uspeli umestiti v čas in tudi izven terapevtskih srečanj utrjujejo novi vzorci. Na prostor, jih ovrednotiti kot sestavni del procesa, ki ne traja začetku pa udeleženci na terapijah le kognitivno dojemajo večno in vedno mine. Četudi se obdobja kriz običajno še dejstva in se čustveno ter telesno stvari kaj bistveno še ne ponovijo, nikoli niso več tako intenzivna oziroma so bolj spreminjajo, kar pa je povsem razumljivo; če je namreč obvladljiva in manj časa trajajo. Ko so posamezniki nekdo živel dvajset, trideset ali več let z vzorci in razmišljali o premaganem obdobju krize, so videli, da so posledicami spolne zlorabe, se to čez noč nikakor ne more prav zaradi takih »zdravilnih« kriz lahko kasneje šli hitreje začeti spreminjati. in drugače naprej, kot bi šli sicer brez njih. Cilj pri razreševanju travme spolne zlorabe je iti samo naprej, 113 četudi včasih najprej korak nazaj, da lahko gre kasneje dva zlorabljenim ne pomaga le preživeti, ampak tudi živeti in koraka naprej! zaživeti človeka vredno življenje. f.) POVEZOVANJE ELEMENTOV KOMPLEKSNEGA DOŽIVLJANJA – SIBAM: vsako kompleksno doživljanje, " Doseženi rezultati so delno nastali v okviru projekta št. dogodek je iz različnih elementov. Levine [24] je za J5-9349, ki ga je financirala Javna agencija za razreševanje travme razvil SIBAM model (S = sensations, I raziskovalno dejavnost Republike = image, B = behaviour, A = affect, M = meaning), ki temelji na predpostavki, da celoten spomin nekega dogodka vključuje priklic petih elementov: slik, senzacij, vedenja, 6. REFERENCE afektov in pomena. Ko posameznik doživi travmatičen dogodek spolne zlorabe, ti elementi med seboj zaradi [1] Rozman, S. 2015. Pogum, kako preboleti travmo. hudega stresa niso povezani, ampak so disociirani, zato tudi Modrijan Založba. kasneje lahko doživlja prave napade panik in groz, pa niti [2] Repič Slavič, T. 2015. Nemi kriki spolne zlorabe in ne ve, kaj jih je povzročilo. Slik in vedenja nima, pomena novo upanje. [2. izd.]. Celje: Celjska Mohorjeva tudi ne, ima pa telesne senzacije in afekt. Pri izogibanju, družba: Društvo Mohorjeva družba. 2015. zamrznitvi in drugih odzivih so prisotne druge kombinacije izmed teh petih elementov. Na terapijah poleg dvojnega [3] Repič Slavič, T. 2016. Sexual abuse in childhood and prepoznavanja terapevt in udeleženec iščeta tudi povezave the unconscious repetition of pain in couple vseh petih elementov doživljanja in tako določene odseke relationships. V: Józef Stala, ur., Jadranka Garmaz, spomina sestavljata v celoto. Najprej proces poteka na ur. Strengthening families, 137-151. The series kognitivnem nivoju, potem pa tudi na čustvenem in Family studies, 5. Kraków: The Pontifical University telesnem. Potrebnih je veliko ponovitev in vztrajanja, da se of John Paul II. začnejo pojavljati novi vzorci vedenja, večji nadzor nad [4] Simonich, H. K., Wonderlich, S. A., Erickson, A. L., prej na videz neobvladljivimi čutenji in navsezadnje tudi Myers, T. C., Hoesel, J., Wagner S. in Engel, K. pozitivne spremembe. 2015. A statewide trauma-focused cognitive g.) VREDNOTENJE SPREMEMB IN NAPREDKA: ob behavioral therapy network: creating an integrated vseh teh doživljanjih, prebujanju bolečih spominov, community response system. Journal of vzponih in padcih pa je nujno potrebno veliko pozornosti Contemporary Psychotherapy 45, št. 4:265-274. posvetili tudi priznanjem in pohvalam osebi, ki je bila [5] Vaillancourt-Morel, M. P., Godbout, N., Sabourin, S., spolno zlorabljena in se trudi s pomočjo terapij v Briere, J., Lussier, Y. in Runtz, M. 2016. Adult vsakdanjem življenju nekaj spremeniti. Še zlasti, ko pridejo sexual outcomes of child sexual abuse vary according obdobja kriz in takrat običajno človek ne vidi ničesar to relationship status. Journal of Marital and Family pozitivnega, je bistveno in pomembno, da terapevt zna Therapy 42, št. 2:341-56 . podpreti ter ovrednotiti prehojeno pot, pohvaliti tisto, kar je [6] van der Kolk, B. A. 2002. Beyond the talking cure: posameznik že naredil, trud in vztrajanje, vložene napore in Somatic experience and subcortical imprints in the moči ter verjeti še naprej in še bolj, da bo šlo vse še na treatment of trauma. V: Francine Shapiro, ur. EMDR boljše in samo naprej. Na skupinskih terapijah se as an integrative psychotherapy approach: Experts of udeleženci podpirajo med seboj in na podlagi svojih diverse orientations explore the paradigm prism, 57- izkušenj povedo, kaj jim najbolj pomaga, jih najbolj 83. Washington: American Psychological napolni z energijo, usmeri naprej, daje moč, da vztrajajo. Association. Vsi pa povedo, da jim zelo veliko pomeni, da terapevt zaupa in verjame v njihov napredek, da pohvali in [7] Yule, W. 2001. Posttraumatic stress disorder in the ovrednoti spremembe ter da lahko pokaže in pove, da je general population and in children. Journal of ponosen na njihovo »garanje« – prehojeno pot. Prav te Clinical Psychiatry. 62. št. 17:23-8. spodbude so med drugim zelo ključne pri procesu okrevanja [8] Solomon, M. F. in Siegel, D. J. 2003. Healing in razreševanja travmatične izkušnje spolne zlorabe. trauma: Attachment, mind, body, and brain. New York: W. W. Norton & Company. [9] Maniglio, R. 2011. The role of child sexual abuse in 5. ZAKLJUČEK the etiology of substance-related disorders. Journal of Klinične izkušnje terapevtov kažejo, da ne moremo nikoli Addictive Diseases 30, št. 3:216-28. reči, da pridemo kdaj na konec poti razreševanja travme [10] Boonmann, C., Grisso, T., Guy, L. S., Colins, O. F., spolnega nasilja – kajti, ko je dosežen en cilj, se pokažejo Mulder, E. A., Vahl, P., Jansen, L. M. C., potrebe po novi rasti, po novem cilju, in proces postane Doreleijers, T. A. H. in Vermeiren, R. R. J. M. 2016. sestavni del življenja. Kljub vsem posledicam, ki Childhood traumatic experiences and mental health zaznamujejo posameznikovo življenje, lahko v problems in sexually offending and non-sexually terapevtskem odnosu posameznik tako napreduje, da ga offending juveniles. Child Adolesc Psychiatry Ment travma nezavedno ne usmerja več, ko pa pridejo sprožilci, Health, št. 10:45. ki sprožijo določne organske senzacije, bo prav kognitivna [11] Cvetek, R. 2009. Bolečina preteklosti: travma, raven tista, ki bo pomagala premostiti obdobja kriz in medosebni odnosi, družina, terapija. Celje: Celjska razmejiti, da se dogodek sedaj ne dogaja več, ampak se le Mohorjeva družba: Društvo Mohorjeva družba. prebujajo določeni segmenti. To kognitivno zavedanje 114 [12] Gostečnik, C. 2008. Relacijska paradigm in travma. Ljubljana: Brat Frančišek in Frančiškanski družinski center. [13] Henry, J., Sloane, M. in Black-Pond, C. 2007. Neurobiology and neurodevelopmental impact of childhood traumatic stress and prenatal alcohol exposure. [14] Gostečnik, C. 2017. Družinske terapije in klinična praksa. Ljubljana: Brat Frančišek in Frančiškanski družinski inštitut. [15] Knauer, S. 2002. Recovering from sexual abuse, addictions, and compulsive behaviors: "Numb" survivors. New York, NY: Haworth Press. [16] Rothschild, B. 2000. The body remembers: The psychophysiology of trauma and trauma treatment. New York: W. W. Norton & Company. [17] Gostečnik, C. 2017. Relational family therapy: the systemic, interpersonal, and intrapsychic experience. New York; London, Abingdon: Routledge. [18] Dorais, M. 2002. Don't tell: The sexual abuse of boys. Quebec: McGill-Queen's University Press. [19] Schore, A. N. 2003. Early relational trauma, disorganized attachment, and the development of a predisposition to violence. V: D. J. Siegel in M. F. Solomon, ur. Healing trauma: Attachment, mind, body, and brain, 107-67. New York: W. W. Norton & Company. [20] Rothschild, B. 2006. Help for the helper: The psychophysiology of compassion fatigue and vicarious trauma. New York: W. W. Norton & Company. [21] Repič Slavič, T. in Gostečnik, C. 2017. Relational family therapy as an aid toward resolving the trauma of sexual abuse in childhood in the process of separation in the couple relationship. Journal of marital and family therapy 43, št. 3:422-434. [22] Scaer, R. 2005. The trauma spectrum: Hidden wounds and human resiliency. New York, NY: W. W. Norton & Company. [23] Simonič, B. 2018. Nežnost in njen pomen v medosebnih odnosih in v pastorali. Bogoslovni vestnik.78, št. 1:209-218. [24] Levine, P. A. 1997. Waking the tiger: Healing trauma: The innate capacity to transform overwhelming experiences. Barkeley, CA: North Atlantic Books. 115 Medgeneracijski prenos nasilja Prof. dr. Christian Gostečnik Teološka fakulteta, katedra za Zakonsko in družinsko terapijo ter psihologijo in sociologijo religije Poljanska 4, Ljubljana Frančiškanski družinski inštitut Prešernov trg 4, Ljubljana christian.gostecnik@teof.uni-lj.si POVZETEK knjigi Upor telesa [20], ko piše, da se zloraba v otroštvu razrešuje Vsak emocionalni odziv na stresne situacije vedno vpliva na na dva načina: odrasli, ki je bil kot otrok spolno zlorabljen, svoja celotno psiho-organsko strukturo posameznika. Vsekakor pa le-to nepriznana čustva, psiho-organske vsebine prenese na svojega najbolj zaznamujejo nasilna, travmatična doživetja, ki se lahko otroka oziroma na druge ljudi okoli sebe, lahko pa posledice zaradi svojih destruktivnih vplivov, prenašajo iz generacije v plačuje telo zlorabljenega, in sicer s psihosomatiko ali kroničnimi generacijo. Nasilna, travmatična doživetja s senzacijami in afekti boleznimi [21]. Tako sekundarna travmatizacija kot tudi se namreč usidrajo v posameznikovo psihosomatsko, psiho- transgeneracijski prenos pripomoreta k pogostemu razvoju nizke organsko strukturo oziroma v njegov implicitni spomin in tam samopodobe otrok [4]. Prav to je raziskovalce še bolj motiviralo, nezavedno bivajo ter lahko dramatično vplivajo na naslednje da niso raziskovali le mehanizmov in učinkov prenosa travme, rodove. V tem prispevku bomo zato skušali prvenstveno pokazati, temveč tudi nezavedno in zavedno komunikacijo med starši in da težja ko je travma, nasilje, močneje posameznika sili k otroki. Ugotovili so, da se je komunikacija lahko razlikovala od zunanjemu izrazu ali ekspresiji in to še prav posebej tedaj, ko gre skoraj popolne tišine do odprtega razpravljanja in pripovedovanja za PTSD, katerih posledice pa lahko zato doživljajo tudi bodoče najbolj krutih zgodb in izkušenj. Otroci so v obeh primerih zatirali generacije. in potlačili svoja čustva in niso bili gotovi, kaj se je v resnici dogajalo z njihovimi starši. [2, 3, 15, 22, 23] Poleg so tudi Ključne besede ugotovili, da je prekomerna komunikacija in z njo povezano travmatična doživetja, implicitni spomin, prenos nasilja, PTSD, razlaganje vseh izkušenj pri otrocih povzročala manj depresije in relacijska družinska terapija anksioznosti, vendar več krivde. Krell [24] trdi, da je bilo za otroke še bolj strašljivo, če starši o svojih izkušnjah niso govorili, 1. UVOD saj so si otroci v svojih glavah s pomočjo fantazije ustvarili Mnogi v sodobnosti, kot npr. [1-5], v svojih raziskavah odkrivajo določene slike, ki so bile zelo patogene. Tako so otroci nezavedno transgeneracijski prenos travme v naslednje generacije in lahko uprizarjali in ponovno doživljali usodo staršev, in če jim je ob tem celo na otroke, ki se sploh še niso rodili, ko se je nasilje zgodilo primanjkovalo informacij in zgodb, je bilo to zanje lahko še hujše [6]. Nekateri avtorji [7, 8] pri tem ocenjujejo, da npr. približno in intenzivnejše [5]. Vsi ti prenosi čutenj in afektov, psiho- ena tretjina spolno zlorabljenih oseb naprej zlorablja, približno organskih vsebin pa se odvijajo prek mehanizmov projekcijsko- dve tretjini žrtev pa ne bo nikoli spolno zlorabljalo naprej, introjekcijske identifikacije in kompulzivnega ponavljanja. predvsem če bodo imele podporo in ljubezen v družini. Johnson [9] ugotavlja, da imajo tisti, ki so bili v preteklosti žrtve čustvene, 3. TRAVMA IN NJENA fizične ali spolne zlorabe, 6-krat večjo možnost, da bodo tudi sami SIMPTOMATOLOGIJA nadaljevali zlorabo, ki so jo izkusili na lastni koži. Druge Tipični znaki travme se odražajo (2, 3, 5, 15-17, 23, 25, 26] raziskave Collin-Vezina [10] ugotavlja, da je bila polovica mater, predvsem hipervzburjenju v AŽS, le-ta pa je rezultat katerih otroci so bili spolno zlorabljeni, tudi sama žrtev spolne travmatičnega doživljanja. Tu govorimo predvsem o hitrejšem zlorabe. In tudi če se dejanje nasilja ne prenaša naprej, to še ne bitju srca, srčni palpitaciji, hladno-vročih potenjih, hitrejšem, a pomeni, da bodo otroci staršev, ki so doživeli nasilje, varni pred bolj plitvem dihanju, hiperbudnosti ali pazljivosti, osebami, ki izvajajo nasilje. mravljinčavosti, strahu, jezi, avotdestrukciji itd. V kroničnih stanjih pa lahko ti simptomi vodijo v motnje spanja, depresijo, 2. RELACIJSKA DRUŽINSKA TERAPIJA izgubi apetita, anksioznosti, spolnim disfunkcijam, zamrznjenosti, Relacijska družinska teorija [11-14] govori o nepredelanih afektih, disociaciji in velikokrat lahko ima ta posameznik težave s psiho-organskih vsebinah, ki so pri nasilju predvsem strah, prezir, koncentracijo in posledično s spominom [2, 3, 15, 23, 27-29]. gnus, sram in jeza, ki se globoko vtisnejo v psiho-organsko Tako te žrtve lahko ponovno doživljajo travmatični dogodek v strukturo posameznika, le-te se prek mehanizma projekcijsko- senzornih oblikah ali »flashbackih« in se zato začno izogibati introjekcijske identifikacije vertikalno prenesejo s travmiranega vsemu, kar bi jih lahko spominjalo na travmo, ali pa doživljajo starša na otroka [2, 3, 15-17]. Četudi skuša ta starš otroka kronično hipervzburjenje v avtonomnem živčnem sistemu [16, 17, opozoriti na vse nevarnosti nasilja [18], sam pri sebi pa ne bo v 25, 30-33]. stiku z nepredelanimi afekti ter se zato tudi ne bo znal zavarovati in postaviti razmejitev, bo veliko večja verjetnost, da otrok Z drugimi besedami, na ta način lahko proces disociacije odcepi postane žrtev nasilja [11, 13, 14, 19]. Podobno pravi Millerjeva v tako narativne komponente doživetja kakor tudi sosledje dogajanja, vključno s fiziološkimi in psihološkimi skratka psiho- 116 organskimi reakcijami, kar pa pomeni, da ostaja implicitni disociirana in še dolgo potem, ko se je travma zgodila, še vedno oziroma organski spomin globoko vrisan v posameznikov čuti, da je »izven sebe«. To stanje pa se, kot rečeno, lahko organizem in le-ta se lahko prenese v naslednjo generacijo [4]. nadaljuje še dolge mesece in leta, včasih pa se zgodi, da se šele Govorimo torej o amneziji, ki je v različnih oblikah, poleg ostalih čez leta prebudi, in sicer v obliki otopelosti, zamrznjenosti, simptomov, lahko vsekakor najpogostejši pojav pri disociativni anksioznosti, »flashbackov«, depersonalizacije, delne ali popolne dinamiki, zato se to nezavedno, psiho-organsko prenaša na amnezije, doživljanja, kot da je žrtev izven telesa, nezmožnosti naslednje generacije. Posameznik lahko doživi anestezijo in ne kar koli čutiti, pa tudi v obliki nerazumljivih vedenjskih in čuti nobene bolečine, lahko vsa občutja in afekte odcepi, zopet emocionalnih reakcij, ki nimajo nobenega pravega razloga v drugi lahko izgubi zavest ali pa čuti, kakor da je izven telesa. konkretnih situacijah. Vse bolj pa raziskave ugotavljajo, da je Lahko pa vse to pozneje v najrazličnejših oblikah zaznavajo v disociacija tudi vedno zvesti spremljevalec PTS- in PTSD- naslednjih rodovih [5, 25, 34]. simptomov in s tem v zvezi paničnih napadov ter napadov 4. PONOVNI VZNIK TRAVME V anksioznosti [2, 3, 15-17, 23, 25, 39], kar ponuja izjemno možnost, da se travme preteklosti, zaradi svoje intenzitete, ki NASLEDNJIH GENERACIJAH formira PTSD simptomatologijo, ponovno prebudijo v Podoživljanje travmatičnih dogodkov pa vedno znova omogočajo naslednjih generacijah, in sicer v najrazličnejših oblikah in »flashbacki«, in sicer lahko v celoti ali pa samo v določenih delih; intenzitetah. le-ti se lahko pojavljajo v naslednjih generacijah in to brez očitnih 5. ZAKLJUČEK travm, ki bi jih ti posamezniki v naslednjih generacijah doživeli [2, 3, 15, 23]. Ti fenomeni so lahko izredno moteči, saj se Za sklep lahko rečemo, da vsako zavestno doživljanje posameznik, ki je doživel travmatični dogodek, ob teh posameznika sestavljeno iz več komponent. Celoten spomin »flashbackih« počuti tako, kot da se celoten travmatični dogodek nekega določenega travmatičnega dogodka zato vedno vključuje ponovno in v vsej intenziteti ponavlja, čeprav se je travma že integrirani priklic vseh elementov posameznikovega doživljanja, zdavnaj končala. Še veliko bolj pa so vznemirjajoči za ki ga sestavljajo: senzacije, slike, vedenje, afekti in pomen. Kadar posameznika v naslednji generaciji, ki nima nobenega spomina, situacija ni preveč stresna, vsekakor pa, kadar je prijetna, se ti da bi se mu karkoli tega zgodilo, občuti pa simptome travme. elementi medsebojno povežejo oziroma ostajajo nedotaknjeni v Drugače povedano, posameznik, ki ni doživel travme, loči, ki ne spominu. Tako se npr. posameznik lahko v vsem spominja lepih nujno pozna PTS in PTSD-travmatičnih spominov, saj so le-ti in vzhičenih trenutkov svojega življenja, spominja pa se tudi čisto odcepljeni že v samem originalu, se pa zato enostavno lahko navadnih dogodkov, npr. izleta, kosila, obiska itd. Pri tem se pojavljajo v vsakem času in se v obliki »flashbackov« vedno posameznik lahko spomni vseh pomembnejših dejanj oziroma lahko vrinejo v sedanjost [16, 25, 35]. Te »flashbacke« ponavadi vedenj, prikliče si sliko, še pozna vse bistvene senzacije, znani so spremlja tudi zelo močna anksioznost, ki je v originalu spremljala mu afekti, ki so jih ta doživetja ustvarila oziroma prebudila, ve pa tudi travmatično doživetje posameznika. Še posebej močno tudi za pomen – bil je npr. sproščen dan, poln lepega, svežega, moteči pa so ob tem še panični napadi, ki lahko žrtev tudi po sklenili so nova znanstva. dolgih letih, ko travme ni več, še vedno spremljajo in velikokrat Kadar pa je stres prevelik oziroma ko govorimo o nasilju in čisto ohromijo tudi naslednjo generacijo. travmah, se ti elementi med sabo cepijo oziroma disociirajo. Pri tem gre za poskus posameznika (njegovega organizma in tudi Seveda pa pri tem ni nujno, da vsaka, tudi zelo stresna situacija njegove psihe), da bi pobegnil, ko boj ali upor nista več mogoča povzroči disociacijo. Vendar, vedno kadar je stres ali travma v [16, 25, 36]. Vsekakor pa gre lahko tu tudi za zamrznitev, ki je še svoji intenziteti premočna, se aktivira dissociativni sistem, ki tretja oblika odziva na travmo. Tudi v tem primeru gre za zavaruje posameznika pred dekompenzacijo. Tedaj lahko disociacijo, in sicer največkrat v najgloblji obliki, saj se žrtev v govorimo o PTSD simptomatologiji, ki pa lahko vedno pomeni teh primerih zelo malo ali pa sploh ne spominja, kaj se je v resnici žarišče travmatičnih organskih spominov, ki se neverjetno zgodilo. V tem primeru lahko gre za zelo globoko stanje amnezije silovitostjo prenašajo v naslednje generacije. ali, kot rečemo, čistega odklopa, vendar pa v posamezniku kljub temu ostajajo elementi travme, in sicer v najrazličnejših afektih, " Doseženi rezultati so delno nastali v okviru projekta št. J5-9349, kot so strah, sram, jeza, ki se vedno lahko prebudijo ob podobnih situacijah, ki npr. samo bledo spominjajo na originalno travmo. V ki ga je financirala Javna agencija za raziskovalno dejavnost posamezniku ostajajo t. i. žarišča, ki še vedno tlijo, in to »tleče Republike Slovenije iz državnega proračuna.” stanje« žrtev travme včasih še veliko bolj vznemirja, še posebej tedaj, ko se pojavlja v obliki paničnih napadov ter izjemno 6. Reference močnih anksioznosti ob določenih spodbujevalcih ali sprožilcih, ki so spremljali originalno travmo [37, 38] in se lahko, kot rečeno, [1] Epstein, H. 1979. Children of the Holocaust: Conversations ponavlja v naslednjih generacijah. with sons and daughters of survivors. New York, NY: Penguin. Če se povrnemo k fenomenu disociacije, potem vidimo, da žrtve [2] Barocas, H. A. in Barocas, C. B. 1973. Manifestations of med travmatičnimi doživetji izkušajo najrazličnejša občutja: da so Concentration Camp Effects on the Second Generation. zapustile telo, da se je čas ustavil, da so zamrznile, omrtvele in American Journal of Psychiatry, 130(7), 820–821. ničesar več doživljale. Žrtev je lahko tudi zelo omejeno videla [3] Lev–Wiesel, R. 2007. Intergenerational Transmission of puško, mučilno orodje, samo silhueto rablja ali napadalca, luč ali Trauma across Three Generations: A Preliminary Study. avto, ki sta se silovito približevala, in se potem ničesar več ne Qualitative Social Work: Research and Practice, 6(1), 75– spomni do prebujenja v bolnišnici, ko je bilo okrog nje polno ljudi 94. v belih haljah itd. [2, 3, 15-17, 23, 25]. Ostaja pa organski, [4] Woollett, N. in Thomson, K. 2016. Understanding the implicitni spomin, ki se lahko vedno znova prebudi v intergenerational transmission of violence. South African najrazličnejših oblikah vedenj, čutenj, fobij in paničnih napadov. Medical Journal 106(11): 1068-1070. Žrtev je ponavadi tudi po travmatičnem doživetju še vedno 117 [5] Dekel, R. in Goldblatt, H. 2008. Is There Intergenerational Israel. International Journal of Social Psychiatry 35, št. Transmission of Trauma? The Case of Combat Veterans’ 3:265–273. Children. American Journal of Orthopsychiatry.78(3). 281- [23] Schore, A. 2003. Affect regulation and the repair of the self. 289. New York, NY: W. W. Norton & Company. [6] Schore, A. N. 2016. Affect regulation and the origin of the [24] Krell, R. 1979. Holocaust families: The survivors and their self: The neurobiology of emotional development. New children, Comprehensive Psychiatry 20, št. 6:560-568. York, NY: Taylor & Frances/Routledge. [25] Rothschild, B. 2006. Help for the helper: The [3] Schore, A. N. 2012. The science of the art of psychophysiology of compassion fatigue and vicarious psychotherapy. New York, NY: W. W. Norton & Company. trauma. New York, NY: W. W. Norton & Company. [4] Steele, K. Boon, S. in van der Hart, O. 2017. Treating [26] Rothschild, B. 2003. The body remembers casebook: trauma-related dissociation: A practical, integrative Unifying methods and models in the treatment of trauma approach. New York, NY:: W. W. Norton & Company. and PTSD. New York, NY: W. W. Norton & Company. [5] Burton, M. S., Feeny, N. C., Connell, A. M. in Zoellner, L. [27] Cohen, J. A., Mannarino, A. P. in Deblinger, E. 2006. A. 2018. Exploring evidence of a dissociative subtype in Treating trauma and traumatic grief in children and PTSD: Baseline symptom structure, etiology, and treatment adolescents. New York, NY: Guilford Press. efficacy for those who dissociate. Journal of consulting and [28] Kitayama, N., Quinn, S. in Bremner, D. J. 2006. Smaller clinical psychology 86, št. 5:439. volume of anterior cingulate cortex in abuse-related [6] Catherall, R. D. 2004. Handbook of stress, trauma, and the posttraumatic stress disorder, Journal of Affective Disorders family. New York, NY: Brunner-Routledge. 90, št. 2-3:171–174. [7] Kaufman, G. 1989. The psychology of shame: Theory and [29] Haagen, J. F., van Rijn, A., Knipscheer, J. W., van der Aa, treatment of shame-based syndromes. New York, NY: N. in Kleber, R. J. 2018. The dissociative post-traumatic Springer Publishing Co. stress disorder (PTSD) subtype: A treatment outcome [8] Hanson, R. K. in Slater, S. 1988. Sexual victimization in the cohort study in veterans with PTSD. British Journal of history of sexual abusers: A review. Annals of Sex Research Clinical Psychology 57, št2:203-222. 1, št. 4:485-499. [30] Siegel, D. J. 2007. Mindsight: The new science of personal [9] Johnson, S. M. 2002. Emotionally focused therapy with transformation. New York, NY: Bantam Books. 2011. trauma survivors: Strenghtening attachment bonds. New [31] D. J. Siegel. The mindful brain: Reflection and attunement York, NY: Guilford Press. in the cultivation of well-being. New York, NY: W. W. [10] Collin-Vézina, D. in Cyr, M. 2003. La transmission de la Norton & Company. violence sexuelle: description du phénomène et pistes de [32] Siegel, D. J. 2017. Mind: A Journey to the Heart of Being compréhension. Child Abuse & Neglect 27, št. 5:489-507. Human. W.W. Norton. [11] Gostečnik, C. 2017. Relational family therapy: The [33] Siegel, D. J. 2015. The developing mind: How relationships systemic, interpersonal, and intrapsychic experience. New and the brain interact to shape who we are. Guilford York, NY: Routledge/Taylor & Francis Group. Publications. [12] Gostečnik, C. 2011. Inovativna relacijska družinska [34] Austern, D. J. 2017. Beyond Binary Thinking: Providing terapija: inovativni psiho-biološki model. Ljubljana: Brat Best Practice Treatment to Veterans with PTSD, Pragmatic Frančišek in Frančiškanski družinski inštitut. Case Studies in Psychotherapy 13, št. 2:154-172. [13] Gostečnik, C. 2005. Psihoanaliza in religiozno izkustvo. [35] Hyland, P., Shevlin, M., Hansen, M., Vallières, F., Murphy, Ljubljana: Brat Frančišek in Frančiškanski družinski J. in Elklit, A. 2016. The temporal relations of PTSD inštitut. symptoms among treatment-seeking victims of sexual [14] Gostečnik. 2004. Relacijska družinska terapija. Ljubljana: assault: a longitudinal study. Journal of loss and trauma 21, Brat Frančišek in Frančiškanski družinski inštitut. št. 6:492-506. [15] Schore. A. N. 2003. Affect dysregulation and disorders of [36] Åkerblom, S., Perrin, S., Fischer, M. R. in McCracken, L. the self. New York, NY: W. W. Norton & Company. M. 2017. The impact of PTSD on functioning in patients [16] Rothschild, R. 2017. The body remembers volume 2: seeking treatment for chronic pain and validation of the Revolutionizing trauma treatment. New York, NY: W. W. posttraumatic diagnostic scale. International journal of Norton & Company. behavioral medicine 24, št. 2:249-259. [17] Rothschild, B. 2010. 8 keys to safe trauma recovery. New [37] Shipman, K. L.,Schneider, R., Fitzgerald, M. M., Sims, C., York, NY: W. W. Norton & Company. Swisher, L. in Edwards, A. 2007. Maternal emotion [18] Howell, E. F. 2002. “Good Girls,” Sexy “Bad Girls,” and socialization in maltreating and non-maltreating families: Warriors: The Role of Trauma and Dissociation in the Implications for children's emotion regulation. Social Creation and Reproduction of Gender. Journal of Trauma Development 16, št. 2:268–285. & Dissociation 3, št. 4:5-32. [38] Lokshina, Y. in Liberzon, I. 2017. Enhancing efficacy of [19] Gostečnik, C. 2008. Relacijska paradigma in travma. PTSD treatment: role of circuits, genetics, and optimal Ljubljana: Brat Frančišek in Frančiškanski družinski timing. Clinical Psychology: Science and Practice 24, št. inštitut. 3:298-301. [20] Miller, A. 2005. Upor telesa: telo terja resnico. Ljubljana: [39] Johnson, E. M. Possemato, K., Barrie, K. A., Webster, B., Tangram. Wade, M. in Stecker, T. 2017. Veterans’ Thoughts About [21] Ogden, P. in Fisher J. 2015 Sensorimotor psychotherapy: PTSD Treatment and Subsequent Treatment Utilization. Interventions for trauma and attachment. New York, NY: International Journal of Cognitive Therapy 10, št. 2:138- W. W. Norton & Company. 160. [22] Okner, D. F. in Flaherty, J. F. 1979. Parental communication and psychological distress in children of Holocaust survivors: A comparison between the U.S. and 118 The Other Side of Neurotechnologies Kristian Elersič ekristian238@gmail.com ABSTRACT As a student of science, I am mainly surrounded by scientific As a response to the overly-optimistic view of technology, I will literature and scientists. What I have found is that there is a general present some of the obstacles when using new neurotechnologies, lack of interest in interdisciplinary discussion of impacts that specifically cognitive enhancers. Those obstacles come in a form techno-scientific progress has on a society. It almost seems that of physical limitations, such as not being able to develop methods there prevails an opinion of that technology and science cannot be sophisticated enough that they would have no (or minimal) negative misused – over-optimism or maybe even ignorance. To clarify side-effects, and in a form of ethical limitations, since even if these further, this opinion comes from my student surroundings – new technologies were physically possible, the extent to which we University of Ljubljana, Biotechnical faculty. I haven’t done any should use these technologies is not fully explored. To fully analysis of other universities. Of course, this interdisciplinary topic consider such obstacles, I propose an outline of the ethical is not strictly a scientific one, but nonetheless it is important. framework that should be used when thinking of applying new, Therefore, I think that discussions of this kind should be included future or even old cognitive technologies. Furthermore, I pose some in the scientific community, especially when it comes to questions for further analysis of the intertwining of techno- scientific progress with social dynamics. biotechnology that directly addresses and modifies the human condition. On the other hand, beside lack of interest of my academic surroundings, there is also an emerging movement that is overly- Keywords optimistic about technology. It is called transhumanism. Max More, Neurotechnology, transhumanism, enhancement, cognition, ethics philosopher and futurist, defined it: ” … [as] both a reason-based philosophy and a cultural 1. INTRODUCTION movement that affirms the possibility and desirability of Neurotechnology, technology that has a fundamental impact on fundamentally improving the human condition by means how we understand the brain, cognitive processes and of science and technology. Transhumanists seek the consciousness, is rapidly developing. Some of the examples of continuation and acceleration of the evolution of neurotechnologies are: imaging technology such as magnetic intelligent life beyond its currently human form and resonance imaging (MRI), electroencephalogram (EEG) or human limitations by means of science and technology, positron emission tomography (PET), brain stimulations such as guided by life-promoting principles and values.” [1] transcranial magnetic stimulations (TMS) or transcranial direct A few of the common characteristics of the mentioned philosophy current stimulation (tDCS), implant technologies, pharmaceuticals, are: a) optimistic view of progress of science and technology, b) gene editing, cell therapy and brain-computer interfaces. In this aiming at the advancement of humans - to alleviate our biological article I will be focusing on technologies that aim to improve the constrains so that we could live longer and healthier, c) using cognitive processes - cognitive enhancers. There is not a specific science and technology as the main means of advancement and d) technology that suits this purpose only. It is rather a combination of claiming that enhancement will transform the whole meaning of different technologies that are believed to alter cognitive processes: being human. In other words, they state that technology will help gene editing, pharmaceuticals and brain stimulations. us as a humanity to overcome our current problems, it will help us to be more ethical, healthy, well in both economic and ecologic My whole stance is formed as a response to the overly-optimistic sense. Transhumanists also believe that by doing so we will view of technology – which is that technology can save many of transform ourselves into some future form where we will no longer our problems, if not all of them. At no point in this paper will I go be able to state we are the same as we were before as species – we into detailed discussion of any topic, but will rather open a broad will become post-human in a transhumanist sense. field of questions that are important to consider when analysing neurotechnology and human enhancement. In the first section of For this purpose I want to analyse and present some questions this paper, I will describe what I mean with the term “overly- related to the human bio-enhancement, specifically cognitive optimistic view of technology”. In the second section, I will enhancement. My aim is not to discredit latest scientific describe shorty what are the cognitive enhancers. Then I will development. New knowledge and new technologies are continue by presenting some of the physical limitations of the appreciated – but with them comes a responsibility of how and enhancers, namely the limitations for gene editing, pharmaceuticals when to use them. My aim is to stimulate an interdisciplinary and brain stimulations. Then I will set the ethical framework that discourse of the mentioned subject – human enhancement and will serve the purpose of considering or questioning any neurotechnologies. In other words, I want that we take a closer look intervention. Finally, I will try to think of the possible connections at the other, darker side of new neurotechnologies that aim at of the techno-scientific progress with society. enhancing humans, because the bright side is known to well. 2. OVERLY-OPTIMISTIC VIEW OF 3. COGNITIVE ENHANCERS TECHNOLOGY AND ENHANCEMENT Cognitive enhancers are technologies that enhance cognitive processes. Like doping in sports that enhances physical 119 performance, cognitive enhancers stimulate memory, perception, is that substances often have side effects such as nausea, vomiting concentration, motivation, learning, problem solving and or cramps. The second obstacle is that brain responds to the computation. [2 3] Most widely known are the so-called “smart chemicals in such a way that a too high or too low concentration of drugs” – cognitive enhancers in the form of pharmaceuticals substance might not produce optimal brain functioning effects.2 In (substances). Those are already used for medicinal and enhancing some individuals one dose may cause a positive effect while in the purposes. [3 4] Besides that, scientists are developing methods for others the effect would be negative. [3] The third obstacle is that a cognitive enhancement in the form of genetic engineering and brain drug may positively affect one brain function while simultaneously stimulations. In a broader sense, even traditional cognitive training negatively affect the other.3 Last but not least, it is important to add can be classified as one form of cognitive enhancers, but I will that we lack long term studies that would tell us how safe is the use rather focus on technological enhancement.1 of these drugs – in other words, what are the long-term side effects of such enhancers. [3 10 11] 4. PHYSICAL LIMITATIONS When describing neurotechnological interventions for 4.3. Brain Stimulation enhancement purposes, we have mentioned three types of Brain stimulation is a technology using electrical currents or modifications: a) genetic engineering, b) drug intake and c) brain magnetic field (which induces electric currents) to stimulate certain stimulation. In this section I will present some of the (possible) parts of brain tissue. Methods of brain simulation can be further limitations of mentioned technologies. divided into invasive and non-invasive ones. The invasive method requires application of currents directly to the brain’s surface or 4.1 Genetic Engineering brain’s core. In order to achieve this, we have to be invasive - the Genetic engineering started in 1973 and has very quickly spread skull has to be opened and sometimes even the brain must be cut. onto many fields of industry and research. In our case, genetic Using the non-invasive methods, the currents or magnetic fields are engineering would be used for manipulation of certain cognitive applied over the head’s surface. This is called non-invasive brain trait like attention, IQ, memory and so on. For example, we could stimulation (NBS), examples of which are transcranial magnetic genetically modify embryo’s DNA, or later in life insert some gene stimulation (TMS) and transcranial direct current stimulation products to tissues or organs (RNA or protein insertions), or “knock (tDCS). For our further discussion it is not of a great importance out”, that is, make inoperative, some genes (gene knockout with what kind of technique is used. DNA cutting proteins), or we could select one out of many embryos with the most preferable genome (prenatal diagnostic or embryo Opposition to the overly optimistic view towards this form of selection). But when it comes to genetics, we often misinterpret that cognitive enhancement may come from understanding brain’s there is a direct link between a gene and a trait. We have to bear in functioning. Key concept in this opposition is net zero-sum mind that only a few traits are monogenetic, meaning that there is framework proposed by Brem et al. (2014). [12] Authors state that only one gene that influences a certain trait. In most cases, there are this concept is grounded on the physical principle of conservation multiple genes that affect one trait. Besides that, in many cases one of energy in a closed system. Furthermore, they clarify that, in this gene also affects many traits. [6] Due to this phenomenon (complex notion, brain operates within the constraints of a finite amount of genetic traits) it is very hard to predict the total outcome of such energy and processing power. Not meaning that all the brains and genetic manipulation. This is nicely illustrated by the study when the same brain through the life span do have equal amount of energy researchers improved mouse’s memory and simultaneously and processing power, but that an x-amount of energy and increases its sensitivity to pain. [7 8 9] processing power is distributed throughout the brain at any given time. When demands shift, so does the distribution of energy and 4.2 Pharmaceuticals processing power. Most importantly, total sum of brain’s activity stays the same. In other words, brain operates in net zero-sum Enhanced cognition can be also achieved with numerous framework. If this prediction is correct, NBS would enhance some substances, even with dietary supplements, such as caffeine and brain function, but at the same time degrade other nicotine, but for our purpose I will describe pharmaceuticals (which (compensation/cost). Some evidence accounting for this hypothesis are classified as technological interventions). One of such comes from the phenomena of paradoxical functional facilitation. pharmaceuticals is methylphenidate, commercially known as [12 13] This phenomenon appears when damage to an intact area Ritalin. Ritalin is nowadays widely used not just for treating the of the brain normalizes a previously reduced level of functioning attention deficit hyperactivity disorder (ADHD), but is popular as a (compensation for previous loss – re-establishing balance between cognitive enhancer among students/academics with the purpose of functional brain networks) or when a patient with damage performs achieving better study or academical results. [3 4] Other similar a certain task better than before or better than healthy control substances are: amphetamines, modafinil, atomoxetine, reboxetine, subjects (energy and processing power of damaged brain tissue was donepezil, galatamine, rivastigmine and memantine. When distributed to other brain parts that are now enhanced). Authors prescribing these drugs it is important to take into account obstacles conclude with raising bioethical concerns: “Is it acceptable to that may come with the use of these substances. One such obstacle 1A special kind of cognitive enhancers are moral enhancers that are 3 Example: when rigastimin improves learning on a motor task more strictly concerned with improving our moral behaviour. [5] and making associations between symbols and digits, but can at 2 This applies for future drugs too. This molecular dynamics is the same time impair verbal and visual episodic memory. [3 10] called U-shaped relationship between substance and receptor. [3] 120 improve certain brain functions at the cost of others and can we take life – long and healthy with no negativity), or a product of the responsibility for its impact on the individual and on society?” advertisement or political viewpoint. We ought to analyse all of the (p.14-15) They also state that “current neuroenhancment studies mentioned subfields before interfering with human nature with the emphasize positive outcome of specific functions and concentrate use of enhancers to the extent that would limit our capacity undo on individual improvements, while related topics such as risk and the damage. safety, as well as social and moral factors are neglected or restricted to specific inquiries.” (p.15) [12] 6. CONCLUSION 5. ETHICAL FRAMEWORK AND Development of new technologies opens up many questions – from physical limitations to ethical considerations. I have focused on INTERTWINING OF TECHNO- neurotechnologies, specifically cognitive enhancers. My general SCIENTIFIC PROGRESS WITH SOCIETY claim is that there is an overly-optimistic view of technology that In the previous section I have described some physical limitations states that technology can solve many, if not all, of our problems. that we may encounter when applying neuroenhancing Hence, we can enhance human beings to be better. Doubt arises, technologies. Now I want to state that even if we do develop better not only because of physical limitations of modifying technologies, technology, we still have to consider questions from a very broad such as in genetic engineering, drug intake and brain stimulation, spectrum of topics. but even more so because of ethical issues. Human condition is a very complex one and it has been, even if this does not seem so, Firstly, I want to propose an ethical framework consisting of three analysed too little for us to drastically modify our nature. In this points that we have to consider in a discussions about any human manner, we need an interdisciplinary and public discourse for interventions such as bio-enhancing technologies. (1) First is about discussions about mentioned topics – intertwining of modern personal freedom. (2) Second is about the line between long-term values, human condition, society and techno-scientific progress. sustainability and short-term solutions. And (3) third is about ecological integrity/sustainability. For example: 1) do enhancers 7. REFERENCES that aim to limit our immoral behaviours, limit our personal [1] Max More: True Transhumanism: A reply to Don Ihde, freedom? Or, do cognitive enhancers prescribed to children for Transhumanism and Its Critics (2010) p.138 enhancement of a specific trait, limit their freedom to [2] N.Bostrom and A. Sandberg, Cognitive Enhancement - choose/develop other characteristic that they would love to have? Methods, Ethics, Regulatory Challenges, Sci Eng Ethics (2009) Or 2) are costs and side-effects of bio-enhancing accounted for? 15:311-341 Are those short term solutions sustainable or do they just prolong [3] Husain and Metha (2011) Cognitive enhancement by drugs in the problem or maybe even worsen the situation? Where is the health and disease. Trends in Cognitive Sciences, January 2011, boundary between enhancement and a disease – former without the vol. 15, No.1 need (but with the want), latter with the immediate need? And 3) is [4] Greely, H et al. Towards responsible use of cognitive-enhancing this technology ecologically sustainable? Does it produce a non- drugs by the healthy. Nature, 456(7223), 702–705 (2008). sustainable mentality – that which is not able to think in a non- [5] Juslian Savulescu adn Igmar Persson. Moral Enhancement, selfish way or in an ecological way or in terms of long-term Freedom and the God Machine. Monist. 2012 July; 95(3): 399-421 solutions? What will this technology do to overpopulation, species [6] Griffiths AJ, Wessler SR, Carroll SB, Doebley JF. Introduction extinction, how many waste will it produce? Those are just a few to genetic analysis (Eleventh ed.): The Inheritance of Complex questions stated to further clarify the mentioned ethical framework. Traits (p.715). New York, NY. (2015) [7] Simmons, D. Genetic inequality: Human genetic Secondly, I want to briefly mention how broad the field of engineering. Nature Education 1(1):173, 2008. investigation in this and similar topics should be. Nature of [8] Wei, F., et al. Genetic enhancement of inflammatory pain by questioning and regulating techno-scientific progress, especially forebrain NR2B overexpression. Nature Neuroscience 4, 164–169 when we think of interfering with our nature (enhancement), (2001) exceeds fields of natural sciences. It includes (or should include) [9] Tang, Y. P., et al. Genetic enhancement of learning and memory questions from philosophy of science (what is knowledge, what is in mice. Nature 401, 63–69 (1999) science, its place and role in modern and future society), philosophy [10] Wezenberg E. Modulation of memory and visuospatial of progress (genealogy of progress, what is progress), intertwining processes by biperiden and rivastigmine in elderly healthy of the techno-scientific progress and the economic system (how is subjects. Psychopharmacology (Berl.) 2005;181:582–594. modern free market system influencing techno-scientific progress, [11] Chris Zarpentine: 'The Thorny and Ardeous Path of Moral business of science), intertwining of the market and the Progress': Moral Psychology and Moral Enhancemen. Neuroethics values/ideological system (how does the market with advertising (2013) 6:141-153 influence the values and how do these values afterwards influence [12] Brem et al. Is neuroenhancement by noninvasive brain the market) and the application of technology in political systems stimulation a net zero-sum proposition? Neuroimage 2014 Jan 15; (possibility of new technologies for achieving political goals, 85(0 3): 1058-1068 possibility of manipulation and control). Furthermore, we have to [13] Kapur N. Paradoxical functional facilitation in brain- ask ourselves why do we want enhancement: is it a by-product of behaviour research. A critical review. Brain. 1996; 119(Pt economic system (competition – to be better and to get a better job), 5):1775–1790. or is it a by-product of modern values (for example, value of ideal 121 Braincrafting: why playing video games is good for you Katarina Fatur katarina.fatur@gmail.com ABSTRACT and pleasurable outcomes, where an action’s value is based on In the article I present cognitive, emotional and social benefits the outcome of that action, and where the evaluation whether the of playing video games, suggest simple guidelines for ethical outcome is good or bad might shift when new scientific evaluation of gaming, and offer an overview of findings which discoveries are made. It appears that this ethical framework fits support the use of video games for cognitive enhancement and neuroscience research better than other normative ethical social collaboration. theories, simply because the subjective virtuous conduct of game-designers (virtue ethics) or a scientists’ firm belief that Keywords undertaking experiments is the right thing to do (deontological video games, cognitive enhancement, attention, complex ethics) is of little or no importance in comparison to unwanted, problem solving, brain plasticity, learning, consequentialism but nonetheless possibly detrimental effects a person might suffer as the consequence of her or his invaded and changed 1. INTRODUCTION brain. In the past, research has been mostly focused on the negative The last conclusion partly rests on the notion of ‘fairness’ – it is effects of video games, but in the last decade, a growing number not fair that a player would be harmed for the sole purpose of of studies has supported the view that playing video games may advancing science, or because the game designer made a mistake significantly enhance our cognitive abilities and agility. These despite the designer’s moral character being impeccable. contradicting facts fuel the discussion on whether playing video Fairness can be fostered by assuming equality of all involved games is harmful or not. Here I focus on the benefits of gaming parties and by clear and honest communication, where the and offer an overview of findings which support using video gaming industry candidly informs the player about possible games as tools for cognitive enhancement and social effects. Informing the player is also the prerequisite for the collaboration, while also outlining the ethical framework which player to give their consent to undergo a benevolent cognitive is to be followed in order to avoid negative aspects of gaming. manipulation when playing a game. At the same time, the player should provide feedback, so that the industry can align and 2. PROPOSED ETHICAL FRAMEWORK improve their practice. This also involves the issue of privacy TO EVALUATE VIDEO GAMES protection, especially in the case of collecting neurodata in Up until a few years ago, research has been mainly focused on neurogaming1. the potential adverse effects of video games, such as increased Simply put, for gaming to be ethical, it has to benefit the player aggression [1] [2], addiction to gaming [3] and increased obesity and it has to fulfill three criteria: it should result in positive, risk [4]. While I do not believe that gaming is inherently bad, I useful outcome for the gamer and/or society, it should respect can understand the reservation and caution. For better or worse, and protect gamer’s privacy, and the gamer must participate playing video games significantly changes our brain [5], which knowingly and willingly. Since new technologies pose difficult carries serious implications. With our growing understanding of ethical questions and could be easily used for ill intentions, it is the brain comes greater responsibility about how to apply this important to show, why is it worth to undergo the gaming knowledge in a benevolent manner. If we consider video games as a tool for self-improvement and cognitive enhancement, they should align with (neuro)ethical guidelines which govern any 1 In neurogaming, the player's heart rate, brain activity, facial other means for cognitive therapy. Ethics is a set of principles expressions, voice, skin conductance, eye movement, pupil which prescribe what is right or wrong in terms of rights, dilation, and similar indicators are measured in order to use obligations, or benefits for individuals and society. The subfield the input to dynamically adjust the gaming content and of neuroethics, which is focused particularly on the issues in provide a completely personalized, immersive gaming neuroscience, is less than 20 years old. The concept was first experience, which takes into account the player's emotional introduced in 2002, when William Safire defined it as “the and cognitive state [7]. In this way, the desired (or feared) examination of what is right and wrong, good and bad about the cognitive manipulation is perfectly suited to the player. Some treatment of, perfection of, or unwelcome invasion of and game engines allow the players to navigate the game merely worrisome manipulation of the human brain” [6]. Video games by directing their gaze or blinking. Future development also possibly constitute a case of ‘worrisome manipulation’ and open includes different techniques which will simulate certain a window for ‘unwelcome invasion’ of the brain, but at the same sensations, e.g. the player holding an object in their hand time they can also act as one of the most pleasurable and safe while there is no object there [7]. The ethical concerns are brain enhancement tools. Do benefits warrant the dangers? significant, but in this article I mainly focus on the benefits of To do good, we first have to determine what is good. The ethical gaming and mention neurogaming merely as an interesting framework to help us evaluate the ‘goodness’ of video games in new branch of the industry, and my purpose is to show, why it this article is an epicurean-like teleological/consequentialist is worthwhile that we acknowledge and tackle these ethical theory of ethics with a bit of pragmatism, where we pursue good issues to be able to benefit from our gaming experience as much as we can. 122 experience despite the risk, or even better, while avoiding the motor control was shown in 2016 [14]. Players who played risk. In the following sections I offer an overview of beneficial action games for 5 hours per week over 6 months were later re- uses of video games and discuss how they affect the brain. tested for their driving abilities and they exhibited better lane- keeping and visuomotor-control skills. Cognitive gains acquired 3. BENEFITS OF VIDEO GAMES in gaming are thus transferable to real-world visual ability. Video games have become increasingly diverse and complex, Another example of induced neural plasticity in the players’ (hyper)realistic, and socially engaging. It has been shown that visual system is the case of adult amblyopia (‘lazy eye’) playing certain types of video games may enhance perceptual, patients, whose fundamental visual functions improved from 16- motor, social, emotional and cognitive learning [8]. While 54% after playing action video games [15]. It was a small scale scientists still do not know exactly how different brain regions pilot study, but researchers are optimistic about the possibility to interact, or what is the role of each and every particular apply the principles to treating other cortical dysfunctions as neuroanatomical element, some hypotheses have been made in well. Additional benefits of playing action games in relation to studies researching the impact of gaming on brain plasticity. vision are improved visual attentional skills in dyslexic children Below I present the supporting studies in more detail. [16]. Without any direct reading training, the reading abilities of these children improved significantly after only 12 hours of 3.1 Learning playing action video games. At a neuroethics conference in 2013 C. Shawn Greene explained that the adult brain does not want to learn anything new, because 3.3 Multi-tasking learning entails costly brain tissue restructuring [9]. So, how do Each time we switch our attention from one task to another we games facilitate learning and foster neuroplasticity? An pay a small cognitive cost and playing action games helps to incentive for the brain to more readily learn something new is reduce this cost [9]. In our everyday life we often have to the possibility of a reward at the end [9]. A study by Shaowen multitask, and the cumulative time spent in task switching is Bao and his team has shown that when stimulation of the relevant. The positive effects that action games training has on dopaminergic system occurs at the same time as a particular tone the brain is lasting and can still be observed months after the last is played, over time a larger area of auditory cortex is allocated gaming session. This effect is also applicable in serious, to process that particular frequency of sound [10]. The brain complex real-life events such as surgery. When novice surgeons structurally changes itself in order to more accurately and played action games for prescribed period of time, the result was quickly process the input that will reward it with dopamine a significant improvement in their ability to perform surgery release. Games foster learning because they activate the reward [17]. Even more surprisingly, surgical skills could be better system. predicted by their screen time than by the number of surgeries performed, or even the amount of time spent practicing the Circumstances which bathe the brain in rewarding hormones are surgery. also excitement and exploratory behavior [11], which both abound in challenging, but ultimately safe virtual environment 3.4 Social and emotional support of your favorite game. Learning is further encouraged by the Multiplayer online games which engage thousands of people fact that you can safely fail – paired with novelty and flexibility from all over the globe are probably the most exciting, of the play, safe environment promotes creativity [12]. In rewarding and productive development in the realm of video addition, the immediate feedback evaluating the player’s game games. Immersive social contexts not only provide a playground (leveling-up, points, new abilities and tools), present a perfect to develop social skills, but also serve as a solid source of ground to acquire an ‘incremental theory’ of players’ ability, emotional support and resilience, which I present in more detail where the players believe that they can change and actively below. improve their skills by investing more time and effort [8]. How is this useful in real-life scenarios? The learned skills can be Massive multiplayer online games (MMOGs) encourage the transferred, as illustrated by the examples below. development of online communities, with associated attachments and social rituals [18]. These communities represent 3.2 Visual attention a suitable model for a variety of human societies [19], because Green explains that action games exhibit particular qualities, they exhibit a strong sense of social connection, identification which sharpen a gamer’s visual attention [9], such as: complex with other members of the group and a sense of distinguishing 3D environment, fast motion, transient visual stimuli, and heavy themselves from other groups, as well as a strong commitment perceptual load with several events occurring simultaneously, to the group [18], [19]. On the continuum from Gemeineschaft where not all events are important. 15 years ago Green and (communal society) to Gesellschaft (associational society) [20], Bavelier demonstrated that selective visual attention improves where the first originates from solidarity, social union and with playing action video games, with effects lasting for 6 spontaneous expressions of emotions in personal relationships, months after the last training [13]. In more complex multiplayer and the second is established on rational self-interest with less first person shooter games, there is also a heavy cognitive load: emphasis on kinship and personal relations, MMOG the player has to observe certain strategy and keep track of other communities show a greater degree of Gemeineschaft [18]. Such players’ actions. While primary visual focus is centrally on the social environment promotes cohesion among the members and screen, peripheral vision is fully engaged as well, since increases social proximity among them [18], which in turn important elements tend to pop up from the sides. creates a safe space for establishing personal relationships which offer similar support as real-life interaction [21]. Research on The causal relationship between playing action games and social support from MMOG relationships and associated levels improved players' ability to coordinate visual input with their of depression in players demonstrated that higher social 123 involvement in MMOG communities resulted in a greater level performance. Journal of Adolescence, 27(1), 5–22. DOI= of perceived social support, which consequentially led to lower http://doi.org/https://doi.org/10.1016/j.adolescence.2003.1 levels of depression [21]. 0.002 On a different note, but still relevant, because studies have [3] Stockdale, L., & Coyne, S. M. (2018). Video game shown that emotional and physical pain share a lot of common addiction in emerging adulthood: Cross-sectional evidence properties [22], [23], playing video games might also have an of pathology in video game addicts as compared to analgesic effect. In patients undergoing treatment for severe matched healthy controls. Journal of Affective Disorders, burns, playing video games proved successful in their pain 225, 265–272. DOI= management [24]. When conducting a study in treatment of http://doi.org/10.1016/j.jad.2017.08.045 combat-related burns where pharmacological therapy was [4] Turel, O., Romashkin, A., & Morrison, K. M. (2016). complemented with VR gaming, the perceived pain intensity Health Outcomes of Information System Use Lifestyles score fell from 6.25 to 4.50 out of 10, unpleasant feelings among Adolescents: Videogame Addiction, Sleep related to pain dropped from 6.25 to 2.83 out of 10, and the time Curtailment and Cardio-Metabolic Deficiencies. PLOS spent thinking about the pain dropped from 76% during no VR ONE, 11(5), e0154764. Retrieved from to 22% during VR-coupled treatment [25]. It is suggested that https://doi.org/10.1371/journal.pone.0154764 inputs from intense, fast-paced immersive virtual environment distract the brain from processing the pain by disturbing the [5] Palaus, M., Marron, E. M., Viejo-Sobera, R., & Redolar- descending pain-control system, which results in pain- Ripoll, D. (2017). Neural Basis of Video Gaming: A attenuating effects [26]. Systematic Review. Frontiers in Human Neuroscience, 11, 248. DOI= http://doi.org/10.3389/fnhum.2017.00248 3.5 Collaboration and problem solving [6] Safire, W. (2002). Our New Prometean Gift. In S. J. In addition, in the last decade a subset of these games has tried Marcus (ed.), Neuroethics: Mapping the Field. San to tackle some of the many problems which trouble our society Francisco: Dana Press. Retrieved September 1, 2018, from and attempted to gamify real life problems. One of the first http://dana.org/Cerebrum/2002/Neuroethics__Mapping_th games which was successful in harnessing the gamers’ superior e_Field/. spatial and problem-solving skills was Foldit in 2008 [27]. The [7] Frank, A. (2013). The Future Of Gaming - It May Be All objective of the game was to predict the native structure of In Your Head. Forbes. Retrieved September 7, 2018, from various proteins to help the scientists to better understand the https://www.forbes.com/sites/singularity/2013/06/03/the- molecules. Over 50,000 players collaboratively interacted with future-of-gaming-it-may-be-all-in-your- protein models using user-friendly manipulation tools, and at the head/#7839776b7790 end of 2010 their results matched or outperformed algorithmically computed solutions. Foldit players achieved ‘an [8] Granic, I., Lobel, A., & Engels, R. C. M. E. (2014). The epic win’ in 2011, when they successfully deciphered the benefits of playing video games. The American structure of an enzyme which is critical for reproduction of the Psychologist, 69(1), 66–78. DOI= HIV/AIDS virus – a problem which had been previously http://doi.org/10.1037/a0034857 unsolved for 15 years [28]. Following in Foldit’s steps is Eterna, [9] Green, C. S. (2013, December 16). Neurogaming: What’s a similar online gaming tool, which asks players to solve puzzles Neuroscience and Ethics Got To Do With It? - Exploring in order to design molecular medicines in collaboration with Ethics. [video file] Retrieved September 7, 2018, from Stanford’s School of Medicine [29]. The mechanisms which are https://www.youtube.com/watch?v=apFE6AR9214 responsible for the success of gamification in solving difficult [10] Bao, S., Chan, V. T., & Merzenich, M. M. (2001). Cortical tasks are probably the same learning-promoting mechanisms remodelling induced by activity of ventral tegmental described in 3.1. dopamine neurons. Nature, 412(6842), 79–83. DOI= 4. CONCLUSION http://doi.org/10.1038/35083586 I summarized important benefits of playing video games, which [11] DeYoung, C. (2013). The neuromodulator of exploration: can be mainly characterized as improved motor, cognitive, A unifying theory of the role of dopamine in personality. social, emotional and problem solving skills. Is there something Frontiers in Human Neuroscience, 7, 762. DOI= as a benevolent brain manipulation? I dare say there is, and as http://doi.org/10.3389/fnhum.2013.00762 studies suggest, playing games moderately and in a responsible [12] Davies, D., Jindal-Snape, D., Collier, C., Digby, R., Hay, way can serve as a therapeutic and cognitive enhancement tool P., & Howe, A. (2013). Creative learning environments in despite the dangers inherent to gaming technologies. education—A systematic literature review. Thinking Skills 5. REFERENCES and Creativity, 8, 80–91. DOI= [1] Anderson, C. A., Shibuya, A., Ihori, N., Swing, E. L., http://doi.org/https://doi.org/10.1016/j.tsc.2012.07.004 Bushman, B. J., Sakamoto, A., Rothstein, H.R., Saleem, [13] Green, C. S., & Bavelier, D. (2003). Action video game M. (2010). Violent video game effects on aggression, modifies visual selective attention. Nature, 423(6939), empathy, and prosocial behavior in Eastern and Western 534–537. DOI= http://doi.org/10.1038/nature01647 countries: A meta-analytic review. Psychological Bulletin, [14] Li, L., Chen, R., & Chen, J. (2016). Playing Action Video 136(2), 151–173. DOI= http://doi.org/10.1037/a0018251 Games Improves Visuomotor Control. Psychological [2] Gentile, D. A., Lynch, P. J., Linder, J. R., & Walsh, D. A. Science, 27(8), 1092–1108. DOI= (2004). The effects of violent video game habits on http://doi.org/10.1177/0956797616650300 adolescent hostility, aggressive behaviors, and school [15] Li, R. W., Ngo, C., Nguyen, J., & Levi, D. M. (2011). 124 Video-game play induces plasticity in the visual system of (2003). Does rejection hurt? An fMRI study of social adults with amblyopia. PLoS Biology, 9(8), e1001135. exclusion. Science, 302(5643), 290–292. DOI= DOI= http://doi.org/10.1371/journal.pbio.1001135 http://doi.org/10.1126/science.1089134 [16] Franceschini, S., Gori, S., Ruffino, M., Viola, S., Molteni, [23] Kross, E., Berman, M. G., Mischel, W., Smith, E. E., & M., & Facoetti, A. (2013). Action video games make Wager, T. D. (2011). Social rejection shares dyslexic children read better. Current Biology : CB, 23(6), somatosensory representations with physical pain. 462–466. DOI= http://doi.org/10.1016/j.cub.2013.01.044 Proceedings of the National Academy of Sciences, [17] Schlickum, M. K., Hedman, L., Enochsson, L., Kjellin, A., 108(15), 6270–6275. DOI= & Fellander-Tsai, L. (2009). Systematic video game http://doi.org/10.1073/pnas.1102693108 training in surgical novices improves performance in [24] Hoffman, H. G., Doctor, J. N., Patterson, D. R., virtual reality endoscopic surgical simulators: a Carrougher, G. J., & Furness, T. A. 3rd. (2000). Virtual prospective randomized study. World Journal of Surgery, reality as an adjunctive pain control during burn wound 33(11), 2360–2367. DOI= http://doi.org/10.1007/s00268- care in adolescent patients. Pain, 85(1–2), 305–309. 009-0151-y [25] Maani, C. V, Hoffman, H. G., Morrow, M., Maiers, A., [18] Simpson, J. M., Knottnerus, J. D., & Stern, M. J. (2018). Gaylord, K., McGhee, L. L., & DeSocio, P. A. (2011). Virtual Rituals: Community, Emotion, and Ritual in Virtual reality pain control during burn wound Massive Multiplayer Online Role-playing Games—A debridement of combat-related burn injuries using robot- Quantitative Test and Extension of Structural Ritualization like arm mounted VR goggles. The Journal of Trauma, Theory. Socius, 4, 2378023118779839. DOI= 71(1 Suppl), S125-30. http://doi.org/10.1177/2378023118779839 http://doi.org/10.1097/TA.0b013e31822192e2 [19] Szell, M., & Thurner, S. (2010). Measuring social [26] Gold, J. I., Belmont, K. A., & Thomas, D. A. (2007). The dynamics in a massive multiplayer online game. Social Neurobiology of Virtual Reality Pain Attenuation. Networks, 32(4), 313–329. DOI= CyberPsychology & Behavior, 10(4), 536–544. http://doi.org/https://doi.org/10.1016/j.socnet.2010.06.001 http://doi.org/10.1089/cpb.2007.9993 [20] Tönnies, F., & Hollis, M. (2001). Community and Civil [27] Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J., Society. (J. Harris, ed.) (1st ed.). Cambridge: Cambridge Beenen, M., Leaver-Fay, A., Baker, D., Popović, Z. and University Press. DOI= Foldit players (2010). Predicting protein structures with a http://doi.org/10.1017/CBO9780511816260.006 multiplayer online game. Nature, 466(7307), 756–760. [21] Dupuis, E. C., & Ramsey, M. A. (2011). The relation of DOI= http://doi.org/10.1038/nature09304 social support to depression in massively multiplayer [28] Foldit. In Wikipedia. Retrieved September 7, 2018, from online role-playing games. Journal of Applied Social https://en.wikipedia.org/wiki/Foldit Psychology, 41(10), 2479–2491. [29] Eterna Game. [Web page] Retrieved September 9, 2018, DOI=http://doi.org/10.1111/j.1559-1816.2011.00821.x from https://eternagame.org/ [22] Eisenberger, N. I., Lieberman, M. D., & Williams, K. D. 125 In search of the authentic self: explaining phenomenology of authenticity Maša Urbančič Independent researcher Štefanova 13 (telo.si) 1000 Ljubljana masa.urbancic@gmail.com The aim here is to provide a model for specifically explaining the ABSTRACT phenomenological (i.e. lived) experience of authenticity, which I will call the authentic-like experience. The main argument will be There are moments in most people’s lives when they feel more or that feelings of authenticity are (usually positive) momentary less themselves. This experience is usually caught in people experiences, which are a result of a convergence between the sayings that they need to find themselves or just be who they current self (beliefs about oneself) and the ideal self (what one ‘really are’. The purpose of this paper is to provide an explanation wants to be, thinks she should be and what is important to her). for these feelings of authenticity. I first explore this concept and On this account, the authentic self comes about as a consequence consequently introduce necessary conditions for the of creation and not as a consequence of discovery. phenomenological experience of authenticity. This is followed by the examination of two problems that authenticity faces and two 2. AUTHENTICITY AND ITS PROBLEMS possible ways of explaining the experience of authenticity, which The purpose of this section are the following four points: 1) to are discovery and creation. I then assess three discovery views introduce the concept of authenticity, 2) to examine necessary and show why they are unsatisfactory in explaining feelings of conditions for its phenomenology 3) to present two problems that authenticity. In response I then provide my own creation model of authenticity faces and 4) to present two ways of thinking about the authenticity, which argues that feelings of authenticity are a result explanation of an authentic-like experience. of a convergence between our current and ideal self. In this sense, authentic self is created. What we are is a matter of choice. The concept of authenticity is defined in multiple ways (see [1], [3], [12]). Here, I take authenticity to denote “a convergence Keywords between what something is and what it claims it is” [1]. In other words, when a person is being authentic, her statements and Authentic self, true self, authenticity, phenomenology, ideal self thoughts, as well as her beliefs and actions are in accordance with 1. INTRODUCTION one another. The idea of an authentic or a true self1 seems to have been present For a person to undergo an authentic-like experience, I argue there since ancient times, from the Greek “Know Thyself” to are two necessary conditions that need to be met: 1) a certain Shakespeare’s “To thine own self be true” to modern advice to degree of enjoyment or dislike of a certain trait or ability and 2) “just be yourself” [11]. One often hears people speaking of their like Bialystok [1] argues, a convergence with a set of criteria need to find themselves or to discover who they “really are” where, when the convergence with those criteria takes place, the (ibid). I take it that most people agree with Bialystok that “almost result will be an authentic-like experience. In other words for the everyone has had an intuitive experience of authenticity that second condition, if a person is sometimes authentic, there must seems to reveal a glimmer of one’s true identity” [1]. However, be times when she is inauthentic [1] so we need to have a standard the problem arises how it is possible that a person sometimes feels that defines when the person is being authentic. Whenever a more or less like herself, when she is never another person, but person’s current state converges with those criteria, an authentic- only herself. However, if that is the case, how can these feelings like experience will take place. I will call those criteria the of authenticity be explained? Do these moments of experiencing authentic self* (and later on in my model, the ideal self), which authenticity reveal a deeper, truer identity of a person, an specifically denotes the thing with which the person converges unchanging inner core that lies beneath, an authentic self? Or do with in an authentic-like experience. It can be thought of as they reveal something else?2 discovered or created. On the other hand, the authentic self denotes the result of a convergence between the person and the authentic self*, i.e. the authentic-like experience and what people refer to as their authentic self. The convergence condition or the authentic self* presents a 1 For clarity purposes, only authentic self is used in this paper, not problem in the discussion of authenticity because it is unclear true or deep self. what the authentic self* represents, which will be the discussion 2 I do not differentiate between identity and self and I only take for the rest of the paper. In other words, it is unclear what makes appropriation as a necessary condition for personal identity or some parts count as the authentic self and how to separate it from self – that we see certain actions, beliefs, etc as ours. For a the inauthentic parts (see [1] for this issue). Authenticity denotes discussion on personal identity, self (and true self) from a the harmony between one’s beliefs, thoughts and actions (see phenomenological point of view, see Jacobs [5]. previous paragraph), but a person rarely identifies with all her 126 thoughts and beliefs and declares them as her authentic self (e.g., argue that it is the latter case that takes place when authenticity is rarely do we encounter a person that would identify having a felt. In the next part, I will present different ways one can think thought of strangling somebody who is annoying as her authentic about the authentic self* as being discovered and why these do self). However, as Bialystok [1] points out, in current not offer a satisfactory explanation for the phenomenal experience understanding of authenticity the authority on the authentic self is of authenticity. found within ourselves, which is why it is to an extent a matter of self-interpretation. There are no objective criteria against which it 3. AUTHENTIC SELF* AS DISCOVERY could be determined which parts of the person are authentic and The first question that needs to be asked is if the authentic self* which not or any empirical facts that could confirm a person’s (the criteria) is to be discovered, what exactly is it that is being authentic self [1]. It will therefore always come down to what the discovered? I will consider three possible answers to this problem, person declares as her authentic self. I do not hold that the matter which are 1) an unchanging ontological substance 2) a unique of authentic self depends on self-interpretation is problematic as personality or 3) a daimon. These can be regarded as Bialystok [1] does, since the final decision with what we identify manifestation of these criteria (or the authentic self*). ultimately lies in our hands. However, the problem of separating The first possibility of what the authentic self* represents is authentic from inauthentic parts still persists. connected to philosophical debates on the concept of self. Metzinger defines it as an enduring individual entity, an There is a second further issue with authenticity, which Bialystok unchanging essence or an ontological substance that could exist [1] labels as the hard case of authenticity. More specifically, it is independently in the world and would represent a basic impossible to determine when the person is being authentic where constituent of reality [6] [7]. He goes on to deny the existence of one’s feelings, thoughts or beliefs change throughout life. To such a self and argues there is currently no substantive empirical imagine such scenarios, Bialystok refers to religious conversions, evidence or any theoretical argument that would necessitate the career change, divorce, change in party affiliations or simply existence of a self [7]. Zahavi, on the other hand, rejects changes in personality [1]. It is not difficult to imagine such cases Metzinger’s definition and characterizes the self rather as being where one is different from how one used to be in the past. The constituted through the first-person givenness of different dilemma however, is, when is the person then authentic? For experiences [15]. In other words, there is no separate entity, but a instance, is a person authentic before or after a religious self exists simply due to the fact that the world is being conversion? Intuitively one might say that one is authentic after experienced from a first-person perspective. Olson [8] goes even the change, but most people claim they were also authentic before. so far as to state we should cease to speak of selves due to lack of If a person is authentic in both cases, then authenticity seems a common definition and its causing dilemmas that would impossible [1]. But if a person is inauthentic in both cases, when otherwise be avoidable. is he then authentic at all if not in the beliefs that constitute a greater part of his identity (ibid)? I believe this argument Any of these definitions do not provide any insight into the mistakenly presupposes that a person's identity should be constant authenticity of the self. If the authentic self* is understood in in order to constitute the authentic self. One can still act in Zahavi’s terms, such a definition presents a problem for accordance with what one feels in that moment, without assuming authenticity, since the givenness of a first-person’s perspective is the authentic self* is constant. Nevertheless, any explanation of always present, regardless whether a person feels authentic or authenticity needs to account for such cases as well. inauthentic. Just having a first-person experience does not solve the issue of why a person sometimes feels more or less like This brings us to two ways of thinking how the authentic self* is herself. If the authentic self* is understood in the way that to be understood. These are discovery and creation, which I take Metzinger defines it, the authentic self* is then understood as an from Waterman’s [14] distinction between different ways of unchanging ontologically independent core that is distinct from understanding identity formation. The metaphor of discovery any characteristic, perception or trait that constitute the person. denotes that something, which has already existed before, has now Furthermore, as Metzinger points out, there is currently no become known or understood [14]. The discovery entails the empirical evidence for this notion of self [7]. Additionally, there is process of causing the unknown to become known (ibid). In this also the issue that even if there seems to be the experience of an case, a person is discovering a prior authentic self* and every time unchanging self from a phenomenological standpoint (i.e. she converges with this authentic self*, she undergoes an constant unity of consciousness), we are nevertheless unable to authentic-like experience. Creation, on the other hand, involves directly access this inner core self if it is a separate and distinct creating something that has never existed before. In this process, entity, as Hume initially pointed out (see [4]). Therefore, if there one starts off from a point of unlimited possibilities and among is an entity as an authentic self* that is separate and distinct from those makes a choice and brings together parts to create all characteristics or perceptions that constitute the person it still something of value [14]. On this view, the assumption would be does not help address the problem of authenticity. It is not there is no prior not-yet-discovered authentic self* and it is only a connected to something that would be over and above the abilities product of our choices [12]. In this way, the authentic-like and characteristics that constitute him and which provide him with experience can be either thought of as being a result of a an authentic-like experience. Hence, feelings of authenticity convergence with a discovered prior authentic self* or as a result cannot be disclosing a convergence with an independent entity- of a convergence with a created authentic self*. In this way, an like authentic self*, but must be presenting something else. For authentic self* can therefore be something that has already existed these reasons it seems viable to conclude that conceptualizing the before and is becoming known or understood in moments of authentic self* as a separate entity cannot satisfactorily explain experiencing authenticity (or most probably parts of it). the phenomenology of authenticity. Alternatively, the authentic self* has not existed before and is The second model avoids the Hume worry in that it created from unlimited possibilities as a matter of choice (see e.g. conceptualizes the authentic self* as a unique personality, i.e. a [13]). Moments of experiencing authenticity are therefore set of characteristics and traits that are unique to every individual moments of experiencing something that we have created. I will and not as something distinct. This model presents the modern 127 conception of authenticity that is especially prevalent in ontology and the nature of self, I focus specifically on the contemporary self-help literature (see [2]). In opposition to older explanation of feelings of authenticity. I argue this conceptions of authenticity, where the task was to become what phenomenology of authenticity is best explained by a model on one can be, i.e. realizing her potentials and purpose, the which feelings of authenticity occur as a result of a convergence contemporary ideal of authenticity requires that the person between the current and the ideal self. As argued above, there realizes and becomes what she already is, the set of characteristics must be a convergence between a person and her authentic self* and traits that are already situated within the person ([2, 13], my for authenticity to occur. From now on I will refer to the authentic emphasis). The main idea of this understanding of authenticity is self* as the ideal self in my model. According to this model, that everyone possesses a deep, authentic self* within – a so- feelings of authenticity are (usually positive) momentary called Real Me – that is distinct from everything else that is not emotional experiences. Feelings of authenticity are usually really me. In other words, there is great emphasis on the connected to positive traits because the ideal self is generally difference between the inner and outer, where the outer is false viewed as intrinsically good (see [8]). However, there can also be and the inner true ([2], my emphasis). Authentic self* represents instances where the ideal self is connected to negative traits (e.g. the collection of all feelings, needs, desires, capacities, unsociable, not funny, etc). dispositions and abilities that constitute a person’s unique Current self denotes all beliefs one holds about herself at the personality and our task is to get in touch with this personality present time, for example being athletic, not funny enough, etc. (e.g. through introspection) and to express it (see [2] on this). The ideal self can be thought of as a collection of all desires of Here therefore, whenever the person is acting in accordance with how one wants to be, norms of how one should be and values that this unique personality, she is feeling authentic. The problems constitute what is important in a person’s life. It is important to here are the assumptions our personality has been shaped at birth note that what will constitute the ideal self will be highly and there seems to be disregard for personal experiences on influenced by what is desirable within the society as well. Both shaping a person’s self because this understanding of the authentic the current and ideal self-beliefs can be either conscious or non- self* assumes the authentic self* is within us and not influenced conscious. When one considers one’s ideal self, there are four by the environment (the outer), neither it is changed throughout possible scenarios that can take place. First is the scenario where a life. With the inner-outer distinction, there is also disregard for convergence between a positive current self belief (e.g. I am societal influences on our identity. funny) and the ideal self (I want to be funny) occurs. In those The third model envisions a daimon (= authentic self*) that cases, feelings of authenticity will be experienced. In the second represents the potentialities and talents within each person which scenario there is a lack of convergence between a person’s are not yet realized but whose realization represents the greatest positive belief about the current self and her ideal self. In this fulfilment in life (an ideal) [14]. In other words, the person aims case, the person most likely experiences a questioning of her to reach her authentic self* or her daimon by realizing her behaviour, or (if such situations happen often enough) an identity potentials, not by becoming a person she already is within. The crisis if the belief is central to her identity. The third scenario is a difficulty with this model is still the persisting assumption that case of negative current self belief (e.g. I am not smart) and a lack daimon is fixed since birth (if daimon is to be discovered with of convergence with the ideal self. In this case, a person believes time and lived, it is difficult to know what is being discovered if it she is lacking certain ability or trait (current self) that she wants to consistently changes). And as the hard case of authenticity possess (ideal self) and situations confirm that it is the case she is highlights, nothing about a person’s personality appears to be lacking them (confirmation of existing current belief). In this permanent enough that it would remain fixed upon discovery. The scenario, there is no self-acceptance present of how she is at the person does not stay the same after unearthing the daimon. It present time (and oftentimes no self-awareness of what constitutes might be objected that the daimon is only a set of potentialities her ideal self). In such scenarios, possible phenomenology might and abilities. It presents us with the limitations of our capabilities be one of sadness or disappointment, but for the purposes of this or reveals in what activities or tasks we find enjoyment in. It is not paper it is important that no feelings of authenticity will be to be understood as a set of fixed properties. This argument holds experienced. The fourth scenario presents the most complicated some merit, but it must nevertheless be argued that we might scenario where the negative current self belief initially does not become aware of our capabilities, interests and talents, but we are converge with the ideal self but can gradually converge with the not equal to them. Feelings of authenticity can be out of alignment ideal self if a) negative current self beliefs change to positive or b) with one’s abilities as in, for example, when a person is highly ideal self changes (to negative belief). The current negative beliefs talented for running but does not enjoy it. Therefore, potentials can change into positive in two ways: 1) by changing behaviour or and abilities are not a sign of an authentic* self. 2) by gradually discovering that the negative beliefs were inaccurate. For illustration of 1 and 2, a person named Peter 4. AUTHENTIC SELF* AS CREATION: believes he is unsociable (negative current belief) and wants to be MY OWN MODEL sociable (ideal self). There is currently no convergence and no The purpose of this section is to present my own model for authentic-like experience is present. He can change his behaviour explaining feelings of authenticity, which views the authentic to being sociable, which will in turn change his negative current self* as created. The idea that the authentic self is created is self belief of unsociable to sociable and a convergence with the mostly found in existentialist writings. Generally put, according to ideal self will occur. In those moments, Peter will have an existentialism, there is no pre-existent essence or a “true self”, but authentic-like experience. Alternatively, Peter is exposed to one’s identity is created through personal choices [14]. Sartre situations where he receives feedback that he is indeed sociable states that “man first of all exists /…/ and defines himself and like his ideal self, despite his thinking otherwise. In this case, afterwards” and that every person chooses herself [10] (see also Peter’s belief of being unsociable will gradually transform into Nietzsche and Foucault). current self belief of being sociable and a convergence between his current and ideal self will take place. Again, authenticity will This model is ultimately compatible with the existentialist view of be present in such a scenario. In terms of changing the ideal self the authentic self, but whereas they focus more broadly on the 128 (option b), Peter comes to accept he is unsociable and realizes that beliefs, thoughts and desires at the present time. The task for his enjoyment of his alone time is more important to him than further research would be to look deeper into the structure and being sociable (or some other reason for preferring not to be creation of the ideal self. It would be further useful to empirically sociable). In this case, a shift within the ideal self takes place in examine which situations elicit feelings of authenticity and to that it now includes not being sociable. Therefore, if Peter is being investigate what drives people’s experiences of authenticity in reproached for not being sociable, he says “This is who I am” and those moments. experiences authenticity because being unsociable is now part of his ideal self.3 If Peter might be disrespecting the latter desire, this 6. ACKNOWLEDGMENTS could lead to scenario two. Thanks to Dr. Alistair Isaac and Dr. Mog Stapleton for their This model avoids the problems faced by the previous presented guidance and advice in preparing this paper. models. Firstly, it avoids the problem of having to find an 7. REFERENCES enduring ontologically distinct entity that is over and above the [1] Bialystok, Lauren. Authenticity and the Limits of traits and abilities of a person, because the ideal self is constituted Philosophy. Dialogue, 53: 271-98. 2014. by them. Secondly, it avoids the problems of the ideal self being pre-determined and separate from the society. The ideal self is a [2] Guignon, Charles. On Being Authentic. USA, Routledge, collection of values, norms and desires that continually develops 2004. and changes throughout life. The ideal self is influenced and [3] Guignon, Charles. Authenticity. Philosophy Compass, 3 (2): constituted in interaction with and by society and shaped by 277-90, 2008. personal experiences. Furthermore, the presented model of the ideal self can help address the two problems with authenticity [4] Hume, David. A Treatise of Human Nature (2nd Edition), raised in section two. First, it can help explain the hard case of L.A. Selby-Bigge and B.H. Nidditch (eds). Oxford, Oxford authenticity since I argue that the bar for authenticity is behaving University Press, 1739-1740/2014. in accordance with the ideal self, not constancy in one’s identity. [5] Jacobs, Hanne. Towards a Phenomenological Account of To the second problem, namely how to distinguish between Personal Identity. In Philosophy, Phenomenology, Sciences: authentic and inauthentic parts, my answer is that it is the Essays in Commemoration of Edmund Husserl, convergence with the ideal self (plus enjoyment or dislike as Phaenomenologica 200. Springer, New York, 2010. discussed before) that makes the difference. In this way, it could [6] Metzinger, Thomas. Being No One: The Self-Model Theory be researched what people aspire to and move away from (i.e. of Subjectivity. USA, MIT Press, 2003. what is part of the ideal self and what not) and how they feel about these characteristics (enjoyment or dislike) by which we [7] Metzinger, Thomas. The No-Self Alternative. In S.Gallagher, could better understand how they perceive their authentic selves editor, The Oxford Handbook of the Self, pages 279-96. and if people act in accordance with the scenarios presented. Oxford, Oxford University Press, 2011. Finally, it might be objected that the phenomenology of becoming [8] Newman, E. George, Bloom Paul and Joshua Knobe. Value more and more self-aware with time pushes us more towards the Judgments and the True Self. Personality and Social discovery view. There is a sense of “peeling-off” layers and Psychology Bulletin, 40 (2): 203-16, 2014. coming to know ourselves better during life. However, I would [9] Olson, T. Eric. There is no problem of the self. Journal of argue that we are not discovering an authentic self* that is waiting Consciousness Studies, 5: 645-57, 1998. to be discovered in those moments. We are becoming more aware of our ideal self and the latter is becoming more specific and [10] Sartre, Jean-Paul. Existentialism and humanism, trans. Philip detailed with time. The one other concern is how does one Mairet. UK, Methuen London, 1980. distinguish what is part of the ideal self and what not? I would [11] Schlegel, J. Rebecca, Hicks, A. Joshua, Arndt Jamie and argue that what the ideal self contains is largely a matter of Laura A. King. Thine Own Self: True Self-Concept choice. There is a matter of making the ideal self conscious and Accessibility and Meaning in Life. Journal of Personality more detailed. But once that becomes the case, a person can make and Social Psychology, 96 (2): 473-90, 2009. a choice in what the ideal self will be. In that way, we decide what [12] Vannini, Phillip and Alexis Franzese. The Authenticity of we want to be. In that sense this does not simply push the Self: Conceptualization, Personal Experience, and Practice. authentic self one step further with the ideal self (instead of an Sociology Compass, 2 (5): 1621-37, 2008. authentic self* we are discovering an ideal self) because the ideal self is not a constant entity, but a collection of our values, beliefs, [13] Varga, Somogy. Existential choices: to what degree is who desires and influences that is simply becoming more and more we are a matter of choice? Continental Philosophy Review, detailed with time. The more detailed it is, the greater choice we 44: 65-79, 2011. have in what we want to become, i.e. our authentic self. [14] Waterman S., Allan. Identity Formation: Discovery or Creation? The Journal of Early Adolescence, 4 (4): 329-41, 5. CONCLUSION 1984. The aim of this paper was to provide a preliminary model for explaining phenomenology of authenticity. On this account, [15] Zahavi, Dan. Unity of Consciousness and the Problem of authentic-like experiences are a result of a convergence between Self. In S. Gallagher, editor, The Oxford Handbook of the our current self and ideal self. The ideal self is a collection of our Self, pages 316-35. Oxford: Oxford University Press, 2011. 3 I would argue that if a characteristic is seen as fixed and cannot be changed, it is scenario three. If the said characteristic is more a choice, then it is scenario four. 129 Exploring Features of Cognitive Science as Natural Epistemology Tine Kolenik Department of Intelligent Systems Jožef Stefan Institute Jamova cesta 39 1000 Ljubljana, Slovenia tine.kolenik@ijs.si ABSTRACT identified by continually updating the image and comparing it Historically as well as conceptually, epistemology and cognitive with stored images of that same object [7] – this happens science seem to be intimately connected. The overlap between especially when identifying the same object from different them seems to be rather extensive, and this overlap can be perspectives. It therefore seems that CS can offer some sort of dubbed as natural epistemology, which has been similarly answers to epistemological questions. I believe the insight into proposed by a few other researchers. In the work, I define the deep intertwinings between CS and epistemology is natural epistemology as the study of epistemological questions important to be able to discern an overlapping research endeavor using scientific methods. Due to the particular nature of that studies epistemological questions with natural scientific studying knowing, which is what cognitive science largely does, methods. This research endeavor, which will be dubbed as I identify other important features, hidden behind the definition. natural epistemology, is strongly embodied in CS. By realizing To explore such features, I apply Kuhn’s ‘essential tension’ the epistemological background of the latter, it is important to between convergent and divergent thinking to how progress in delve into its features as natural epistemology to understand its cognitive science had historically played out, especially particular nature, organization and scientific workings, which regarding its paradigm shifts (cognitivism, connectionism, have significant influences on how research and progress in embodied/embedded cognition, enactivism). I propose that natural epistemology is manifested. ‘essential tension’ that fuels progress in cognitive science manifests as a loop between knowing of cognitive science 2. NATURAL EPISTEMOLOGY (epistemology of cognitive science) and knowing within While gaining insight from studying epistemological questions cognitive science (epistemological insights in cognitive with contemplative philosophical work is an extremely valuable science). I describe the knowledge loop’s importance and relate endeavor, philosopher and logician Quine followed what it to the history of cognitive science. In the end, I discuss future Dennett [8, p. 134] colorfully described with the following work, pointing to the problem of what is epistemological or not words: “Just as you cannot do very much carpentry with your as well as the problem of potential convergence of the bare hands, there is not much thinking you can do with your knowledge loop with the ideas of second-order cybernetics. bare brain.” Quine [9] described a view wherein epistemology uses scientific methods and subsequently coined the phrase Keywords ‘naturalized epistemology’. In parallel, similar views arose from cognitive science, epistemology, Kuhn, philosophy of cognitive cybernetics and second-order cybernetics circles as well as from science, philosophy of science the sociological domain. Bateson [10] used the terms ‘empirical epistemology’ and ‘experimental epistemology’ when describing 1. INTRODUCTION an endeavor of researching perceivers’ presuppositions built into the product or the final image of perceiving. Keeney [11] used Cognitive science (CS) has been by many characterized by its the term ‘natural epistemology’ to research how living beings strong historical [1] and contemporary connection to make distinctions in the world that construct it while at the same epistemology [2], despite some opposition to the possibility of time knowing that these distinctions are wholly dependent on expanding philosophical ideas to scientific disciplines [3]. these same living beings. Luhmann [12] used ‘natural Kvanvig [4] indicates that the question of the relationship epistemology’ to characterize that knowledge depends on an between mind and world (and more broadly of ‘what it is to observer’s inner processes and not on the question of what is know’) is the deepest question of epistemology, and some [2] ‘true’ or ‘false’ as such. He notes that there is a certain have claimed that this same relationship is the foremost circularity in this, observing self-reference in natural investigative matter of CS. Both CS and epistemology ask epistemology, which is hinted by Keeney as well – we certain questions that seem to be the same, especially in terms of supposedly know that the construction of the world is what sources of knowledge of the external world, e.g., perception [5], living beings do, which we want to research, but at the same where epistemology offers answers through philosophical time we are aware that we are these same living beings who analysis [6], while CS uses different scientific methods for its construct the world in the first place. I therefore propose that the answers [7]. For example, both are interested in perception, base definition of natural epistemology is this: Natural being one of the sources of knowledge, and how it is used to epistemology is the study of epistemological questions with the construe the world. Berkeley [6] argues against material objects use of natural scientific methods. However, due to the particular using philosophical analysis, while one of the constituent nature of the study, stemming from the fact that the construction disciplines of CS, artificial intelligence, uses computer vision to of knowledge is studied by constructors of knowledge or that examine different ways that the world is constructed. Namely, in knowing is investigated by others’ ‘knowings’, natural computer vision, material objects are largely construed and epistemology as manifested in CS possesses some unique 130 features. I try to ascertain some of them in this work. There are facts or concepts without necessarily accepting them” [Ibid.]. some similarities of the ascertained features to those of second- This is the last step before “one of those shifts in fundamental order cybernetics, which is “the study of the organization in theory, in problem field, and in scientific standards […], autopoietic machines, which are able to build their own referred as scientific revolutions” [14, p. 234]. components, and possess cognizance which allows them to observe and control other machines” [13, p. 73]. This insight is I believe that understanding Kuhn’s concept of ‘essential self-referenced in natural epistemology and it thus determines tension’ and its role in scientific progress is necessary to the progress in CS as such. To identify the features and their articulate how the self-reference of natural epistemology in CS influence in the progress of CS, I describe and apply Kuhn’ plays out. This articulation will follow after a short examination concept of ‘essential tension’ between convergent and divergent of how progress worked historically in CS in relation to thinking in science1 [14] to CS. Kuhnian scientific revolutions or paradigm shifts. 3. KUHN’S ESSENTIAL TENSION 4. SHORT EXAMINATION OF PROGRESS Kuhn [14] proposed that progress in science is typified by the IN COGNITIVE SCIENCE interplay between convergent thinking and divergent thinking Froese [15] distinguishes (roughly) four paradigm shifts in CS which emerges as ‘essential tension’. According to Kuhn, so far, which can be seen in the figure below. convergent thinking in science is what scientists usually do and the way they normally operate, where “the research scientist is not an innovator but a solver of puzzles, and the puzzles upon which he concentrates are just those which he believes can be both stated and solved within the existing scientific tradition” [14, p. 234]. The usual research scientist, “the practitioner of a mature science, from the beginning of his doctoral research, continues to work in the regions for which the paradigms derived from his education and from the research of his contemporaries seem adequate” [14, p. 245] and tries to “elucidate topographical detail on a map whose main outlines are available in advance” [Ibid.]. Kuhn believes, partly opposed to usual views on the topic, that convergent thinking is extremely important as it ensures safe, steady, sure and stable production the likes of which cannot be achieved with divergent thinking as prevalent; it is also why, apart from being the norm, Figure 1. Froese's interpretation of paradigm shifts in CS convergent thinking is so abound and common. It is exactly its [from 15, p. 76]. Circles do not represent the end of a exuberance that eventually leads to encountering “a problem in paradigm, only emergence of a new one. which the anticipated does not occur, a problem that goes wrong The four paradigms are cognitivism, connectionism, to ways suggestive of a fundamental weakness in the paradigm embodied/embedded cognition and enactivism. Following itself” [Ibid.]. Convergent thinking in established sciences is Kuhn's 'essential tension', convergent thinking was researchers' therefore a prelude to divergent thinking, which has to happen in modus operandi within these paradigms, while divergent order to find a solution to the described problem. However, thinking ultimately spawned a paradigm shift. The short scientists do not always go beyond the prelude as they do not overview below is meant as a conceptual validation of Kuhn's identify the problem as paradigm-breaking. They believe that ideas rather than a comprehensive and nuanced treatise of the continuing research in the established paradigm will exhibit history of CS. It is more focused on the role of artificial desired outcomes. Alas, according to Kuhn, the desired outcome intelligence (AI) in it, but it has to be noted that AI was only one cannot occur, as convergent thinking is “neither intended nor wave in the necessary tsunami that caused a paradigm shift. This likely to produce fundamental discoveries or revolutionary means that ideas from many other consitutent fields of CS that changes in scientific theory” [14, p. 233]. The scientist therefore directly contributed to paradigm shifts are not mentioned. has to have “the ability to recognize trouble when confronted by it” [14, p. 235] and to recognize “that something has gone wrong The insight that cognition is supposedly computation with with existing knowledge and beliefs” [Ibid.]. This modus arbitrary symbols started the cognitive revolution and the era of operandi continues into divergent thinking, which means that the cognitivism in the 1950s. This idea on cognition has been scientist must “rearrange the intellectual and manipulative manifested in most research since. A few decades long prosper, equipment he has previously relied upon, discarding some especially in AI, signalled to scientists that their idea about elements of his prior belief” [14, p. 226] and “lack prejudice” cognition was correct. One of the most publicly known examples [Ibid.], which is imposed by the established paradigm, “to a of this prosper is the AI chess player, as it symbolized human degree where [the scientist] can look at the most `self-evident' higher-order cognitive abilities. The artificial chess player's skills grew exponentially, and in 1996, the reigning world champion of the time Kasparov was beaten by it. However, the 1 Kuhn described and published his work on ‘essential tension’ idea of the top-down, centralized cognition, embodied in the AI as a result of lecturing at a conference on scientific creativity, of the time proved to have many problems. For example, large where the knowledge of creativity, which is gained by domains were a big issue [16], which could not be solved by scientists who study creative talent, was applied to scientists convergent thinking. It was recognized “that something ha[d] themselves. gone wrong with existing knowledge and beliefs” [14, p. 235], 131 and in the 1980s, the idea on cognition shifted to it being This outline forms a loop, as the last step inevitably leads into parallel, distributed and bottom-up, which was largely the first. The figure below summarizes this process: synonymous with the method of artificial neural networks (ANN). Connectionism opened doors to new research in many phenomena where the cognitivist paradigm failed (e.g., natural language processing [16]). ANNs are still predominantly used today and exhibit numerous achievements in mimicking human abilities2. However, many scientists [17] felt that connectionism was missing a key point in what cognition supposedly is and EPISTEMOLOGICAL EPISTEMOLOGY OF INSIGHTS IN how organisms perceive and act by dismissing the constitutive COGNITIVE SCIENCE COGNITIVE SCIENCE role of bodies in cognition and focusing solely on the brain. Again, convergent thinking shifted the view on cognition that started to incorporate organisms’ bodies and how they constitute organisms’ cognition. By being aware that cognition is embodied as well as embedded in the environment, research in robotics (as well as using robots to investigate cognition) started to flourish like never before, convergent thinking leading the Figure 2. The loop between epistemological insights in way again. cognitive science and epistemology of cognitive science. The last paradigm shift to enactivism will not be covered here, Relating this loop to Kuhn’s ‘essential tension’, convergent as its manifestation seems to be a bit more unclear than the thinking represents epistemological insights in CS or research previous ones as well as it being harder to delineate from the on knowing and cognition that occurs on daily basis. Divergent embodied cognition paradigm [18]. The short overview was thinking represents epistemology of CS, as this is where a shift meant to be exemplary as to apply Kuhn’s ‘essential tension’ to has to occur to overcome the problems in epistemological historical progress in cognitive science as natural epistemology. insights in CS. The interplay of knowings is apparent here: epistemology of CS may be dubbed as knowing of cognitive In the next chapter, I will build on the manifestation of Kuhn’s science, while epistemological insights in CS may be dubbed as ‘essential tension’ in CS by noting the role the circularity of knowing within cognitive science. One refers to the established knowing (see chapter 2) plays and relating it to convergent and paradigm and idea on cognition, from which the other stems. To divergent thinking. In the chapter I will show how, when overcome paradigm-breaking problems in knowing within conceptualized in this way, CS’s particular organization works. cognitive science, scientists have to become aware of epistemological presuppositions they hold on knowing and shift 5. THE KNOWLEDGE LOOP OF them. Knowing within cognitive science can therefore only COGNITIVE SCIENCE AS NATURAL advance at crucial points when something in knowing of cognitive science happens, and vice versa. The knowledge loop EPISTEMOLOGY emphasizes the importance researchers play in researching and From the way the progress in CS was described in the previous defining cognition and knowing. chapter, I can outline an abstract step-by-step template: 1. Scientists research a particular cognitive phenomenon 6. DISCUSSION AND CONCLUSION with their existing knowledge (Kuhn’s ‘prior beliefs’ In this work, I identify and specify certain features of cognitive or presuppositions from an established paradigm) science as natural epistemology. This is only the first step in a about knowing. process to try to characterize how scientific studying of knowing 2. Scientists make great advances, but then encounter an manifests itself. There are a few issues that I will explore in insurmountable problem, which cannot be solved with future work. First, the issue of what is epistemological and what convergent thinking (current epistemological is not will have to be addressed. Are the crucial questions in presuppositions). cognitive neuroscience like “What is the complete connectome of the human brain?” [19, p. 173] epistemological questions or 3. If the problem is identified as such, scientists have to not3? Second, the knowledge loop and the analysis of the work through a different set of epistemological interplay between knowing within cognitive science and presuppositions and new ideas about cognition, knowing of cognitive science will have to be taken further to therefore exhibiting divergent thinking. incorporate stronger ideas of second-order cybernetics, as it 4. The problem is solved, which opens the door to seems that this is where the concept of Kuhn’s ‘essential numerous new research not envisioned as such before. tension’ in cognitive science as natural epistemology is A paradigm shift within which research is conducted gravitating towards. Hopefully, the endeavor will not completely occurs, changing how the discipline sees knowing and converge into ideas of second-order cybernetics and offer changing epistemological presuppositions of the something new to philosophy of CS; if not, the connection scientists. between Kuhn’s ‘essential tension’ and second-order cybernetics was somewhat established. 2 And, interestingly, being used to model game players in more 3 Interestingly, some researchers [2] are sceptical of putting and more complex domains outside of strict-rules based ones neuroscience under the interdisciplinary umbrella of CS for as chess (e.g., OpenAI in Dota 2). precisely this reason. 132 As CS is considered as a too loose of a research program by [18] Ward, D., Silverman, D. and Villalobos, M. 2017. some [20], especially because it lacks its own methods, it seems Introduction: The varieties of enactivism. Topoi. 36, 3 important to characterize it by its own, unique properties. I (2017), 365–375. believe that the knowledge loop between knowing within [19] Adolphs, R. 2015. The unsolved problems of neuroscience. cognitive science and knowing of cognitive science may be one Trends Cogn Sci. 19, 4 (2015), 173–175. such property. What does seem certain is that epistemology works as a sort of glue between the constituents of CS, which [20] Coulter, J. 1982. Theoretical problems of cognitive science. gives validity to the latter’s interdisciplinary nature. Inquiry. 25, 1 (1982), 3–26. 7. REFERENCES [1] Fabricius, W. V. 1983. Piaget’s theory of knowledge: Its philosophical context. Hum Dev. 26, 6 (1983), 325–334. [2] Ó Nualláin, S. 2002. The Search For Mind: A new foundation for Cognitive Science. Intellect Books, Bristol. [3] Hacker, P. M. S. 2001. An orrery of intentionality. Lang Commun. 21, 2 (2001), 119–141. [4] Kvanvig, J. 2003. The Value of Knowledge and the Pursuit of Understanding (Cambridge Studies in Philosophy). Cambridge University Press, Cambridge. [5] Alston, W. 1999. Perceptual Knowledge. In The Blackwell Guide to Epistemology. J. Greco, E. Sosa, Eds., Blackwell, Oxford, 223–242. [6] Berkeley, G. 1710/1982. A treatise concerning the principles of human knowledge. Hackett. (Original work published in 1710). [7] Marr, D. 1982. Vision. Freeman, San Francisco. [8] Dennett, D. 1996. Kinds of Minds. Basic Books, New York. [9] Quine, W. V. O. 1969. Epistemology naturalized. In Ontological Relativity and Other Essays. W. V. O. Quine, Ed., Columbia University Press, New York, 69–90. [10] Bateson, G. 1979. Mind and Nature. A Necessary Unity. E. P. Dutton, New York. [11] Keeney, B. P. 1983. Aesthetics of Change. Guilford Press, New York. [12] Luhmann, N. 1996. Social Systems (J. Bednarz, Jr. & D. Baecker, Trans.). Stanford University Press, Stanford, CA. [13] Mancilla, R. 2011. Introduction to Sociocybernetics (Part 1): Third Order Cybernetics and a Basic Framework for Society. J Sociocybern. 42, 9 (2011), 35–56. [14] Kuhn, T. S. 1959. The essential tension. Tradition and innovation in scientific research. In The third (1959) University of Utah research conference on the identification of scientific talent. C. W. Taylor, Ed., University of Utah Press, Salt Lake City, 162–174. [15] Froese, T. 2010. From Cybernetics to Second-Order Cybernetics: A Comparative Analysis of Their Central Ideas. Construct Found. 5, 2 (2010), 75–85. [16] Rumelhart, D. E. and McClelland, J. 1986. On learning the past tense of English verbs. In Parallel Distributed Processing, Volume 2. D. E. Rumelhart, J. L. McClelland, Eds., MIT Press, Cambridge, Mass, 216–271. [17] Varela, F. J., Thompson, E. and Rosch, E. 1991. The embodied mind: cognitive science and human experience. MIT Press, Cambridge, MA. 133 Učinki telesne poze na prag termične bolečine za vroče Novaković Dušanka Puh Urška Vidmar Gaj MEi: CogSci, Univerza v Ljubljani Zdravstvena fakulteta Univerzitetni rehabilitacijski inštitut 1000 Ljubljana Univerza v Ljubljani Republike Slovenije – Soča nd333@protonmail.com Zdravstvena pot 5, 1000 Ljubljana Linhartova 51, 1000 Ljubljana urska.puh@zf.uni-lj.si IBMI, Medicinska fakulteta Univerza v Ljubljani Vrazov trg 2, 1000 Ljubljana Moharić Metka Bresjanac Mara FAMNIT, Univerza na Primorskem Univerzitetni rehabilitacijski inštitut Inštitut za patološko fiziologijo Glagoljaška 8, 6000 Koper Republike Slovenije – Soča Medicinska fakulteta gaj.vidmar@ir-rs.si Linhartova 51, 1000 Ljubljana Univerza v Ljubljani metka.moharic@ir-rs.si Vrazov trg 2, 1000 Ljubljana maja.bresjanac@mf.uni-lj.si POVZETEK telesnih poz na različne parametre [4 - 6], vedno več novejših raziskav [3, 7, 8] oporeka izsledkom predhodnih. Obvladovanje bolečine je pomembno področje vsakodnevnega življenja, pa tudi klinične in raziskovalne prakse. Ker so novi Zaradi kontroverznih izsledkov, vedno dobrodošlih novih načinov načini obvladovanja bolečine vedno dobrodošli in je povezava obvladovanja bolečine in neraziskanega področja morebitnega med telesno pozo ter bolečinskim pragom slabo raziskana, smo se vpliva telesnih poz na bolečinski prag smo se namenili ponoviti in namenili preveriti, ali zavzetje naključno dodeljene dominantne nadgraditi predhodni eksperiment. V navedeni raziskavi [1] so ali submisivne poze lahko povzroči akuten dvig ali znižanje praga udeležencem pred in po vzdrževanju ene od poz izmerili termične bolečine. Uporabili smo najvišje in najnižje ocenjeno bolečinski prag za ishemično bolečino, ki je bila izzvana z pozo moči iz predhodno izvedene spletne raziskave ter ponovili in manšeto za merjenje arterijskega tlaka. Rezultati so pokazali, da je nadgradili eksperiment Bohnsove in Wiltermutha [1]. Dobljeni dominantna poza zvišala bolečinski prag, pri submisivni pozi pa rezultati so pokazali, da je dominantna poza statistično značilno statistično značilnega učinka ni bilo. vplivala na dvig praga bolečine za vroče, kar je skladno z izsledki izvirne raziskave [1]. Naša raziskava je potekala tako, da smo najprej izbrali najbolj dominantno in najbolj submisivno pozo, nato pa izbrani pozi Ključne besede uporabili pri preverjanju povezave med naključno dodeljeno pozo in spremembo v pragu termične bolečine. Predhodno raziskavo Neverbalna komunikacija, poza visoke moči (dominantna poza), smo nadgradili z večjim vzorcem, zanesljivejšo metodo merjenja poza nizke moči (submisivna poza), termični bolečinski prag. ter z zagotovitvijo nepristranskosti in ponovljivosti navodil za zavzemanje poze. 1. UVOD Moč in nadvlado izražamo poleg besedne govorice tudi skozi 2. METODOLOGIJA nebesedno govorico telesa [2], ki za sporočanje poleg intonacije in 2.1 Udeleženci barve glasu uporablja telesno držo, geste in obrazno mimiko. Posameznik kaže lastno moč skozi nadrejeno, ekspanzivno držo in V raziskavi je sodelovalo 130 udeležencev, 56 moških in 74 asertivne gibe ter podrejenost skozi zaprto, kolabirano držo in žensk, starih od 19 do 62 let. En udeleženec je bil naknadno skromne gibe. izločen zaradi neupoštevanja navodil. Nekatere raziskave [1 - 3] nakazujejo, da odnos med nebesedno 2.2 Pripomočki in merski instrumentarij govorico in počutjem lahko poteka tudi v obratni smeri: namerno zavzeta dominantna poza spodbudno vpliva na dojemanje lastne Za merjenje praga termične bolečine za vroče smo uporabili moči, namerno zavzeta submisivna poza pa ima nanj negativen napravo Medoc Pathway Pain & Sensory Evaluation System učinek. Izraz poza moči uporabljamo, ko govorimo o telesnih (PSES, model ATS). S pomočjo spletnega vmesnika je udeleženec pozah, ki posnemajo dominantno ali submisivno držo telesa. odgovorili na nekaj vprašanj, si kasneje ogledal naključno Glede na njeno sporočilnost in občutenje moči osebe, ki jo dodeljeno fotografijo poze, sledil navodilom za njeno zavzemanje vzdržuje, jih imenujemo tudi poze visoke moči (dominantne poze) ter jo ob koncu ocenil glede na zaznani občutek moči oziroma ali poze nizke moči (submisivne poze), glede na držo telesa ter nemoči. položaj rok in nog pa tudi ekspanzivne (odprte) in kolabirane (zaprte) poze. Pri dominantnih pozah so udi običajno stran od 2.3 Postopek telesa, hrbtenica je zravnana in brada nekoliko privzdignjena. Pri Udeleženec je ob prihodu podpisal soglasje o sodelovanju in prek submisivnih pozah so udi običajno tesno ob trupu ali prekrižani, spletnega vmesnika odgovoril na nekaj demografskih vprašanj. Za brada je nemalokrat nekoliko spuščena in drža je sključena [1, 2]. zagotovitev anonimnosti podatkov mu je bila avtomatsko Čeprav številne raziskave iz zadnjih treh desetletij navajajo učinke dodeljena štirimestna koda, pod katero so bili obravnavani vsi 134 nadaljnji podatki. Sledile so testne meritve s termodo na koži submisivno razmeroma enostavna za zavzetje, bila je bolj stabilna palčne kepe nedominantne roke. Po testni meritvi je sledila prva in je verjetno zahtevala manj vzdrževane mišične aktivnosti. Zato eksperimentalna meritev na enakem mestu dominantne roke. dvomimo, da bi odsotnost vpliva submisivne poze in prisotnost vpliva dominantne poze na prag termične bolečine za vroče lahko Meritve bolečinskega praga za vroče so bile izvedene trikrat, pri pojasnili z večjo aktivnostjo mišic pri vzdrževanju dominantne čemer je bila izhodiščna temperatura termode vedno 32 °C. Po poze. meritvah je bil udeleženec povabljen pred namizni računalnik. Naročeno mu je bilo, naj po odhodu eksperimentatorja iz prostora Nekateri raziskovalci [2] so poročali, da je vzdrževanje na spletnem vmesniku nadaljuje na naslednjo stran, kjer mu bodo dominantne poze privedlo do zvišanja testosterona v slini in podana nadaljnja pisna navodila, fotografija naključno dodeljene vzdrževanje submisivne poze do njegovega znižanja. Drugi so poze ter posnetek glasovno predvajanih navodil za zavzetje poze. poročali o višjem bolečinskem pragu po zvišani ravni testosterona Pozo je udeleženec izvêdel in v njej vztrajal 3-krat po 20 sekund, pri živalskem modelu [9]. Raziskovalci študije o učinkih pri čemer je glasovno navodilo oznanilo iztek 20 sekund, začetek testosterona na moške, ki so bili razdeljeni v skupine glede na in konec premora, ponovno zavzemanje ter konec preizkusa. kronično izpostavljenost eksogenemu ali endogenemu testosteronu, pa so poročali o zmanjšani občutljivosti na taktilne Po vrnitvi v prostor je eksperimentator povabil udeleženca k drugi dražljaje pri skupini moških z najvišjo ravnijo testosterona [10]. Iz eksperimentalni seriji meritev na predelu palčne kepe dominantne tovrstnih raziskav bi lahko sklepali o povezavi med vzdrževanjem roke, ki so potekale enako kot prej. Vrednosti eksperimentalnih poz, zvišanjem ravni testosterona in zaznavanjem bolečine, vendar meritev so se shranjevale preko spletnega vmesnika ter bile novejše raziskave takšnim povezavam oporekajo [3, 7]. V naši kasneje povprečene za vsak sklop treh vrednosti. raziskavi se namenoma nismo odločili za meritve koncentracij Po drugi meritvi je udeleženec ocenil, s kolikšnim občutkom moči hormonov. Z nepristransko metodo smo želeli preveriti domnevo oziroma nemoči ga je na lestvici od 1 do 11 navdala zavzemana o učinku poze na bolečinski prag in se pri tem namerno izognili poza. dodatnim virom variabilnosti. Poleg tega so bili v raziskavo vključeni udeleženci obeh spolov in med spoloma razlik v 2.4 Analiza podatkov spremembi praga nismo zaznali. Čeprav bi nevroendokrini dejavniki lahko prispevali k vplivu, bi za vpogled v njihovo Učinek poze smo analizirali z univariatno primerjavo razlik v vpletenost morali izvesti novo, kompleksnejšo raziskavo, kjer bi srednji vrednosti med skupinama. Za analizo podatkov smo na ustrezen način vrednotili tudi morebitno spreminjanje uporabili programje IBM SPSS Statistics 23 za okolje Windows. nevroendokrinih parametrov. 3. REZULTATI 5. ZAHVALA Sprememba v pragu bolečine za vroče se je med skupinama z Zahvaljujemo se prostovoljcema Mateji Drolec Novak in Janu dominantno in submisivno pozo pred in po vzdrževanju poz Zibelniku za pomoč pri izvajanju raziskave. statistično značilno razlikovala ( t(127) = -3,11, p = 0,002). Pri udeležencih z dodeljeno dominantno pozo je bila povprečna 6. VIRI vrednost temperature praga bolečine za vroče na začetku 44,79 °C ( SD 3,55 °C) in po vzdrževanju poze 45,97 °C ( SD 3,51 °C). Pri [1] Bohns, V. K. in Wiltermuth, S. S. 2012. It hurts when I do udeležencih z dodeljeno submisivno pozo je bila povprečna this (or you do that): Posture and pain tolerance. Journal of vrednost temperature praga bolečine za vroče pred vzdrževanjem Experimental Social Psychology. 48,1, 341–345. DOI= poze 46,40 °C ( SD 3,14 °C) in po vzdrževanju 46,34 °C ( SD https://doi.org/10.1016/j.jesp.2011.05.022. 3,17 °C). [2] Carney, D. R., Cuddy, A. J. C. in Yap, A. J. 2010. Power Udeleženci, ki so vzdrževali submisivno pozo, so na lestvici od 1 Posing: Brief Nonverbal Displays Affect Neuroendocrine do 11 v povprečju poročali o manjšem občutku moči ( M = 3,8, Levels and Risk Tolerance. Psychological Science. 21,10, SD = 1,4) v primerjavi z udeleženci, ki so vzdrževali dominantno 1363–1368. DOI= pozo ( M = 8,7, SD = 1,5; t(127) = -19,06, p < 0,001). Rezultati so https://doi.org/10.1177/0956797610383437. primerljivi z ocenami udeležencev v spletni raziskavi za izbrani [3] Ranehill, E., Dreber, A., Johannesson, M., Leiberg, S., Sul, dve pozi. S. in Weber, R. A. 2015. Assessing the Robustness of Power Posing. Psychological Science. 26,5, 653–656. DOI= 4. RAZPRAVA https://doi.org/10.1177/0956797614553946. Namen naše raziskave je bil ugotoviti, ali imata izbrani pozi [4] Carney, D. R., Cuddy, A. J. C. in Yap, A. J. 2015. Review učinek na dvig oziroma spust praga termične bolečine za vroče. and Summary of Research on the Embodied Effects of Rezultati so pokazali, da je dominantna poza vplivala na dvig Expansive (vs. Contractive) Nonverbal Displays. praga za bolečino in da submisivna poza nanj ni imela učinka. Psychological Science, 26, 5, 657–663. DOI= Tovrstni izsledki so skladni s predhodno raziskavo, v kateri so https://doi.org/10.1177/0956797614566855. prag bolečine merili z izzivanjem ishemične bolečine [1]. Avtorja slednje sta domnevala, da je višji bolečinski prag dominantne [5] Nair, S., Sagar, M., Sollers, J., Consedine, N. in Broadbent, poze povezan z višjo aktivacijo mišic med njenim vzdrževanjem E. 2015. Do slumped and upright postures affect stress ter da to lahko vpliva na fiziološke procese v telesu, ki so responses? A randomized trial. Health Psychology, 34, 6, povezani z zaznavo bolečinskega praga. V naši raziskavi mišične 632–641. DOI= https://doi.org/10.1037/hea0000146. aktivnosti nismo neposredno merili, zato o tovrstnem učinku ne [6] Peña, J. in Chen, M. 2017. Playing with power: Power poses moremo argumentirano razpravljati. Lahko pa izpostavimo, da je affect enjoyment, presence, controller responsiveness, and bila izbrana dominantna poza iz naše raziskave v primerjavi s arousal when playing natural motion-controlled video games. 135 Computers in Human Behavior. 71, 428–435. DOI= [9] Rao, S. S. in Saifi, A. Q. 1981. Effect of testosterone on https://doi.org/10.1016/j.chb.2017.02.019. threshold of pain. Indian Journal of Physiology and Pharmacology, 25(4), 387–388. [7] Smith, K. M. in Apicella, C. L. 2017. Winners, losers, and posers: The effect of power poses on testosterone and risk- [10] Burris, A. S., Gracely, R. H., Carter, C. S., Sherins, R. J. in taking following competition. Hormones and Behavior. 92, Davidson, J. M. 1991. Testosterone therapy is associated 172–181. DOI= https://doi.org/10.1016/j.yhbeh.2016.11.003. with reduced tactile sensitivity in human males. Hormones and Behaviour, 25, 2, 195–205. [8] Garrison, K. E., Tang, D. in Schmeichel, B. J. 2016. Embodying Power. Social Psychological and Personality Science, 7, 7, 623–630. DOI= https://doi.org/10.1177/1948550616652209. 136 The effect of distractors in lateralized change detection working memory task Anka Slana Ozimič Mind and Brain Lab, Department of Psychology, Faculty of Arts, University of Ljubljana Aškerčeva 2 1000 Ljubljana anka.slana@ff.uni-lj.si ABSTRACT relevant visual information in the absence of the external environment, is needed to carry out the tasks at hand. The capacity of working memory to maintain visual information is highly limited and varies significantly across individuals. An important research effort is to understand the mechanisms of its An important effort of VWM research is to understand the limitation, one being the efficient selection of the relevant items reasons for the highly limited capacity to maintain visual from the immediate external environment to encode and representations, which spans from 3 to 5 meaningful items in maintain in working memory stores, while preventing the young adults, varies significantly across individuals and irrelevant items to occupy it’s capacity. Recently, a series of declines in healthy aging [2]. Recently, a series of EEG studies EEG studies using lateralized change detection task, in which [5, 3] focused on the neural substrates of VWM processes and the participants only have to maintain items presented on one identified a neurophysiological index of storage capacity in the visual hemifield, while irrelevant items are also presented on form of the slow negative ERP difference wave, known as the opposite hemifield, identified a neurophysiological correlate contralateral delay activity (CDA). Studies have shown that of storage capacity in the form of contralateral delay activity CDA is sensitive to the number of objects maintained in VWM (CDA) wave. Moreover, studies revealed that low-capacity [3]—its amplitude increases as the number of objects participants maintain irrelevant items along the target items, maintained in VWM increases, but reaches an asymptote at when both are presented in the same visual hemifield, indicating around 3-4 items (Figure 1F), depending on each individual’s a reduced ability to filter irrelevant stimuli from visual working memory capacity. Interestingly, the extent of the increase in memory. These studies, however, do not consider the possibility amplitude in CDA when working memory load is increased that participants might also maintain the irrelevant items from two to four items was found to strongly correlate with presented to the opposite visual hemifield. To address this individual’s VWM capacity and is therefore considered a concern, we designed an experiment in which we directly neurophysiological index of VWM capacity [5]. Moreover, manipulated the presence of distractors in the irrelevant visual studies have shown [3] that low-capacity participants find it hemifield to estimate and control for their effect. Twenty-eight hard to ignore irrelevant distractors when presented either participants took part in a visual working memory experiment concurrently with or successively to the target items [6]. The in which they were asked to maintain orientation of items authors suggested that irrelevant items burden the limited VWM presented to the left or right visual hemifield, while the capacity, leading to lower working memory performance. distracting items were either present or absent in the opposite visual hemifield. The results revealed significantly lower A typical paradigm used to study CDA is lateralized change estimates of the capacity in the presence vs. absence of detection task [5] (Figure 1A), in which participants are distractors, suggesting that participants were not able to ignore presented with a number of items on both sides of the screen, the distracting items presented to the opposite visual hemifield, but only have to focus on the side that was previously indicated challenging the validity of the estimates of visual working with a cue (usually an arrow pointing either left or right), memory capacity in CDA and other studies employing encode and maintain the relevant item information (e.g. colors lateralized change detection task. or orientations), while keeping their eye-gaze focused at the fixation point in the middle of the screen. Such tasks evoke an increase in electrical activity above the parietal-occipital cortex Keywords of the hemisphere contralateral to the visual hemifield in which Change detection task, capacity, contralateral delay activity, the target items were presented (Figure 1C). CDA is computed distractors, visual working memory. as the difference in EEG activity between contralateral and ipsilateral hemispheres (Figure 1E). In this way, any nonspecific, task-general bilateral activity such as perceptual 1. INTRODUCTION responses (Figure 1D), is removed and the remaining signal In everyday life, access to immediate external environment should be solely related to working memory processes [3]. obtained through different sensory systems is crucial not only to respond to the sources of threat, being one of the basic functions While this logic seems well reasoned for eliminating task- of sensory systems, but also to guide complex mental tasks irrelevant activity, we have identified a potential issue in the needed to carry out goal directed behavior. For example, when experimental design that could lead to false conclusions. When driving a car, one must have access to the visual environment at testing VWM capacity, researchers usually don’t consider the all times, in order to follow the driving rules and avoid potential possibility that participants might be either distracted by or also obstacles, such as pedestrians and other vehicles in the traffic. encode and maintain the items presented to the irrelevant Frequently, however, sensory input from the relevant visual hemifield. Encoding and maintenance of irrelevant distractors information is either interrupted by short events, such as eye- was demonstrated in previous studies [6, 3], however, in these blinks or saccades, or redirected to irrelevant visual stimuli, studies the irrelevant distractors were present in the same visual such as a phone screen when receiving a call. In such cases, a hemifield interspersed between relevant targets. In this study we temporary memory buffer known as visual working memory considered the possibility that the distractors significantly affect (VWM) [1], which allows us to actively maintain and integrate VWM performance even when presented to the irrelevant visual 137 hemifield. In this case, the participants might try to encode 2.2 Task and procedure items from both hemifields, spreading the VWM resources Participants completed multiple trials of the VWM task (Figure across both relevant and irrelevant items and reducing the 2). On each trial they were shown a brief array of black ability to maintain the items from the relevant visual hemifield. rectangles of different orientations ( targets) presented either on If that is the case, in such studies the results would one or both sides of the screen and were asked to remember the underestimate the VWM capacity for the relevant hemisphere. items presented in either left or right hemifield only, as indicated with an arrow ( initial cue). Following a brief delay a To address this concern, we designed an experiment in which second cue, which matched the initial cue and reminded the we directly manipulated the presence of distractors in the participants which objects they will need to base their response irrelevant visual hemifield, which enabled us to estimate and on, was presented. After another delay, the probe items were control for their effect. shown on both sides of the screen and the participants had to indicate by a button-press, whether there was a change in the 2. METHOD orientation of any of the rectangles on the relevant, previously indicated side of the screen, ignoring a possible change on the other side of the screen. 2.1 Participants Twenty-eight students (22 females) aged between 19 and 25 (M Two main factors were manipulated: side, which corresponded = 20.5, SD = 1.6) signed an informed consent to participate in to the side of the screen from which the participants had to an 1.5-hour experimental session. Five participants were remember orientation of the presented items ( left, right) and excluded from the analysis due to inadequate performance in distractors, corresponding to whether the items were presented the VWM task (accuracy was lower than chance in one of the to both or the relevant visual hemifield only ( distractor and no- task conditions). distractor condition) . Overall we tested 4 experimental conditions: 4 target items presented to the left (L4N) or right (R4N) visual hemifield with no distractors on the opposite 138 hemifield; and 4 target items presented to the left (L4D) or right (R4D) visual hemifield, with distractors present. In order to hinder verbal recoding of the visually presented stimuli, in addition to the VWM task, participants also performed a concurrent verbal suppression (VS) task, which was embedded between the trials of the main VWM task (Figure 2). Before the start of the VWM task, a suppression stimulus (VS target) consisting of blue-colored letters was presented for 2 s. Participants were instructed to maintain the letter sequence while performing the VWM task. After every eight VWM trials a VS probe was presented in red and the participants had to indicate by pressing the appropriate key, whether the probe was the same or different from the target. Data were collected in a sound-isolated room, allowing participants to attend to the task without any interruption. To control for eye-movements, we used an EyeLink 1000 system. 2.3 Data analysis Statistical analyses are based on estimates of working memory capacity (K). The capacity estimates were computed with Pashler’s formula [4]: ! K = N h − f , 1 − f where h and f are the observed hit and false alarm rates and N is the number of to-be-remembered items. 3. RESULTS Whereas the estimated VWM capacity did not differ between The goal of this study was to assess the effect of distractors in left and right hemispheres, the results indeed revealed lateralized change detection task on the VWM capacity. A significantly lower estimates of VWM capacity in the presence repeated measures ANOVA with within-subject factors side (left vs. absence of distractors. This suggests that participants were vs. right) and distractor (distractors vs. no-distractors) revealed not able to ignore the distracting items in the irrelevant a main effect of distractor, F(1, 22) = 19.7 , p < .001 , η2 = .06, contralateral visual hemifield, which resulted in lower estimate reflecting lower capacities in the presence of distractors (see of successfully remembered items (K) in the distractor present Figure 3). Neither the effect of side, F(1, 22) = 0.754 , p = . 394, conditions. These results are in line with prior studies using nor its interaction with distractor, F(1, 22) = 1.36 , p = .254, CDA [3], which have shown that participants find it hard to were significant. ignore irrelevant distractors when presented along the target items [6]. These studies, however, assessed the effect of 4. DISCUSSION distractors presented in the same visual hemifield, while not taking into consideration the possibility that the distractors can The aim of this experiment was to identify a potential significantly affect working memory performance even when shortcoming of lateralized change detection tasks used in the presented to the irrelevant visual hemifield. CDA studies in assessing VWM capacity, by manipulating the presence of distractors in the irrelevant visual hemifield. The effect of the distractors could be explained either by a Specifically, our concern was, that the presence of distractors reduced ability to encode the target items, or—as suggested by could result in reduced estimates of VWM capacity, when previous research [3]—by leading the participants to encode defined as the number of remembered target items. and maintain also irrelevant distractors, thereby reducing the effective capacity for encoding of the relevant items. 139 Though confirming the negative effect of the distractors, the 6. REFERENCES results do not enable unequivocal explanation of the mechanism underlying their effect. One possibility is that, similar to the [1] Baddeley, A., & Hitch, G. J. (1974). Working memory. In G. observations of the CDA studies [3], participants fail to ignore A. Bower (Ed.), Recent Advances in Learning and the irrelevant stimuli and maintain them along the relevant Motivation (Vol. 8, pp. 47-90). New York: Academic Press. items occupying limited working memory resources and leaving less of them available to successfully encode and maintain [2] Cowan, N. (2010). The Magical Mystery Four: How is target items. The second possibility is that distractors do not Working Memory Capacity Limited, and Why? Current occupy the limited capacity working memory store, but rather Directions in Psychological Science, 19(1), 51–57. disrupt the initial encoding of the relevant items, so that the active maintenance system fails to successfully engage and then [3] Luria, R., Balaban, H., Awh, E., & Vogel, E. K. (2016). The sustain the activity of the representations in the first place. contralateral delay activity as a neural measure of visual working memory. Neuroscience and Biobehavioral Whereas our experiment does not provide sufficient information Reviews, 62, 100–108. to distinguish between these two possibilities, future studies tracking the magnitude of the CDA in the presence and absence [4] Rouder, J. N., Morey, R. D., Morey, C. C., & Cowan, N. of distractors in the contralateral visual hemifield, could help (2011). How to measure working memory capacity in the resolve the dilemma. change detection paradigm. Psychonomic Bulletin & Review, 18(2), 324–330. [5] Vogel, E. K., & Machizawa, M. G. (2004). Neural activity predicts individual differences in visual working memory capacity. Nature, 428(6984), 748–751. 5. ACKNOWLEDGEMENTS [6] Vogel, E. K., McCollough, A. W., & Machizawa, M. G. This research was supported by the Slovenian Research Agency (2005). Neural measures reveal individual differences in grant J7-5553, J3-9264 and P5-0110. controlling access to working memory. Nature, 438(7067), 500–503. 140 Zmota luči v hladilniku in značilnosti opazovanja fenomenov z obrobja zavesti Urban Kordeš Viktorija Lipič Univerza v Ljubljani Univerza v Ljubljani Pedagoška fakulteta Pedagoška fakulteta Kadeljeva pl. 16, Ljubljana, Slovenija Center za kognitivno znanost +38615892200 +38631460174 urban.kordes@pef.uni-lj.si viktorija.lipic@gmail.com POVZETEK ujeto v tople roke: »snežinka namreč ob stiku z rokami spremeni V prispevku predstavljamo delne rezultate študije, v kateri smo svojo kvaliteto – ni več kristal, temveč le še kapljica«. poskušali odgovoriti na vprašanje, kaj se zgodi s fenomeni, ki se V nadaljevanju vprašanje naslovimo iz vidika treh raziskovalnih nahajajo na obrobju zavedanja, ko vanje usmerimo luč pozornosti. področij: kognitivne znanosti, fenomenološke filozofije in Osredotočamo se na fenomen srži, ki je del širšega doživljanja empiričnega raziskovanja doživljanja. udejanjanja znanja. V uvodnem poglavju predstavimo tako imenovano zmoto luči v hladilniku – vprašanje, ali so doživljajska Kognitivna znanstvenica Blackmore, podobno kot Dehaene, stanja, ki se jih zavedamo, ko o doživljanju reflektiramo, prisotna uporablja metaforo luči v hladiliniku [4]. Blackmoreova meni, da tudi takrat, ko o njih ne reflektiramo. Na podlagi empiričnih je naš občutek tekočega, neprekinjenega toka zavestnega podatkov o doživljanju fenomena srži in transformacijah tega doživljanja »velika iluzija« [4, str. 26] - vsakič ko se vprašamo doživljanja pod lučjo refleksivnega preiskovanja, poskušamo »sem zdaj zavesten?« se seveda ponudi pozitiven odgovor. Napaka, sklepati o možnostih reflektiranja fenomenov na obrobju in ki jo delamo je, da iz tega sklepamo, da smo zavestni tudi, ko tega veljavnosti empiričnih podatkov o fenomenih iz predrefleksivne ne preverjamo. Refleksivni akt oz. preizkušanje (angl. probing) je dimenzije zavesti. vedno na voljo, zaradi tega ni nikoli lukenj v našem (iluzornem) občutku neprekinjenega toka zavesti. Ključne besede Doživljanje, Po drugi strani, Rosenthal [5] ne vidi nobene iluzije. Njegovo empirična fenomenologija, refleksija stališče je, da zavestno izkustvo sovpada z zmožnostjo poročanja o njem. Ob tem pa poudarja, da je pogosto prisotna tudi vsebina, ki ni zavestna in o kateri udeleženci ne morejo poročati. Rosenthal 1. UVOD zagovarja t.i. plitko konceptualizacijo zavesti [6], katere stališče je, Raziskovanje doživljanja, torej subjektivne (oziroma da se je možno zavedati le doživljanja, ki je prej že bilo predmet fenomenološke) dimenzije zavesti, predstavlja za uveljavljene refleksije. To kar je 'zavestno' je vselej neposredno doživeto med paradigme empiričnega raziskovanja kopico problemov. Med njimi procesom refleksije. je verjetno največji pa tudi epistemološko najbolj zanimiv, Sodobni fenomenologi, na primer Zahavi [7], so po večini mnenja, problem, ki ga Dehaene in sodelavci [1] imenujejo zmota luči v da je zavestno izkustvo takoj na voljo, ko se po njem vprašamo, hladilniku (angl. refrigerator light fallacy)1. ravno zaradi predrefleksivnega samo-zavedanja. Pojasnjuje, da Po Dehaeneju, vsakič, ko se vprašamo o doživljanju, metaforično refleksivno samo-zavedanje temelji na predrefleksivnem samo- odpremo hladilnik in pogledamo ali je luč še vedno prižgana. zavedanju in da ravno zaradi tega lahko neposredno poročamo o Takoj, ko na določen del doživljajskega polja usmerimo pozornost, svojem izkustvu. Za ponazoritev predlaga naslednji primer: lahko opazimo bogato doživljajsko pokrajino, naseljeno z čustvenimi, »Če sem vključen v določeno zavestno dejavnost, kot je na primer miselnimi, telesnimi in drugimi fenomeni. O opaženem doživljanju branje zgodbe, moja pozornost ni usmerjena name ali na aktivnost lahko razmišljamo in poročamo. Vemo, da hladilnikova luč branja, temveč je na zgodbi. Če bi me nekdo v tem trenutku ugasne, ko vrata zapremo. Kako je s tem pri doživljanju? So doživljajska povprašal o mojem početju, bi takoj lahko odgovoril, da sem (že stanja, ki se jih zavedamo, ko o njih reflektiramo, kar nekaj časa) bral. Samo-zavedanje, na podlagi katerega lahko prisotna tudi takrat, ko o njih ne reflektiramo? Je akt raziskovanja odgovorim na to vprašanje, ni nekaj, kar je bilo pridobljeno v tistem (refleksivni akt) samo odprtina, skozi katero opazujemo doživljanje takšno trenutku, temveč je bilo prisotno v sami zavesti že ves čas« [7, str. kot je, ali refleksija nekaj »naredi« doživljanju – prispeva k 21]. Zahavi trdi, da se vselej predrefleksivno zavedamo lastnega konstrukciji bogate fenomenološke pokrajine? doživljanja in se lahko ponavadi takoj odzovemo »brez sklepanja Vprašanje, ki si ga zastavljamo, ni novo. Navajajo ga že avtorji kot ali opazovanja, če nas kdo vpraša, kaj počnemo, razmišljamo ali so Comte [2] in James [3], ki tok zavesti deli na na osrednjo, jasno vidimo, ali kaj smo občutili neposredno pred vprašanjem« [prav vsebino in območje, ki to glavno vsebino spremlja – t. i. obrobje. tam]. Poskus opazovanja obrobja James [3, str. 244] primerja s snežinko 1 Zmotna domneva, da je luč v hladilniku prižgana tudi, ko so vrata hladilnika zaprta – saj je vedno prižgana, ko odpremo vrata in pogledamo. 141 Za Zahavija je ključno sklicevanje na trenutek refleksivnega samo- izsek iz raziskave, usmerjen na vprašanje, kaj akt opazovanja zavedanja, ki razkriva obstoj (vselej prisotnega) predrefleksivnega (refleksije) »naredi« fenomenom, ki se navadno izmikajo fokusu samo-zavedanja. Jasno zanika možnost, da akt refleksije spremeni pozornosti. doživljanje. Zanj akt refleksije le razpira to, kar je vselej že bilo Predstavljen izsek je del širše študije doživljanja udejanjanja implicitno (predrefleksivno) zaznano. znanja, v kateri smo z uporabo tehnike drugoosebnega globinskega 1.1. Vloga akta opazovanja v raziskovanju fenomenološkega raziskovanja [13] zasledili fenomen srži. Gre za doživljanja posebno vrsto doživljanja, ki ga Kordeš in Demšar [14] opišeta kot občutek, da imamo na voljo védenje o določeni stvari oziroma občutenje bistva vsebine. Odgovor še ni jasno opredeljen v zavesti Kot vidimo, filozofska razprava o vplivu akta refleksije razkriva – imamo pa občutek, da je na voljo in da vemo kam je treba usmeriti problematiko luči v hladilniku, in z njo povezane različne pozornost, da se vsebina razkrije. Bistvena značilnost srži je, da se epistemološke možnosti pogleda na problematiko. Problem vpliva nahaja na obrobju zavedanja – kot taka je idealni poskusni zajček akta opazovanja na opazovano pa ni le teoretski – razumevanje za raziskovanje vpliva akta refleksije. Kaj se zgodi, če na ta obrobni vloge akta refleksije pri opazovanju doživljanja je ključno v fenomen posvetimo z lučjo pozornosti? Odgovor na to vprašanje empiričnem fenomenološkem raziskovanju, saj sta z njim bistveno lahko pomaga pri oceni veljavnosti predpostavk raziskovalnega povezani vprašanji vloge raziskovalnega aparata pri raziskovanju pristopa globoke konceptualizacije zavesti (ki trdi, da je možno doživljanja in narave s takšnim raziskovanje dobljenih raziskovati tudi fenomene, ki se po svoji naravi nahajajo na (fenomenoloških) podatkov. predrefleksivni dimenziji zavesti). Vodilni predstavniki prvoosebnega raziskovanja doživljanja priznavajo, da združevanje akta opazovanja z vsebinsko komponento opazovanja predstavlja pomemben izziv za empirično 2.1. Opis metode raziskovanje doživljanja [8]. Podobno kot vodilni predstavniki Osnovo drugoosebnega globinskega fenomenološkega sodobne fenomenološke filozofije, večina empiričnih raziskovanja predstavljata mikro-fenomenološki intervju, opisan v raziskovalcev doživljanja verjame, da je mogoče pridobiti Petitmengin [10] in metodološki obrat, pri čemer udeleženec doživljajske podatke o pristnem, z refleksijo neomadeževanem postane raziskovalec svojega doživljanja. S tem se njegova vloga doživljanju. spremeni iz udeleženca v soraziskovalca. Bistvena značilnost raziskovanja te vrste je, da udeleženec ni več samo vir informacij, Sodobno empirično prvoosebno raziskovanje vključuje celo vrsto ampak postane dejaven sodelavec v študiji. Metodološki obrat metodoloških pristopov opazovanja doživljanja, ki poskušajo ujeti udeleženca v soraziskovalca pa je možen samo, če udeleženca neokrnjeno, pristno (angl. pristine) doživljanje (npr. [9]). zanima raziskovanje in opazovanje lastnega doživljanja. Globinsko Raziskovalci, povezani s tehniko elicitacijskega in mikro- raziskovanje postane zares »globinsko« šele, ko udeleženci fenomenološkega raziskovanja [10, 11] verjamejo, da njihova vzamejo raziskovalno vprašanje za svoje in se raziskovani tematiki metoda omogoča razširitev pozornosti na predrefleksivno posvetijo. Takrat postanejo soraziskovalci [15]. dimenzijo doživljanja. Z drugimi besedami, verjamejo, da lahko vedno artikulirajo vsebine, ki se jih udeleženec v trenutku, ko so Raziskava je razdeljena v dve fazi. Prva faza vključuje začetne nastale ni zavedal (oziroma se jih je zavedal samo v intervjuje in je primarno namenjena urjenju udeležencev v predrefleksivnem »obrobju«). Dotični raziskovalni pristop Froese, samoopazovanju lastnega doživljanja. Udeleženci prve faze, ki Gould in Seth [6] imenujejo globoka konceptualizacija zavesti, kar skozi izvedbo začetnih intervjujev ugotovijo, da jih raziskovalna pomeni, da sta raziskovalno dostopni tako predrefleksivna kot tema in opazovanje doživljanja zanimata, postanejo aktivni refleksivna dimenzija zavesti. soraziskovalci in preidejo v drugo fazo raziskovanja. Na tej točki so soraziskovalci že dovolj izurjeni v samoopazovanju in lahko Mikro-fenomenološki intervju poskuša kultivirati posameznikovo začnejo samostojno raziskovati doživljanje fenomena v refleksijo in retrospekcijsko poročanje o doživljajskih podatkih. vsakodnevnem življenju. V drugi fazi se tako začenja razvijati Predrefleksivno dimenzijo opisujejo kot skrito, obrobno, skoraj že dialog v smislu skupnega participatornega sodelovanja med nezavedno raven zavesti, refleksivno pa kot zaznano vsebino, na raziskovalcem in soraziskovalcem [13]. katero je usmerjena pozornost [10]. Skozi proces izvedbe intervjuja raziskovalec vodi udeleženca do ponovnega podoživetja izkustva in pozornost udeleženca poskuša preusmeriti z eksplicitnega, refleksivnega dela, na implicitni, predrefleksivni, »obrobni« del 2.2. Poskusna teorija doživljanja [12]. Osredotočamo se na dobljene doživljajske opise, ki se nanašajo na primere, kjer je soraziskovalec poskušal pri sebi opaziti in Avtorji torej svojo metodo gradijo na predpostavki, da je možno opredeliti srž v intervjujih. reflektirati doživljanje s predrefleksivne dimenzije. Njihovo razumevanje razdelitve zavesti in vloge akta opazovanja je bistveno 2.2.1. Neoprijemljivost srži za razumevanje podatkov, ki jih producira njihova metoda. Je vera Soraziskovalci so poročali o velikih težavah pri poskusih avtorjev v sam obstoj predrefleksivne dimenzije zavesti in v reflektiranja na srž. Pogosto so se pojavljali opisi, da se srž izmika možnost razširitve pozornosti na to dimenzijo upravičena? pozornosti, da jo poskus opazovanja ne ujame, ali da izginja s poskusom opazovanja. Tipična poročila soraziskovalcev so bila v obliki občutkov, ki jih niso znali konkretizirati. Soraziskovalec 2. ŠTUDIJA: FENOMEN SRŽI Vprašanje veljavnosti in pomena prvoosebnih podatkov seveda presega okvire tega prispevka. Upamo pa, da bo opis rezultatov empirične študije lahko pomenil droben korak na poti k razumevanju fenomenalne zavesti. V tem razdelku predstavljamo 142 K01 2 je občutek neoprijemljivosti srži opisal kot: »bolj kot ga 3. ZNAČILNOSTI OPAZOVANJA skušam opisat, bolj ga ne morem«, »bolj ko ga poskušam najt, bolj OBROBNIH FENOMENOV mi izginja«. V istem intervjuju je v nadaljevanju poročal, da se doživljanje srži začenja pojavljati, ko srži aktivno ne poskuša iskati Ugotovitve o srži predstavljajo težavo predpostavki, na kateri in opazovati: »ko pa neham tko na en način to iskat, ko pa temeljijo svoja spoznanja raziskovalni pristopi, ki trdijo, da sta doživljam, pa se pojavi«. raziskovalno dostopni tako predrefleksivna kot refleksivna dimenzija zavesti (tj. globoka konceptualizacija zavesti [6]). 2.2.1.1. Opisi srži med udejanjanjem znanja Naši podatki kažejo, da je refleksivna dimenzija zlahka dostopna Nekateri soraziskovalci so poročali tudi o srži znotraj udejanjanja poročanju, na predrefleksivno pa ima raziskovanje transformirajoč znanja – kako se je občutek, da imajo védenje na voljo, kazal med učinek. Srž so soraziskovalci namreč uspeli zaznati, ampak se je procesom udejanjanja (pojavljanja znanja v zavesti). razprla v vsebino ali izginila takoj, ko je prešla v fokus pozornosti. Bistven skupni imenovalec soraziskovalcev, ki so poročali o srži Ko so se soraziskovalci pričeli osredotočati na fenomen srži, ga med udejanjanjem znanja, je zaznavanje srži na obrobju, ko je niso bili več zmožni opisati, saj ga je pozornost izničila ali pozornost usmerjena na pojavljanje vsebin v zavesti. Srž se kaže spremenila. Pokazalo se je, da reflektiranje vodi v spreminjanje kot občutek še neoprijemljivih možnosti oziroma nejasnih vsebin v fenomena – namesto jasnejšega zaznavanja fenomena srži se zgodi ozadju. Pozornost je že na določeni vsebini, obenem pa so razpiranje vsebin v obliki slik, besed itd. soraziskovalci zmožni zaznati srž kot oblak potencialnih novih Reflektiranje izbranega izkustva ne omogoča jasne zaznave ali vsebin, v katere se lahko misli nadaljujejo: »Torej poleg tega, da je razširitve pozornosti na predrefleksivno dimenzijo, temveč ustvari bil zvok grmenja, je bil hkrati kokr v ozadju blazno ene vsebine, novo vrsto doživljanja. To vzbuja dvom v konceptualizacijo take neoprijemljive. [...] Kot neke ne zares predstave, ampak možnosti. zavesti, ki jo zagovarja Petitmengin [10, 12], da je možno pozornost Gist [srž] možnih predstav, kaj bi ta zvok lahko bil. [...] preusmeriti na predrefleksivno doživljanje in le-to opazovati Takrat je bilo samo kot neko ne zares oprijemljivo ozadje« (F01). nedotaknjeno s strani refleksije. Soraziskovalec F01 je poročal o udejanjeni vsebini, ki je bila v Najbližje središču poskusu opisa predrefleksivnega doživljanja smo prišli z zavedanja in občutku prisotnosti srži, ki jo je zaznaval na opisi »neoprijemljivosti srži«, kamor spadajo poročila robu »kot neko ne zares oprijemljivo ozadje«, občutek, da je »v soraziskovalcev, ki so srž zaznali, a se je ta izmikala opazovanju. backgroundu [ozadju] že nekaj«. Podobno so drugi soraziskovalci Srž, zaznano v predrefleksivni dimenziji, so soraziskovalci opisali, govorili o občutku, »kot da se nek proces dogaja zadaj«. kot da izginja, ko poskušajo nanjo usmeriti pozornost. Doživljanje 2.2.1.2. Razpiranje srži v vsebino srži pa se spet pojavi, ko soraziskovalec preneha vanjo posegati z Naslednji primer soraziskovalca F01 ilustrira, da z usmeritvijo aktom refleksije. pozornosti na občutek védenja vsebina začne postajati vedno bolj Opisi srži med udejanjanjem znanja in razpiranjem srži v vsebino dostopna. Soraziskovalec obenem nima občutka, da sam aktivno kažejo, da ni jasnega prehoda med fenomenom na obrobju in povzroča to razpiranje vsebine, ampak da vsebina sama nastaja, reflektiranjem o tem fenomenu. Nekateri soraziskovalci so uspeli vse, kar je potrebno, je ohranjanje pozornosti: med reflektiranjem in poročanjem o vsebini na obrobju zaznati »Da sam usmerim fokus nanj in potem se mi itak razvije samo od fenomen srži, a le, ko je bila pozornost osredotočena na drugih sebe. [...] To razpiranje srži. Ko iz srži nastaja neka vsebina. Kjer v vsebinah v zavesti. bistvu nimam občutka, kokr da jaz to nekaj razpiram, da jaz povzročam to. Jaz samo držim pozornost na tem in pol se zadeva razvije in pol, ker imam pozornost na tem, vem, kaj to je.« 4.VIRI [1] Dehaene, S., Changeux, J. P., Naccache, L., Sackur, J. in Prepletanje srži in pojavljanja vsebin (udejanjanja znanja) opiše Sergent, C. 2006. Conscious, preconscious, and subliminal soraziskovalec F01 v naslednjem primeru: processing: a testable taxonomy . Trends in cognitive »Kokr, da začne se samo z eno vsebino, enim vedenjem. In pol se sciences, 10(5), 204-211. to začne razvijat v neko artikulirano misel. Ampak hkrati, ko se to [2] Overgaard, M. 2006. Introspection in science. Consciousness počas artikulira, ko se iz tega razvijajo neke besede, ki jih nek moj and Cognition, 15, 629-633. notranji glas izgovarja, je hkrati skoz še ta, ta srž tega, kar prihaja pol.« [3] James, W. 1890. The Principles of Psychology. Baltimore: Holt. Opis sledi poteku udejanjanja, ki se začenja s sržjo in nadaljuje v [4] Blackmore, S. 2002. There Is No Stream of Consciousness. udejanjanje znanja in njegovo artikulacijo. Srž se v podanem What is all this? What is all this stuff around me; this stream primeru opazi kot občutek možnosti vsebine, ki je lahko zanj na of experiences that I seem to be having all the time?. Journal voljo, kot prisotnost nečesa, kar še prihaja. Vsebina, ki je na voljo of Consciousness Studies, 9(5-6), 17-28. in se lahko pojavi v zavesti, je zaznana na obrobju kot »srž tega, kar prihaja pol« in lahko preide v središče pozornosti, kjer bo [5] Rosenthal, D. 2005. Consciousness and Mind. Oxford ilustrirana in se bo lahko razvila »v neko artikulirano misel«. University Press. New York. [6] Froese, T., Gould, C. in Seth, A. K. 2011. Validating and calibrating first-and second-person methods in the science of consciousness. Journal of Consciousness Studies, 18(2), 38. 2 Šifriranje soraziskovalcev jemljemo iz širše študije, kjer je vsak kodiran z določeno črko in zaporedno številko po principu X01. 143 [7] Zahavi, D. 2006. Subjectivity and selfhood: Investigating the [12] Petitmengin, C. 2007. Towards the source of thoughts: The first-person perspective. Cambridge, Mass: MIT Press. gestural and transmodal dimension of lived experience. [8] Depraz, N., Varela, F. J. in Vermersch, P. 2003. On Journal of consciousness Studies, 14(3), 54-82. becoming aware. John Benjamins. [13] Kordeš, U. in Klauser, F. 2016. Second-person in-depth phenomenological inquiry as an approach for studying [9] Hurlburt, R. T. 1992. Sampling Normal and Schizophrenic Inner Experience. Plenum Press, New enaction of beliefs. Interdisciplinary Description of Complex York. Systems, 14(4), 369–377. [14] Kordeš, U. in Demšar, E. 2018. Excavating belief about past [10] Petitmengin, C. 2006. Describing one’s subjective experience in the second person: An interview method for experience: experiential dynamics of the reflective act. the science of consciousness. Phenomenology and the Constructivist Foundations 13(2): 219–229. Cognitive Sciences, 5(3-4), 229-269. [15] Kordeš, U. in Smrdu, M. 2015. Osnove kvalitativnega [11] Vermersch, P. 1994. L'entretien d'explicitation. Paris: ESF. raziskovanja. Založba Univerze na Primorskem. 144 Zemljevid prepričanj – od propozicije do doživljanja Florian Klauser Urban Kordeš Univerza v Ljubljani Univerza v Ljubljani Pedagoška fakulteta Pedagoška fakulteta Kardeljeva pl. 16, Ljubljana, Slovenija Kardeljeva pl. 16, Ljubljana, Slovenija florian.klauser@pef.uni-lj.si urban.kordes@pef.uni-lj.si POVZETEK morajo biti ti konkretni primeri, oziroma katere vrste prepričanj je V prispevku predstavimo nekaj izsledkov iz še odvijajoče se za nas najbolj smiselno raziskovati. Hitro je postalo jasno, da raziskave doživljanja udejanjanja prepričanj, natančneje iz vprašanje izbora vrste prepričanj ni le tehničen problem, ampak ustvarjanja t.i. zemljevida prepričanj, ki naj bi služil razjasnitvi da kaže na širšo (in morda usodno) nejasnost: kaže nam, da je – pred empiričnim preverjanjem – pojma prepričanje in s prepričanji povezanih ekoloških situacij, za dobro razčistiti, kaj sploh namene empirične raziskave. Na splošno opredelimo pogled mislimo s pojmom prepričanja. Prvi poskusi določitve nabora analitične filozofije in ga primerjamo z ustreznih vrst prepričanj so namreč pokazali na ogromno širino opisi vsakodnevnih situacij, kot jih soraziskovalci beležijo v dnevnik. Na koncu situacij, ki jih lahko povežemo s prisotnostjo prepričanj. Nabor možnih primerov je tako raznolik, da utemeljuje dvom v predstavimo, katera vprašanja o naravi prepričanj in možnosti njihovega empiričnega raziskovanja je zemljevid prepričanj smiselnost obravnavanja pojma prepričanje kot jasno razčistil, in katera ostajajo še odprta. opredeljenega (kljub vsakodnevnim intuicijam, skozi katere ga navadno vidimo kot neproblematičnega ali celo očitnega). Ključne besede Pričujoč prispevek predstavlja del našega poskusa odgovoriti na prepričanje, doživljanje, propozicija, to vprašanje. Poudarjamo, da iščemo odgovor za potrebe fenomenologija fenomenološke raziskave (torej: kaj so prepričanja z vidika empirične fenomenologije in katera od teh so najustreznejša za 1. UVOD raziskovanje?) in ne splošnega odgovora, ki bi predstavljal V prispevku predstavimo nekaj izsledkov iz še odvijajoče se dokončno zamejitev pojma. raziskave doživljanja udejanjanja prepričanj. Celotna raziskava temelji na predpostavki, da se med pozivom po nekem prepričanju 2. SPLOŠNA PREDSTAVITEV POGLEDA in njegovo manifestacijo odvija vrsta vsebinsko bogatih ANALITIČNE FILOZOFIJE doživljajskih procesov. Naš raziskovalni cilj je te doživljajske V zadnjem stoletju je bilo brez dvoma največ povedanega o procese raziskati. Na primer, če nas nekdo vpraša (pozove po prepričanjih v sklopu analitične filozofije, zaradi tega smo prepričanju), kaj je voda, se odgovor ne pojavi takoj, temveč se predvidevali, da bi to področje moralo služiti kot dobra morda najprej izzove spomin na učbenik kemije iz osnovne šole, v izhodiščna točka za naše iskanje (za raziskovalno delo) uporabne katerem piše »voda je tekočina brez vonja in okusa«; ali pa morda razmejitve pojma. predstava kozarca vode in pričakovanje, da ob njenem zaužitju ne bo ne vonja ne okusa. Drug primer: v raziskavi je soraziskovalec1 Schwitzgebel [4] predstavi pregled splošno sprejetih pogledov in poročal o tem, da je poslušal predavanje v angleščini kjer je bil glavnih tokov razmišljanja analitične filozofije o prepričanjih, ki omenjen organ »adrenal gland« (slov. nadledvična žleza). Občutil jih opredeli kot propozicionalno naravnanost – kot to, ko neko je radovednost po tem, kje se ta organ v telesu nahaja in hkrati propozicijo vzamemo za resnično. Te propozicionalne tudi zavedanje, da on to vé, četudi se mu 'odgovor' v tem trenutku naravnanosti pa služijo kot vzrok za vedênje. Schwitzgebel ne pojavi v zavesti2. Z vztrajnim upiranjem pozornosti v ta opozori, da sam obstoj propozicionalnih naravnanosti kot vzrokov občutek se mu je le pojavila predstava trupa in občutek določene za delovanje ni splošno sprejeto gledišče. Izpostavi, da je polje razmišljanja o prepričanjih zelo široko in se ukvarja z različnimi lokacije v njem. povezanimi problematikami [4]. Osnovna metodologija, ki jo uporabljamo za raziskovanje je intervju, kjer poskušamo pridobiti iz konkretnih primerov čim Ena od teh je, na primer, vprašanje reprezentacij: ali so bolj natančne doživljajske prepričanja podatke o procesu manifestacije propozicije, ki so na kakšen način reprezentirane v prepričanj (več o metodologiji umu; ali bi bilo bolj primerno misliti prepričanja kot v Kordeš in Klauser [1]). Pri samo stanje, načrtovanju raziskave smo se tako soočili z vprašanjem, kakšni da je dana propozicija reprezentirana v umu. Ne glede na to, ali gre za reprezentacije prepričanj, ali prepričanja kot reprezentacije sveta, je vprašanje, v kakšni obliki se te reprezentacije nahajajo v 1 Izraz soraziskovalec nadomesti sicer uveljavljen izraz umu. Ali se pojavljajo v obliki notranjega jezika misli, ali gre udeleženec. Izbiro pojasnjujeta Kordeš in Klauser [1]. morda za bolj kompleksne sisteme zemljevidov. Med tem ko se z 2 reprezentacijami in njhovo strukturo ukvarja struja Občutek, ki je v tovrstnih doživljajskih procesih tako pogost, da si je zaslužil svoje ime – reprezentacionalistov, drugo strujo, dispozicionalisti (in srž. Več o srži in nekaterih drugih izsledkih raziskave v Kordeš in Demšar interpretacionalisti), bolj zanimajo vzorci dejanskega in [2], glej tudi Kordeš in potencialnega vedênja. Ti t.i. dispozicionalisti (in Klauser [1] ter Kordeš in Lipič [3] v tej publikaciji. 145 interpretacionalisti) analizirajo, kakšno vedênje mora oseba orodji logike. Zgoraj omenjen govor o t.i. implicitnih izkazovati, da zanjo lahko rečemo, da je prepričana. Nekateri prepričanjih, na primer, da vedeti, da ima vsak posameznik predstavniki – podstruja liberalnih dispozicionalistov – dopušča nešteto prepričanj (da planetov ni 10, da planetov ni 11 itd.). To tudi zasebne mentalne epizode (doživljanje?) kot relevantne za pomeni, da so (vsaj implicitna prepričanja) konstrukt logike in ne določanje prepričanja [4]. morejo imeti (ne fizične, ne mentalne) reprezentacije – z vidika Pojavi pa se tudi vprašanje o tem, ali prepričanja res obstajajo (kot empirije jih torej ni. V analitičnih razpravah (e.g. [4], [5]) nismo to zanikajo eliminitavisti), in ali so, tudi če ne obstajajo, našli jasnejše nastavke za možno empirično preizpraševanje prepričanja še vedno smiseln (ali celo nepogrešljiv) koncept za predstavljenih argumentov in ugotovitev. razlaganje vedênja [4]. Ni tudi povsem jasno, kaj predstavlja enoto prepričanja. Analitična filozofija se najpogosteje sklicuje na eno prepričanje Schwitzgebel [4] predstavi tudi pogosto sprejeto tipifikacijo prepričanj v kot povezano z eno propozicijo. A, kot že Schwitzgebel pojavne (angl. occurent) in dispozicijske (angl. [4] sam dispositional). Če se prepričanje le nahaja v spominu ali t.i. izpostavi, se lahko dve propozicionalno izraženo identični »škatlici prepričanj« prepričanji med seboj razlikujeta v kontekstu spremljajočih je to prepričanje dispozicijsko (i.e. subjekt ima dispozicijo ali nagnjenost k temu, da izrazi to prepričanje). prepričanj. Tako Ani kot Sanjay sta, na primer, prepričana, da so Ko pa subjekt prepričanje prikliče iz spomina za uporabo pri lososi ribe. A Ani je poleg tega prepričana, da so tudi kiti ribe, načrtovanju, sklepanju ali aktivnemu izjavljanju, pa je to med tem ko je Sanjay prepričan, da kiti niso ribe. Sta prepričanji prepričanje pojavno. Schwitzgebel od Ani in od Sanjaya o tem, da so lososi ribe, potem identični? [4] navede primer subjekta Harryja, ki meni, da so kariraste kravate grde. O tem le redko Mnenja se delijo: zagovorniki holizma bi to zanikali, zagovorniki razmišlja, tako da je to prepričanje večino časa dispozicijsko. Ko atomizma pa pritrdili [4]. pa le privre na površino njegovega uma, bi ga označili kot pojavno. 3. ZEMLJEVID PREPRIČANJ Schwitzgebel [4] izpostavi tudi podobno delitev na eksplicitna in Ker pregled razprav analitične filozofije ni zadovoljil naše potrebe implicitna prepričanja. Če je dani subjekt eksplicitno prepričan, po razmejitvi prepričanj za raziskovalne namene, smo se odločili da P, se v njegovi »škatlici prepričanj« nahaja reprezentacija z sami zgraditi zemljevid situacij, povezanih s prepričanji. Namen vsebino P. Če pa je subjekt prepričan implicitno, tovrstne je bil preiskati čim več čim bolj ekološko veljavnih tipov raprezentacije tam ni, a bi to prepričanje lahko hitro izpeljal iz že primerov situacij, ki jih soraziskovalci povezujejo z obstojem obstoječih. Schwitzgebel [4] to ilustrira s primerom števila prepričanj. Dobljenemu naboru, ki smo ga kasneje poskušali na planetov. Subjekt S sicer ni eksplicitno prepričan, da je planetov različne načine razmejiti, pravimo zemljevid prepričanj. manj kot 9, in manj kot 10, in manj kot 11; a ker ima eksplicitno Primere za zemljevid smo začeli zbirati s pomočjo skupine prepričanje, da je planetov 8, lahko hitro izpelje vsa omenjena soraziskovalcev – kolegov, ki jim je tema znana in so bili prepričanja o tem, da je število planetov manjše od katerega koli pripravljeni opazovati ter beležiti dogodke iz svojega življenja, za števila, ki je večje od 8. Ko za dano prepričanje to stori, prav to katere sodijo, da govorijo o prisotnosti prepričanja. V zemljevid prepričanje sicer postane eksplicitno, a pred tem ostaja implicitno. smo dodali tudi nekatere primere iz tekstov analitične filozofije Iz analitične filozofije pridobljeno znanje nam že pomaga zožiti (predvsem iz Schwitzgebel[4]), za katere sklepamo, da so nabor prepričanj, ki bi bila primerna za empirično raziskovanje – predstavniki posameznega tipa situacij, povezanih s prepričanji. dispozicijskih ali implicitnih prepričanj, na primer, ni mogoče Vsak primer smo razdelili v tri komponente: situacijsko, ki direktno preiskati, saj jih naša na ozaveščanju in eksplikaciji opisuje širši kontekst dogajanja in/ali vedênje osebe; doživljanja temelječa metoda takoj spremeni v pojavna ali fenomenalno, ki opisuje doživljanje osebe; in propozicionalno, ki eksplicitna. (skladno s tradicijo analitične filozofije) navede vsebino Pojem (propozicionalna) naravnanost vsekakor namiguje o prepričanja v obliki povedi 'S je prepričan, da P'. Če iz primera doživljajskem fenomenu (naravnanosti do določenega katera od komponent ni bila razvidna, je njen razdelek ostal intencionalnega sistema; [5]). Ni pa jasno kakšne vrste doživljanje prazen. Vsakemu primeru smo tudi dodelili ime, za lažje kasnejše naj bi takšna naravnanost bila, še manj pa – če je možno v nanašanje. doživljajskem polju opaziti intencionalni sistem ali vsebino Nad zbranimi primeri smo izvedli več kategorizacij in tako propozicije do katerega/katere smo naravnani. Po drugi strani izdelali zemljevid z grupiranji opisov situacij, ki kažejo na razne mnogi analitični avtorji podajajo vedênjske primere za svoje vrste prepričanj. Zemljevid smo razdelili na več podzemljevidov, razprave o prepričanjih – nekdo vzame s seboj dežnik, ker je ki predstavljajo različne kategorizacije primerov, kot na primer na prepričan, da bo deževalo. Predhodni izsledki raziskav [1][2] implicitna in eksplicitna prepričanja, na primere z opisano kažejo na to, da eno in isto vedênje lahko spremlja mnogo fenomenološko komponento in primere brez. Edina razdelitev, ki različnih doživljanj in obratno – kateri vidik je torej relevanten za skoraj v celoti pokrije ves prostor zbranih primerov je delitev na raziskovanje? Če pri blagajni sežem v žep po denarnici, katero »občutek, artikulacija, vedênje«, ki jo predstavljamo v prepričanje se tu manifestira: prepričanje, da imam v žepu nadaljevanju. denarnico; prepričanje, da je v denarnici denar; prepričanje, da žep in/ali denarnica obstajata; prepričanje, da svet obstaja; Delitev je osnovana na treh deskriptorjih. Vsak deskriptor predstavlja eno možno raziskovalno perspektivo na prepričanja: prepričanje, da jaz obstajam; prepričanje, da blagajničar od mene fenomenalno, verbalno in vedênjsko. Fenomenalni deskriptor pričakuje plačilo;…? Po večini pregledanih razprav sklepamo, da občut je odgovor: prepričanje, ki ga v opisanem vedênju želi videti avtor ek označuje prisotnost »občutka, da tako je.« Verbalni deskriptor artikulacija označuje prisotnost artikulacije prepričanja opisa. (v propozicionalni obliki) – »izrekanja, da…« . Vedênjski Z vidika analitične filozofije je očitno pomembna predvsem deskriptor vedênje pa označuje prisotnost »vedênja, kot da…« – vsebina propozicije – tisti element torej, ki ga lahko preiskujemo z torej obnašanja, ki namiguje na vpletenost določenega 146 prepričanja3. Vsak primer lahko vsebuje od enega do vseh treh Tabela 4. Primer za občutek in artikulacija: deskriptorjev. PRIMER »Voda« Tabela 1. Primer za samo občutek: PRIMER »Bambus« Vedenjski in Janeza nekdo vpraša, kaj je voda. Janez kontekstualni odgovori s »tekočina brez vonja in okusa.« Vedenjski in Hodim po cesti in za sekundo pogledam opis: kontekstualni desno v neka drevesa in nato spet drugam. opis: Fenomenološki Vidim dve tanjši debli, desno od njiju pa še Fenomenološki Janezu se pojavi vizualna predstava telesa opis: dve zeleni podolgovati navpični stvari – ena opis: vode, na katerem so valovi – ki ga pa še ima vodoravno rumenkasto črto. Ob nikoli ni videl. Skupaj s to predstavo se trenutku kot preblisk védenje, da je to tak pojavi splošen občutek o lastnostih vode, od debel in visok bambus. Občutek triumfa. teh se najbolj izpostavijo, nebesedno, »tekoče«, »brez vonja«, »brez okusa«. Propozicionalni opis: Propozicionalni Janez je prepričan, »da je voda tekočina opis: brez vonja in okusa.« Tabela 2. Primer za samo artikulacija: PRIMER »Robot« Vedenjski in Človeku podoben robot ima v sebi Tabela 5. Primer za občutek in vedênje: kontekstualni shranjeno, v strojnem jeziku, stavek opis: katerega prevod je »kemijska formula za PRIMER »Deževalo bo« vodo je H2O«. Če ga vprašamo, »iz katerih Vedenjski in Janez se odpravlja od doma. Na nebu vidi elementov je sestavljena voda?«, Robot kontekstualni temne oblake. S seboj vzame dežnik. dostopa do omenjenega stavka in ga opis: manipulira tako, da poda odgovor, kot bi ga Fenomenološki Ko že stopi ven opazi temne oblake in čuti podal človek. opis: védenje, da bo deževalo. Nameni se vzeti Fenomenološki dežnik. opis: Propozicionalni Janez je prepričan, »da bo deževalo.« Propozicionalni Robot je prepričan, »da je kemijska opis: opis: formula za vodo H2O.« Tabela 6. Primer za artikulacija in vedênje: Tabela 3. Primer za samo vedênje: PRIMER »Filozofski zombi« PRIMER »Bakterija« Vedenjski in Filozofski zombi je po vedenju identičen Vedenjski in V vodi živeča bakterija ima v svojem telesu kontekstualni navadni osebi, le da ničesar ne doživlja. kontekstualni magnet, ki se odziva na Zemljino magnetno opis: Tako kot Janez v prejšnjem primeru se opis: polje. Na severni polobli magnetni sever zombi od doma odpravlja z dežnikom. Ko kaže navzdol, tako se bakterija, pod ga vprašamo, zakaj s seboj nosi dežnik, vodstvom magneta, giblje navzdol proti odvrne: »ker bo deževalo.« globlji vodi in sedimentu, stran od kisika, ki Fenomenološki je zanjo škodljiv. opis: Fenomenološki Propozicionalni Filozofski zombi je prepričan, »da bo opis: opis: deževalo.« Propozicionalni Bakterija je prepričana, »da je v smeri, v opis: katero jo vodi magnet, manj kisika.« Tabela 7. Primer za vse tri: PRIMER »Deževalo bo 2« Vedenjski in Janez se od doma odpravlja z dežnikom. Ko kontekstualni ga vprašamo, zakaj ga nosi s seboj, odvrne: opis: »ker bo deževalo.« Fenomenološki Janez se ob vprašanju spomni na občutek, opis: ki ga je čutil ob pogledu na temne oblake – na védenje, da bo deževalo. 3 Zavedamo se, da je strogo gledano artikulacija tudi le oblika vedênja, a akt izrekanja ocenjujemo za pomembno drugačnega Propozicionalni Janez je prepričan, »da bo deževalo.« od drugih načinov vedênja. opis: 147 4. ZAKLJUČEK: V KOLIKŠNI MERI spremljata manifestacijo odgovora »tekočina brez vonja in okusa.« Recimo, da Janez doživlja predstavo kozarca vode in PROPOZICIJE RES ODRAŽAJO pričakovanje, da ob njenem zaužitju ne bo ne vonja ne okusa; in PREPRIČANJA Jože doživlja spomin na učbenik kemije iz osnovne šole, v V zaključku navajamo nekaj spoznanj, ki nam jih je prinesel katerem piše »voda je tekočina brez vonja in okusa.« razmislek o možnostih empiričnega raziskovanja prepričanj. Lista Pripisovanje enake propozicionalne vrednosti obema primeroma je razdrobljena in ni popolna – gre bolj za nastavke oziroma smeri (»voda je tekočina brez vonja in okusa«) bi bilo najverjetneje za nadaljnjo razpravo. napačno. Če bi že morali pripisati propozicionalno vrednost Prvo presenečenje, ki smo ga opazili po razmejitvi prostora (zavedajmo se, kako arbitrarno je to dejanje), bi bili dejanskemu situacij, povezanih s prepričanji je, da so primeri, ki vključujejo stanju še najbližje naslednji propoziciji: propozicionalno komponento skoraj izključno tisti, ki prihajajo iz 1) Janez je prepričan, da je voda tekočina brez vonja in okusa. teoretskih logov analitične filozofije. Izjemno malo ekološko veljavnih primerov (torej tistih, nabranih iz dnevniških zapiskov 2) Jože je prepričan, da je ustrezen odgovor na vprašanje »kaj je soraziskovalcev) ima jasno definirano propozicionalno vsebino. voda?« zapisan v učbeniku kemije iz osnovne šole in se glasi Fenomenalna komponenta tovrstnih primerov največkrat ne »voda je tekočina brez vonja in okusa.« vključuje artikuliranih propozicij (e.g. PRIMER »Bambus«). V obeh primerih je možno govoriti o propozicionalni Propozicionalni opis prepričanja bi lahko bil venomer podan šele naravnanosti. Razlika je, da je prva naravnana na stališče do post hoc in arbitrarno – v skladu s pričakovanji in razumevanjem narave elementa sveta. Druga pa na stališče o izjavi avtoritete (v interpreta. takšni ali drugačni obliki). Zanimivo vprašanje povezano z ugotovitvijo, da ne iz vedênja, ne Takšna analiza odpre nova vprašanja, nekatera zelo praktična in iz doživljanja, ne moremo enoznačno sklepati na propozicionalno povezana z vsakodnevnim življenjem. Na primer vprašanje o vrednost prepričanja je: kakšno vrednost ima razprava o naravi in možnostih preverjanja znanja (imajo res vsi, ki opravijo propozicijah za empirična razmišljanja o duševnosti? Sestavljanje zemljevida prepričanj je pokazalo na test z enakim rezultatom, enako znanje – oziroma bolje: enako izziv fenomenološkega vrsto znanja?). aspekta tovrstnih fenomenov (in morda s tem pojasnilo izogibanje tej temi v analitični filozofiji) – fenomenologija drže (angl. attitude), še bolj pa fenomenologija tistega, na kar se drža nanaša 5. VIRI (torej propozicije), odpira zelo zapletena vprašanja. Kako se lotiti [1] Kordeš, U., in Klauser, F. 2016. Second-person in-depth raziskovanja doživljanja fenomena, ki ga bistveno (izključno?) phenomenological inquiry as an approach for studying določa vsebina (oz. propozicionalna vrednost) ni jasno. Kljub enaction of beliefs. Interdisciplinary Description of Complex temu ocenjujemo, da se odgovor skriva v natančni empirični Systems: INDECS, 14(4), 369-377. preiskavi. [2] Kordeš, U., in Demšar, E. 2018. Excavating belief about past Kako nam lahko pomaga dobljeni zemljevid pri nadaljnjem experience: experiential dynamics of the reflective act. raziskovanju doživljajskih procesov, povezanih z manifestacijo Constructivist Foundations, 13(2): 219-229. prepričanj? Vsekakor smo jasno razmejili področja, ki niso [3] Kordeš, U., in Lipič, V. 2018. Zmota luči v hladilniku in dosegljiva našim raziskavam (npr. celotno področje z vedênjem značilnosti opazovanja fenomenov z obrobja zavesti. Zbornik določenih situacij). Ostajata še področji artikuliranih prepričanj in 21. mednarodne multikonference Informacijska družba – IS tistih situacij, ki jih opredeljuje doživljajski občutek, »da tako je« 2018. (oz. – z besedami analitične filozofije – propozicionalna [4] Schwitzgebel, E. 2015. Belief. The Stanford Encyclopedia of naravnanost). Philosophy (Summer 2015 Edition). Edward N. Zalta (ur.), Pomembno je opaziti, da med tema področjema ni popolnega URL=https://plato.stanford.edu/archives/sum2015/entries/bel prekrivanja. Ne le, da mnogokrat propozicionalna naravnanost ni ief/. artikulirana (ampak je prisoten le občutek takšnosti); še bolj [5] Fodor, J. A. 1990. A theory of content and other essays. The zanimivo je neujemanje v nasprotno smer: velikokrat je enaka MIT press. artikulacija pospremljena z različnimi doživljajskimi vsebinami. Vrnimo na primer odgovarjanja na vprašanje »kaj je voda?«, kjer smo predstavili dva možna doživljajska procesa, ki lahko 148 Zbornik 21. mednarodne multikonference INFORMACIJSKA DRUŽBA – IS 2018 Zvezek C Proceedings of the 21st International Multiconference INFORMATION SOCIETY – IS 2018 Volume C Odkrivanje znanja in podatkovna skladišča - SiKDD Data Mining and Data Warehouses - SiKDD Uredila / Edited by Dunja Mladenić, Marko Grobelnik http://is.ijs.si 11. oktober 2018 / 11 October 2018 Ljubljana, Slovenia 149 150 PREDGOVOR Tehnologije, ki se ukvarjajo s podatki so v devetdesetih letih močno napredovale. Iz prve faze, kjer je šlo predvsem za shranjevanje podatkov in kako do njih učinkovito dostopati, se je razvila industrija za izdelavo orodij za delo s podatkovnimi bazami, prišlo je do standardizacije procesov, povpraševalnih jezikov itd. Ko shranjevanje podatkov ni bil več poseben problem, se je pojavila potreba po bolj urejenih podatkovnih bazah, ki bi služile ne le transakcijskem procesiranju ampak tudi analitskim vpogledom v podatke – pojavilo se je t.i. skladiščenje podatkov (data warehousing), ki je postalo standarden del informacijskih sistemov v podjetjih. Paradigma OLAP (On-Line-Analytical-Processing) zahteva od uporabnika, da še vedno sam postavlja sistemu vprašanja in dobiva nanje odgovore in na vizualen način preverja in išče izstopajoče situacije. Ker seveda to ni vedno mogoče, se je pojavila potreba po avtomatski analizi podatkov oz. z drugimi besedami to, da sistem sam pove, kaj bi utegnilo biti zanimivo za uporabnika – to prinašajo tehnike odkrivanja znanja v podatkih (data mining), ki iz obstoječih podatkov skušajo pridobiti novo znanje in tako uporabniku nudijo novo razumevanje dogajanj zajetih v podatkih. Slovenska KDD konferenca pokriva vsebine, ki se ukvarjajo z analizo podatkov in odkrivanjem znanja v podatkih: pristope, orodja, probleme in rešitve. INTRODUCTION Data driven technologies have significantly progressed after mid 90’s. The first phases were mainly focused on storing and efficiently accessing the data, resulted in the development of industry tools for managing large databases, related standards, supporting querying languages, etc. After the initial period, when the data storage was not a primary problem anymore, the development progressed towards analytical functionalities on how to extract added value from the data; i.e., databases started supporting not only transactions but also analytical processing of the data. At this point, data warehousing with On-Line-Analytical-Processing entered as a usual part of a company’s information system portfolio, requiring from the user to set well defined questions about the aggregated views to the data. Data Mining is a technology developed after year 2000, offering automatic data analysis trying to obtain new discoveries from the existing data and enabling a user new insights in the data. In this respect, the Slovenian KDD conference (SiKDD) covers a broad area including Statistical Data Analysis, Data, Text and Multimedia Mining, Semantic Technologies, Link Detection and Link Analysis, Social Network Analysis, Data Warehouses. 151 PROGRAMSKI ODBOR / PROGRAMME COMMITTEE Dunja Mladenić, Artificial Intelligence Laboratory, Jožef Stefan Institute, Ljubljana Marko Grobelnik, Artificial Intelligence Laboratory, Jožef Stefan Institute, Ljubljana 152 Preparing multi-modal data for natural language processing Erik Novak Jasna Urbančič Miha Jenko Jožef Stefan Institute Jožef Stefan Institute Jožef Stefan Institute Jožef Stefan International Ljubljana, Slovenia Ljubljana, Slovenia Postgraduate School jasna.urbancic@ijs.si miha.jenko@ijs.si Ljubljana, Slovenia erik.novak@ijs.si ABSTRACT to find similar items based on the model input. Throughout the In education we can find millions of video, audio and text educa- paper we focus on educational material but the approach can be tional materials in different formats and languages. This variety and generalized to other multi-modal data sets. multimodality can impose difficulty on both students and teachers The reminder of the paper is structured as follows. In section 2 since it is hard to find the right materials that match their learning we go over related work. Next, we present the data preprocessing preferences. This paper presents an approach for retrieving and pipeline which is able to process different types of data – text, video recommending items of different modalities. The main focus is on and audio – and describe each component of the pipeline in section the retrieving and preprocessing pipeline, while the recommenda- 3. A content based recommendation model that uses Wikipedia tion engine is based on the k-nearest neighbor method. We focus concepts to compare materials is presented in section 4. Finally, we on educational materials, which can be text, audio or video, but the present future work and conclude the paper in section 5. proposed procedure can be generalized on any type of multi-modal data. 2 RELATED WORK KEYWORDS In this section we present the related work which the rest of the paper is based on. We split this section into subsections – multi- Multi-modal data preprocessing, machine learning, feature extrac- modal data preprocessing and recommendation models. tion, recommender system, open educational resources Multi-modal Data Preprocessing. Multi-modal data can be seen ACM Reference Format: as classes of different data types from which we can extract similar Erik Novak, Jasna Urbančič, and Miha Jenko. 2018. Preparing multi-modal features. In the case of educational material the classes are video, data for natural language processing. In Proceedings of Slovenian KDD Con- audio and text. One of the approaches is to extract text from all ference (SiKDD’18). ACM, New York, NY, USA, Article 4, 4 pages. https: class types. In [6] the authors describe a Machine Learning and //doi.org/10.475/123_4 Language Processing automatic speech recognition system that can convert audio to text in the form of transcripts. The system can 1 INTRODUCTION also process video files as they are also able to extract audio from There are millions of educational materials that are found in dif- it. Their model was able to achieve a 13.3% word error rate on an ferent formats – courses, video lectures, podcasts, simple text doc- English test set. These kind of systems are useful for extracting uments, etc. Because of its vast variety and multimodality it is text from audio and video but would need to have a model for each difficult for both students and teachers to find the right materi- language. als that will match their learning preferences. Some like to read a Recommendation models. These models are broadly used in short scientific papers while others just like to sit back and watch many fields – from recommending videos based on what the user a lecture that can last for hours. Additionally, materials are written viewed in the past, to providing news articles that the user might in different languages, which is a barrier for people who are not be interested in. One of the most used approaches is based on fluent in the language the material is written in. Finding a good collaborative filtering [16], which finds users that have similar approach of providing educational material would help improving preferences with the target user and recommends items based on their learning experience. their ratings. Recommender systems now do not contain only one In this paper we present a preprocessing pipeline which is able algorithm but multiple which return different recommendations. to process multi-modal data and input it in a common semantic Authors of [10] discuss about the various algorithms that are used space. The semantic space is based on Wikipedia concepts extracted in the Netflix recommender system (top-n video ranker, trending from the content of the materials. Additionally, we developed a con- now, continue watching, and video-video similarity), as well as the tent based recommendation model which uses Wikipedia concepts methods they use to evaluate their system. A high level description of the Youtube recommender system is found in [3]. They developed Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed a candidate generation model and a ranking model using deep for profit or commercial advantage and that copies bear this notice and the full citation learning. Both Netflix and Youtube recommend videos based on on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). users’ interaction with them and the users history. To some extent SiKDD’18, October 2018, Ljubljana, Slovenia this can be used for educational resources but cannot be generalized © 2018 Copyright held by the owner/author(s). on the whole multi-modal data set since we cannot acquire data ACM ISBN 123-4567-24-567/08/06. about users’ interaction with, for instance, text. https://doi.org/10.475/123_4 153 SiKDD’18, October 2018, Ljubljana, Slovenia Erik Novak, Jasna Urbančič, and Miha Jenko A collaborative filtering based recommendation system for the Crawling. The first step is to acquire the educational materials. We educational sector is presented in [8]. They evaluated educational have targeted four different OER repositories (MIT OpenCourse- content using big data analysis techniques and recommended courses Ware, Università di Bologna, Université de Nantes and Videolec- to students by using their grades obtained in other subjects. This tures.NET), for which we used their designated APIs or developed gives us insight into how recommendations can be used in educa- custom crawlers to acquire their resources. For each material we tion but our focus is to recommend educational materials rather acquired its metadata, such as the materials title, url, type, language than courses. In a sense courses can be viewed as bundles of ed- in which it is written and its provider. These values are used in the ucational material; thus, our interest is recommending “parts of following steps of the pipeline as well as to represent the material courses” to the user. in the recommendations. Formatting. Next, we format the acquired material metadata. We designate which attributes every material needs to have as well as 3 DATA PREPROCESSING set placeholders for the features extracted in the following steps In this paper we focus on open educational resources (OER), which of the pipeline. By formatting the data we set a schema which are freely accessible, openly licensed text, media, and other digi- makes checking which attributes are missing easy. We do not have tal assets that are useful for teaching, learning and assessing [21]. a mechanism for handling missing attributes in the current pipeline These are found in different OER repositories maintained by univer- iteration but we will dedicate time to solve this problem in the sities, such as MIT OpenCourseWare [12], Università di Bologna [7], future. Université de Nantes [4] and Universitat Politècnica de València [5], Text Extraction. The third step, we extract the content of each as well as independent repositories such as Videolectures.NET [20], material in text form. Since the material can be a text, video or a United Nations award-winning free and open access educational audio file to handled each file type separately. video lectures repository. For text we employed textract [1] to extract raw text from the For processing the different OER we developed a preprocessing given text documents. The module omits figures and returns the pipeline that can handle each resource type and output metadata content as text. The extracted text is not perfect - in the case of used for comparing text, audio and video materials. The pipeline is materials for mathematics it does not know how to represent mathe- an extension of the one described in [11]; its architecture is shown matical equations and symbols. In that case, it replaces the equations in figure 1. What follows are the descriptions of each component with textual noise. Currently we do nothing to handle this problem in the preprocessing pipeline. and use the output as is. For video and audio we use the subtitles and/or transcriptions to represent the materials content. To do this, we use transLectures [18] which generates transcriptions and translations of a given video and audio. The languages it supports are English, Spanish, German and Slovene. The output of the service is in dfxp format crawling [17], a standard for xml caption and subtitles based on timed text markup language, from which we extract the raw text. Wikification. Next, we send the material through wikification - a forma�ng process which identifies and links material textual components to the corresponding Wikipedia pages [15]. This is done using Wikifier aud [2], which returns a list of Wikipedia concepts that are most likely i vi o de text o related to the textual input. The web service also supports cross- and multi-linguality which enables extracting and annotating materials textract transLectures text in different languages. extrac�on Wikifier’s input text is limited to 20k characters, because of which longer text cannot be processed as a whole. We split longer text into chunks of at most 10k characters and pass them to Wikifier. Here we are careful not to split the text in the middle of a sentence wikifica�on and if that is not possible, to at least not split any words. We split the text as follows. First we make a 10k characters long substring of the text. Next, we identify the last character in the substring that signifies the end of a sentence (a period, a question storing mark, or an exclamation point) and split it at that character. If there is no such character we find the last whitespace in the substring and split it there. In the extreme case where no whitespaces are found we take the substring as is. The substring becomes one chunk Figure 1: The preprocessing pipeline architecture. It is de- of the original text. We repeat the process on the remaining text signed to handle each data type as well as extract features to until it is fully split into chunks. support multi- and cross-linguality. When we pass these chunks into Wikifier, it returns Wikipedia concepts related to the given chunk. These concepts also contains 154 Preparing multi-modal data for natural language processing SiKDD’18, October 2018, Ljubljana, Slovenia the Cosine similarity between the Wikipedia concept page and the can be represented in various file formats, such as pdf and docx given input text. To calculate the similarity between the concept for text, wmv and mp4 for video, and mp3 for audio. We visualized and the whole material we aggregated the concepts by calculating the distribution of materials over file types in figure 4, but we only the weighted sum show types with more than 100 items available. n Õ Li S , k = s L ki i =1 where Sk is the aggregated Cosine similarity of concept k, n is the number of chunks for which Wikifier returned concept k, Li is the length of chunk i, L is the length of the materials raw text, and ski is the Cosine similarity of concept k to chunk i. The weight Li represents the presence of concept k, found in chunk i, in the L whole material. The aggregated Wikipedia concepts are stored in the materials metadata attribute. Data Set Statistics. In the final step, we validate the material at- Figure 4: Number of items per file type in logarithm scale. tributes and store it in a database. The OER material data set consists The dominant file type is text (pdf, pptx and docx), followed of approximately 90k items. The distribution of materials over the by video (mp4). four repositories is shown in figure 2. As seen from the figure, the dominant file type is text (pdf, pptx and docx) followed by video (mp4). The msi file type is an installer package file format used by Windows but it can also be a textual document or a presentation. If we generalize the file type distribu- tion over all OER repositories we can conclude that the dominant file type is text. This will be taken into count when improving the preprocessing pipeline and recommendation engine. Figure 2: Number of materials per repository crawled in log- 4 RECOMMENDER ENGINE arithm scale. Most materials come from MIT OpenCourse- There are different ways of creating recommendations. Some em- Ware followed by Videolectures.NET. ploy users’ interests while other are based on collaborative filter- ing. In this section we present our content based recommendation Some of the repositories offer material in different languages. engine which uses the k-nearest neighbor algorithm [13]. What All repositories together cover 103 languages, however for only 8 follows are descriptions of how the model generates recommenda- languages the count of available materials is larger than 100. The tions based on the user’s input, which can be either the identifier distribution of items over languages is shown in figure 3 where we of the OER in the database or a query text. only show languages with more than 100 items available. Most of Material identifier. When the engine receives the material identi- the materials is in English, followed by Italian and Slovene. The fier (in our case the url of the material) we first check if the material “Unknown” column shows that for about 6k materials we were is in our database. If present, we search for k most similar mate- not able to extract the language. To acquire this information, we rials to the one with the given identifier based on the Wikipedia will improve the language extraction method in our preprocessing concepts. Each material is represented by a vector of its Wikipedia pipeline. concepts where each value is the aggregated Cosine similarity of the corresponding Wikipedia concept page to the material. By calcu- lating the Cosine similarity between the materials the engine then selects k materials with the highest similarity score and returns them to the user. Because of the nature of Wikipedia concepts this approach returns materials written in different languages - which helps overcoming the language barrier. Query text. When the engine receives the query text we search for materials with the most similar raw text using the bag-of-words model. Each material is represented as a bag-of-words vector where each value of the vector is the tf-idf of the corresponding word. The Figure 3: Number of materials per language in logarithm materials are then compared using the Cosine similarity and the scale. Most of the material is in English, followed by Italian engine again returns the k materials that have the highest similarity and Slovenian. score. This approach is simple but it is unable to handle multilingual documents. This might be overcome by first sending the query text As shown in before the preprocessing pipeline is designed to to Wikifier to get its associated Wikipedia concepts and use them handle different types of material - text, video and audio. Each type in a similar way as described in the Material identifier approach. 155 SiKDD’18, October 2018, Ljubljana, Slovenia Erik Novak, Jasna Urbančič, and Miha Jenko 4.1 Recommendation Results In the future we will evaluate the current recommendation en- The described recommender engine is developed using the QMiner gine and use it to compare it with other state-of-the-art. We intend platform [9] and is available at [14]. When the user inputs a text to use A/B testing to optimize the models based on the user’s inter- query the system returns recommendations similar to the given action with them. We wish to improve the engine by collecting user text. These are shown as a list where each item contains the title, url, activity data to determine what materials are liked by the users, description, provider, language and type of the material. Clicking explore different deep learning methods to improve results, and on an item redirects the user to the selected OER. develop new representations and embeddings of the materials. We have also discussed with different OER repository owners We also aim to improve the preprocessing pipeline by improving and found that they would be interested in having the recommen- text extraction methods, handle missing material attributes, and dations in their portal. To this end, we have developed a compact adding new feature extraction methods to determine the topic and recommendation list which can be embedded in a website. The rec- scientific field of the educational material as well as their quality. ommendations are generated by providing the material identifier or raw text as query parameters in the embedding url. Figure 5 shows ACKNOWLEDGMENTS the embed-ready recommendation list. This work was supported by the Slovenian Research Agency and X5GON European Unions Horizon 2020 project under grant agree- ment No 761758. REFERENCES [1] David Bashford. 2018. GitHub - dbashford/textract: node.js module for extracting text from html, pdf, doc, docx, xls, xlsx, csv, pptx, png, jpg, gif, rtf and more! https://github.com/dbashford/textract. Accessed: 2018-09-03. [2] Janez Brank, Gregor Leban, and Marko Grobelnik. 2017. Annotating documents with relevant Wikipedia concepts. Proceedings of SiKDD. [3] Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems. ACM, 191–198. [4] Université de Nantes. 2018. Plate-forme d’Enseignement de l’Université de Nantes. http://madoc.univ-nantes.fr/. Accessed: 2018-09-03. [5] Universitat Politècnica de València. 2016. media UPV. https://media.upv.es/#/ portal. Accessed: 2018-09-03. [6] Miguel Ángel del Agua, Adrià Martínez-Villaronga, Santiago Piqueras, Adrià Giménez, Alberto Sanchis, Jorge Civera, and Alfons Juan. 2015. The MLLP ASR Systems for IWSLT 2015. In Proc. of 12th Intl. Workshop on Spoken Language Translation (IWSLT 2015). Da Nang (Vietnam), 39–44. http://workshop2015.iwslt. org/64.php [7] Università di Bologna. 2018. Universita di Bologna. https://www.unibo.it/it. Accessed: 2018-09-03. [8] Surabhi Dwivedi and VS Kumari Roshni. 2017. Recommender system for big data in education. In E-Learning & E-Learning Technologies (ELELTECH), 2017 5th National Conference on. IEEE, 1–4. [9] Blaz Fortuna, J Rupnik, J Brank, C Fortuna, V Jovanoski, M Karlovcec, B Kazic, K Kenda, G Leban, A Muhic, et al. 2014. » QMiner: Data Analytics Platform for Processing Streams of Structured and Unstructured Data «, Software Engineering for Machine Learning Workshop. In Neural Information Processing Systems. [10] Carlos A Gomez-Uribe and Neil Hunt. 2016. The netflix recommender system: Algorithms, business value, and innovation. ACM Transactions on Management Figure 5: An example of recommended materials for the lec- Information Systems (TMIS) 6, 4 (2016), 13. ture with the title “Is Deep Learning the New 42?” published [11] Erik Novak and Inna Novalija. 2017. Connecting Professional Skill Demand with Supply. Proceedings of SiKDD. on Videolectures.NET [19]. The figure shows cross-lingual, [12] Massachusetts Institute of Technology. 2018. MIT OpenCourseWare | Free Online cross-modal, and cross-site recommendations. Course Materials. https://ocw.mit.edu/index.htm. Accessed: 2018-09-03. [13] Leif E Peterson. 2009. K-nearest neighbor. Scholarpedia 4, 2 (2009), 1883. [14] X5GON Project. 2018. X5GON Platform. https://platform.x5gon.org/search. The recommendation list consists of the top 100 materials based Accessed: 2018-09-04. [15] Lev Ratinov, Dan Roth, Doug Downey, and Mike Anderson. 2011. Local and on the query input. As shown in the figure the recommendation global algorithms for disambiguation to wikipedia. In Proceedings of the 49th contain materials of different types, are provided by different reposi- Annual Meeting of the Association for Computational Linguistics: Human Language tories and written in different languages. We have not yet evaluated Technologies-Volume 1. Association for Computational Linguistics, 1375–1384. [16] Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. 2001. Item-based the recommendation engine but we intend to do it in the future. collaborative filtering recommendation algorithms. In Proceedings of the 10th international conference on World Wide Web. ACM, 285–295. 5 FUTURE WORK AND CONCLUSION [17] Speechpad. 2018. DFXP (Distribution Format Exchange Profile) | Speechpad. https://www.speechpad.com/captions/dfxp. Accessed: 2018-09-04. In this paper we present the methodology for processing multi- [18] transLectures. 2018. transLectures | transcription and translation of video lectures. modal items and creating a semantic space in which we can compare http://www.translectures.eu/. Accessed: 2018-09-03. [19] VideoLectures.NET. 2018. Is Deep Learning the New 42? - Videolectures.NET. these items. We acquired a moderately large open educational re- http://videolectures.net/kdd2016_broder_deep_learning/. Accessed: 2018-09-03. sources data set, created a semantic space with the use of Wikipedia [20] VideoLectures.NET. 2018. VideoLectures.NET - VideoLectures.NET. http:// videolectures.net/. Accessed: 2018-09-03. concepts and developed a basic content based recommendation en- [21] Wikipedia. 2018. Open educational resources - Wikipedia. https://en.wikipedia. gine. org/wiki/Open_educational_resources. Accessed: 2018-09-03. 156 TOWARDS SMART STATISTICS IN LABOUR MARKET DOMAIN Inna Novalija Marko Grobelnik Jožef Stefan Institute Jožef Stefan Institute Jamova cesta 39, Ljubljana, Slovenia Jamova cesta 39, Ljubljana, Slovenia inna.koval@ijs.si marko.grobelnik@ijs.si ABSTRACT respect to defined scenarios – demand analysis, skills ontology development and skills ontology evolution. In this paper, we present a proposal for developing smart labour market statistics based on streams of enriched textual data and illustrate its application on job vacancies from European 2. BACKGROUND countries. We define smart statistics scenarios including demand The development of smart labour market statistics touches a analysis scenario, skills ontology development scenario and skills number of issues from labour market policies area and would ontology evolution scenario. We identify stakeholders – provide contributions to questions related to: consumers for smart statistics and define the initial set of smart - job creation, labour market statistical indicators. - education and training systems, - labour market segmentation, KEYWORDS - improving skill supply and productivity. Smart statistics, labour market, demand analysis. For instance, the analysis of the available job vacancies could offer an insight into what skills are required in the particular area. Effective trainings based on skills demand could be organized and 1. INTRODUCTION that would lead into better labour market integration. A number of stakeholder types will benefit from the development An essential feature of modern economy is the appearance of new of smart labour market statistics. In particular, the targeted skills, such as digital skills. For instance, e-skills lead to the stakeholders are: exponential increases in production and consumption of data. - Statisticians from National and European statistical offices While job profiles vary and are still in the process of being who are interested in the application of new technologies for defined, organizations agree that they need the new breed of production of the official statistics. workers. - Individual persons who are searching for new employment Accordingly, the European institutions take major initiatives opportunities. In particular, individuals are interested in the related to digitalization of labor market, training of new skills and job vacancies that are compatible with their current skills and meeting the labour demand. in the methods (like trainings) providing the possibilities to Historically, the labour market statisticians use standard measures obtain new skills in demand. of the labour demand and labour supply based on traditional - Public and private employment agencies interested in up-to- surveys – job vacancy surveys, wage survey, labour force surveys. date employees profiles. The unemployment rate provides information on the supply of - persons looking for work in excess of those who are currently Education and training institutions from different levels and employed. Data on employment provide information on the forms of education - general/vocational education, higher demand for workers that is already met by employers. education, public/private, initial/ adult education. Educational institutions are interested in relevant skills and The data-driven smart labour market statistics intends to: topics that should be part of the curriculum programs. - use the available historical job vacancies data, - Ministries of labour/manpower, economy/industry/trade, - use the available real-time job vacancies data, education, finance, etc. The policy makers, such as - use the available real-time and historical dataset of additional ministries, are interested in the overall labour market data (described below), situation, with respect to location and time, in the labour - align data sources, market segmentation and in the processes of improving - construct models and obtain novel smart labour market supply and productivity. indicators that will complement existing labour market - Standards development organizations. National or statistics, International organizations whose primary activities are - provide a system for delivering results to the users. developing, coordinating, promulgating, revising, amending, reissuing, interpreting, or otherwise producing technical The smart labour market statistics approach will combine standards that are intended to address the needs of some advanced data processing, modelling and visualization methods in order to develop trusted techniques for job vacancies analysis with 157 relatively wide base of affected adopters. Interested in new - Social media data, such as news, Twitter data that might be technologies developed in relation to labour market. relevant for labour market. - Academic and research institutes. Public and private entities - Labour supply data (based on user profile analysis). who conduct research in relevant areas. Research institutions Open job vacancies can be found using job search services. These are interested in the development of novel methodologies services aggregate job vacancies by location, sector, applicant and usage of appearing new data sources. qualifications and skill set or type. One such service is Adzuna [4], a search engine for job ads, which mostly covers English- 3. RELATED WORK speaking countries. The European Data Science Academy (EDSA) [1] was an H2020 For data acquisition and enrichment, dedicated APIs, including EU project that ran between February 2015 and January 2018. Adzuna API, are used, as well as custom web crawlers are The objective of the EDSA project was to deliver the learning developed. The data is formatted to JSON to aid further tools that are crucially needed to close the skill gap in Data processing and enrichment. The job vacancy dataset is obtained Science in the EU. The EDSA project has developed a virtuous with respect to trust and privacy regulations, the personal data is learning production cycle for Data Science, and has: not collected. - Analyzed the sector specific skillsets for data analysts across Job vacancies usually contain the information, such as job Europe with results reflected at EDSA demand and supply position title, job description, company and job location. In such dashboard; way, job vacations that are constantly crawled/web-scraped - Developed modular and adaptable curricula to meet these present a data stream. The job title and job description are textual data science needs; and data that contain information about skills that employee should - have. Delivered training supported by multiplatform resources, introducing Learning pathway mechanism that enables On the obtained data wikification - identifying and linking textual effective online training. components (including skills) to the corresponding Wikipedia EDSA project established a pipeline for job vacancy collecting pages [5] is performed. This is done using Wikifier [6], which and analysis that will be reused for the purpose of smart statistics. also supports cross and multi-linguality enabling extraction and annotation of relevant information from job vacancies in different An ontology called SARO (Skills and Recruitment Ontology) [2] languages. The data is tagged with concepts from GeoNames has been developed to capture important terms and relationships ontology [7]. To job postings where latitude and longitude have to facilitate the skills analysis. SARO ontology concepts included been available, GeoNames location uri and location name are relevant classes to job vacancy datasets, such as Skill and added. To the postings where only location name has been JobPosting. Examples of instances of class Skill would be skills, available, the coordinates and location uri are added. such as “Data analysis”, “Java programming language” et al. The job vacancy data representation level depends on the specific ESCO [3] is the multilingual classification of European Skills, country. For the United Kingdom, France, Germany and the Competences, Qualifications and Occupations. It identifies and Netherlands there is a substantial collection of job vacancies in categorizes skills/competences, qualifications and occupations the area of digital technologies. relevant for the EU labour market and education and training, in 25 European languages. The system provides occupational 4.2 CONCEPTUAL ARCHITECTURE profiles showing the relationships between occupations, The labour market statistics conceptual structure is built upon the skills/competences and qualifications. For instance, one example following major blocks: of existing ESCO skill is “JavaScript” (with alternative labels “Client-side JavaScript”, "JavaScript 1.7" et al.). 1. Data sources related to different aspects of smart labour market. The main data source aggregates historical and current job Both SARO and ESCO ontologies are useful for the aim of smart vacancies in the area of digital technologies and data science statistics, in particular for skills ontology development and skills around Europe. ontology evolution scenarios. However, the ontologies usually are manually manipulated, and the methods developed for smart 2. Modelling smart labour market statistics takes central part of labour market statistics should overcome the difficulties related to the smart labour market statistics approach, where the goal is to this issue. The ontology evolution scenario of smart labour market construct models based on different data sources, updated in statistics envisions automatic identification of emerging and business-real-time (as needed or as data sources allow). Models decreasing skills from the data perspective. shall bring understanding of the smart labour market statistics domain and shall be used for aggregation, ontology development and ontology evolution. 4. PROBLEM DEFINITION 3. Targeted users are smart statistics consumers. There are several 4.1 DATA SOURCES major groups of users (described above). The example users might include statisticians, policy makers, individual users (residents The main data sources available for the development of smart and non-residents), training and educational organizations and labour market statistics are historical and current data about job other. vacancies in the area of digital technologies and data science around Europe (~5.000.000 job vacancies 2015-2018). 4. Finally, applications of smart labour market statistics are multiscale - they can be presented at cross-country level (around Additional data sources may include: 158 Europe) country level (UK, France, the Netherlands etc.), relationships, and other distinctions that are relevant for modeling city/area level and conceptual level (ontology). a domain. The specification takes the form of the definitions of Figure 1 illustrates the conceptual architecture diagram for smart representational vocabulary (classes, relations, and so forth), labour market statistics. which provide meanings for the vocabulary and formal constraints on its coherent use. Figure 1: Conceptual Architecture The key characteristics of the development techniques will include: - Interpretability and transparency of the models – the aim is, for a model to be able to explain its decision in a human readable manner (vs. black box models, which provide results without explanation). - Non-stationary modelling techniques are required due to changing data and its statistical properties in time. For instance, the ontology evolution process will be modeled taking to the account the incremental data arriving to the system. - Multi-resolution nature of the models, having the property to observe the structure of a model on multiple levels of granularity, depending on the application needs. - Scalability for building models is required due to the nature of incoming data streams. 4.3 SCENARIOS The smart labour market statistics proposal includes three scenarios - demand analysis scenario, ontology development Figure 2: Example of Job Vacancies Crawled and scenario and ontology evolution scenario described below. Processed 4.3.1 DEMAND ANALYSIS Ontologies are often manually developed and maintained, what Demand analysis scenario suggests production of statistical requires a sufficient user efforts. indicators based on the available job vacancies using techniques In the ontology development scenario an automatic (or semi- for data preprocessing, semantic annotation, cross-linguality, automatic) bottom-up process of creating ontology from available location identification and aggregation. job vacancies will be suggested. Job vacancies in structural and semi-structural form are the input The relevant skills (extracted from the job vacancies) will be to into the system, while statistics related to overall job demand, defined and formalized. Using semantic annotation and cross- job demand with respect to particular location, job demand with linguality techniques for skills extraction based on JSI Wikifier respect to particular skill (skill demand) and time frame are the tool [6] will enable the possibility of including the newest outputs of the system. available skills “on the market” that are not yet captured in the Figure 2 presents an example of crawled and processed job ontologies, taxonomies and classifications that are manually vacancies. developed. The input to the ontology development scenario is a set of job vacancies and the output is ontology of skills presenting 4.3.2 SKILLS ONTOLOGY DEVELOPMENT the domain structure that can be compared to or used for official Ontologies reduce the amount of information overload in the classifications. working process by encoding the structure of a specific domain and offering easier access to the information for the users. Gruber 4.3.3 SKILLS ONTOLOGY EVOLUTION [8] states that an ontology defines (specifies) the concepts, Ontology Evolution is the timely adaptation of an ontology to the arisen changes and the consistent propagation of these 159 changes to dependent artefacts [9]. Ontology evolution is a - Ontology evolution statistics. Example: emerging skills in process that combines a set of technical and managerial activities the ontology in the last 3 months and ensures that the ontology continues to meet organizational Since the data has a streaming nature, different kinds of multiscale objectives and users’ needs in an efficient and effective way. and aggregation options can be handled with respect to time Ontology management is the whole set of methods and techniques parameters. that is necessary to efficiently use multiple variants of ontologies from possibly different sources for different tasks [10]. Scenario 3 will suggest an automatic (or semi-automatic) ontology 6. CONCLUSION AND FUTURE WORK evolution process based on the real-time job vacancy stream. With In this paper, we presented a proposal for developing smart labour respect to the nature of job vacancy data stream and skills market statistics based on streams of enriched textual data, such as extracted from job it will be possible to see the dynamics of job vacancies from European countries. We define smart statistics evolving skills – when the new skills (not included into the scenarios, such as demand analysis scenario, skills ontology current ontology versions appear) and how the skills ontology is development scenario and skills ontology evolution scenario. The changing with time. future work would include the implementation of the smart labour In particular, it could be possible to observe appearing new skills market scenarios, quality assessment and evaluation of the and suggest them for inclusion into official skills classifications. produced statistical outcomes. In addition, it could be visible how fast the ontology changes, which could be the indicator of the technological progress on the 7. ACKNOWLEDGMENTS relevant market. This work was supported by the Slovenian Research For instance, the current version of ESCO classification does not Agency and EDSA European Union Horizon 2020 project contain “TensorFlow” skill (TensorFlow [11] is an open-source under grant agreement No 64393. software library for dataflow programming across a range of tasks, appeared in 2015). TensorFlow, which is already present in job vacancies, could be captured during ontology evolution process 8. REFERENCES and suggested as a new concept for official classifications. [1] EDSA, http://edsa-project.eu (accessed in August, 2018). [2] Sibarani, Elisa & Scerri, Simon & Mousavi, Najmeh & Auer, Sören. (2016). Ontology-based Skills Demand and Trend 5. STATISTICAL INDICATORS Analysis. 10.13140/RG.2.1.3452.8249. Traditionally the indicators related to labour market have been [3] ESCO taxonomy, https://ec.europa.eu/esco/portal (accessed in based on survey responses. The smart labour market statistics August 2018). proposal introduces a possibility to complement standard statistical indicators, such as job vacancy rate with novel “data [4] Adzuna developer page, inspired” knowledge. https://developer.adzuna.com/overview (accessed in August, 2018). The smart labour market statistics indicators use data sources, previously not covered by official statistics, and in such way [5] Ratinov, L., Roth, D., Downey, D. and Anderson, M. Local complementary to traditional data sources. The smart labour and global algorithms for disambiguation to wikipedia. In market statistics indicators are based on real-time data streams, Proceedings of the 49th Annual Meeting of the Association for which makes possible to obtain not only historical, but also Computational Linguistics: Human Language Technologies- current values for job vacancies that could be used for different Volume 1, pages 1375–1384. Association for Computational purposes, such as nowcasting. In addition, the smart labour Linguistics, 2011. market statistics indicators take into the account data cross-lingual [6] JSI Wikifier, http://wikifier.org (accessed in May, 2018). and multi-lingual nature of streaming data and can be produced at the multiscale levels – cross-country, country, city (area) levels. [7] GeoNames ontology, http://www.geonames.org/ontology/documentation.html (accessed The scenarios described above would result into a number of in August, 2018). smart labour market indicators with multiscale options. In particular: [8] Ontology (by Tom Gruber), - http://tomgruber.org/writing/ontology-definition-2007.htm Up-to date job vacancies statistics on a cross- (accessed in August, 2018). country/country/city(area) level. Example: job vacancies in UK and France in the last month [9] M. Klein and D. Fensel, Ontology versioning for the Semantic - Web, Proc. International Semantic Web Working Symposium Up-to date skills statistics on a cross- (SWWS), USA, 2001 country/country/city(area) level. Example: top 10 skills in UK in the last month [10] L. Stojanovic, B. Motik, Ontology evolution with ontology, - in: EKAW02 Workshop on Evaluation of Ontology-based Tools Up-to date location statistics. Example: top locations for (EON2002), CEUR Workshop Proceedings, Sigüenza, vol. 62, specific skill 2002, pp. 53–62 - Ontology development statistics. Example: number of [11] TensorFlow, https://en.wikipedia.org/wiki/TensorFlow concepts in the ontology (accessed in August, 2018). 160 Relation Tracker - tracking the main entities and their relations through time M. Besher Massri Inna Novalija Marko Grobelnik Jožef Stefan Institute Jožef Stefan Institute Jožef Stefan Institute Jamova cesta 39, Ljubljana, Slovenia Jamova cesta 39, Ljubljana, Slovenia Jamova cesta 39, Ljubljana, Slovenia besher.massri@ijs.si inna.koval@ijs.si marko.grobelnik@ijs.si ABSTRACT contextual information provided as characteristic keywords, for a In this paper, we present Relation Tracker, a tool that tracks main quick detection of information from the original articles. entities [people and organizations] within each topic through time. The main types of relations between the entities are detected Regarding classifying news, we observe in [3] a new technique and observed in time. The tool provides multiple ways of that uses Deep Learning to increase the accuracy of prediction of visualizing this information with different scales and durations. online news popularity. The tool uses events data from Event Registry as a source of In the paper explaining Event Registry [1], we see how articles information, with the aim of getting holistic insights about the from different languages are grouped into events and the main searched topic. information and characteristics about them are extracted. Additionally, a graphical interface is implemented which allows search for events and visualize the results in multiple ways that KEYWORDS together give a holistic view about events. Information Retrieval, Visualization, Event Registry, Wikifier, Dmoz Taxonomy This work begins with the events as a starting point, and it is one more step on the same path; it groups events further into topics 1. INTRODUCTION and trends, then it focuses on tracking how some entities are Every day, tremendous amounts of news and information are appearing as main entities regarding the selected topic, and how being streamed throughout the Internet, which is requiring the the relationship between them is changing through time. implementation of more tools to aggregate this information. With technology advancement, those tools have been increasing in complexity and options provided. However, there has been a 3. DESCRIPTION OF DATA demand for tools that give simple yet holistic summary of the We used part of the events from Event Registry as our main searched topic in order to acquire general insights about it. source of data. We obtained a dataset of ~ 1.8 million events as a list of JSON files, with event’s dates between Jan 2015 and July Hence, we provide the Relation Tracker tool that tries to achieve 2016. Each event consists of general information like title, event this goal; it is based on the data from Event Registry [1], which is date, total article count, etc., and a list of concepts that a system for real-time collection, annotation and analysis of characterize the event, which is split into entity concepts and non- content published by global news outlets. The tool presented in entity concepts. Entity concepts are people, organizations, and this paper takes the events and groups them into topics, and locations related to the event. Whereas non-entity concepts within each topic, it provides an interactive graph that shows the represent abstract terms that define the topic of the event, like main entities of each topic at each time and the main topic of technology, education, and investment. Those concepts were relations between those entities. In addition, a summary extracted using JSI Wikifier [4] which is a service that enables information about entities and their relationship is visualized semantic annotation of the textual data in different languages. In through different graphs to help understand more about the topic. addition, each concept has a score that represents the relevancy of that concept to the event. The remainder of this paper is structured as follows. In section 2, we show the related work done in this area. In section 3, we provide a description of the used data. Section 4 explains the 4. METHODOLOGY methodology and main challenges that were involved in this work. Next, we explain the visualization features of the tool in section 5. Finally, we conclude the paper and discuss potential future work. 4.1 Clustering and Formatting Data To group the events into topics, we used K-Means clustering 2. RELATED WORK algorithm, where each event is represented as a sparse vector of the non-entity concepts it has, with the weights equal to their Similar works have been done in the area of visualizing scores in that event. The constant number of topics is set information extracted from news. We see in [2] a tool for efficient experimentally to be 100 clusters, in a balance between mixed visualization of large amount of articles as a graph of connected clusters and repeated clusters. Each cluster describes a set of entities extracted from articles, enriched with additional events that fall under the same topic, whereas the centroid vector of each cluster represents the main characteristics of it. To name 161 the clusters, we used category classifier service from Event 4.3 Detecting the Characteristics of Registry, which uses Dmoz Taxonomy [5], a multilingual open- content directory of World Wide Web links, that is used to Relationship classify texts and webpages into different categories; for each The main goal was to model the relationship between any two cluster, we formed a text consisting of the components of its entities through a vector of words where two entities are centroid vector, taking into account their weights within the collocated. Since the relationship between two entities at any vector. The resulted cluster names were ranged from technology given time is based on the shared events between them, and each and business to refugees and society, and clusters were exported event is characterized by a set of concepts, we decided on using as a JSON file for processing them in the visualization part. those concepts - specifically the abstract or the non-entity concepts - to characterize such relationships. For each pair, we 4.2 Choosing the Main Entities aggregated all the non-entity concepts from the shared events between them, and each one of them was assigned a value based Under any topic, the top entities at each duration of time has to be on the number of events it is mentioned in and its score in those chosen. At first, the concepts were filtered from outliers like events. Those concepts were sorted and ranked depending on their publishers and news agencies. Then, an initial importance value values, and the top ones were chosen as the main features of the has been set for each concept based on two parameters: the TF- relationship. In addition, these values of the concepts were used to IDF score of concept with respect to each event, and the number rank the shared events and extract the top ones; by giving each of articles each event contains. If we denote the set of events that event a value equal to the aggregated values (the ones calculated occur in the interval of time D by ED, the number of articles that in previous step) of all non-entity concepts it has. To summarize event e contains is Ae, the TF-IDF score of concept c at event e by the set of characteristics, we classified them using Dmoz category Sc,e, then the importance value of each item with respect to the classifier in a similar way to what we have done in determining interval D is calculated by the formula: the names of the clusters. These categories were used to label the relationship between the entities, indicating the main topic of the shared events between them. 5. VISUALIZING THE RESULTS To access a topic, a search bar is provided to select among the list The TF-IDF function is used to give importance to the concept of extracted topics from clustering step. Once the user selects a based on its relevance to the events, and the number of articles is topic, a default date is chosen and a network graph is shown used to give more importance to the events that have more articles explaining the topic. talking about it, and hence, more importance to the concepts that it has. We decided on using the product of summation rather than 5.1 Characteristics of the Main Graph summation of product because of its computation efficiency while Since the tool’s main goal is to show the top entities and their still producing good results. However, to prevent the case where relations, the network graph is the best choice for this matter. all the chosen entities get nominated because of one or two big Following that, we have built an interactive network graph that events (which results in a bias towards those few events), a has the following features: modification to the importance value formula has been made by - The main entities within that topic at the selected interval introducing another parameter, which is the links between of time are represented by the vertices of the graph. concepts (whenever two concepts occur in the same event, there is - The size of the vertices reflects the importance value of a link between them). Each concept now affects negatively the each entity, scaled to a suitable ratio to fit in the canvas. other concepts it is linked to by an amount equal to the initial - The colors represent the type of the entity, whether it is a importance value divided by the number of neighbors. If we person [red] or an organization [blue]. denote the set of neighbors of concept c during the interval of - The links between the entities represent the existence of time D by Nc,D, then the negative importance value is defined by: shared events in that interval of time between them under that topic, and hence indicating some form of relations. The thickness of the links is proportional to the number of shared events, whereas the labels are the ones calculated in previous section. Figure 1 presents top companies with relevant relations in July The final score is just the initial importance value minus the 2015 found among business news. negative importance value, which is then used to sort and nominate the top entities. 162 Figure 1: Top companies in July 2015 and their relations Figure 3: The changes in top entities under the same topic under the business topic. after moving the interval for 15 days. 5.2 Main Functionality 5.3 Displaying Relation Information As the tool is concerned about tracking the changes with time. Whenever the user selects a pair of entities, detailed information The graph is supported with a slide bar that allows the user to about their relationship in the selected interval of time is given, choose from the dates where there is at least one event occurred such as the number of shared events and articles, along with the with respect to the selected topic. Different scales for moving top events both concepts were mentioned in. Also, the top shared dates are also provided; the user can choose to move day by day, characteristics that shape the relationship between them at this week by week, or month by month and see the changes period is shown and sorted by percentage of importance. As seen accordingly. In addition, the user can choose a specific interval of in Figure 4; when selecting Jeff Bezos and Elon Musk under the time, and track how the entities and their relations are changing space topic between January and September 2015, we see a list of when the interval moves slightly with respect to its length. An the top events that involve both of them during this period. We interval magnifier is also given if the user wants to get a closer see also that the relationship between them is mainly about look at the changes that happen in a small interval. sending astronauts by rockets to the international space station, as it can be understood from the top shared characteristics. An example illustrating that can be seen in Figures 2 and 3. In Figure 2, we see the top 10 entities under the refugee topic in the last two months of 2015. When the interval is moved by 15 days, we notice that some of the entities disappear, like European Commission, indicating that they are no longer among the top 10 entities, whereas “United States House of Representitive” entity emerges and connects to “Barack Obama” and “Repulican Party”. The change in size indicates the change in the importance value of each one, while Society is the general theme among all labels. Figure 4: Relationship summary about Jeff Bezos and Elon Musk between January and September 2015 under the Space Figure 2: Top entities for the last two months of 2015 under topic. the refugee topic. 163 To illustrate how the importance of those top features with respect 6. CONCLUSION AND FUTURE WORK to the relationship is changing through time, a stream graph is In this paper, we provide a tool that uses events data from Event used as shown in Figure 5. Registry to show the main entities within each topic, and how the characteristics of relationship among them is changing through time. However, there are a couple of limitation to the tool that we want to improve in the future. Although we were able to detect the characterestics of the relationship between entities and how they are changing through time, the main type of relation that we used to label the links were very broad and hence rarely changing- improving the methodology for relation extraction and observation of relations in time will be the subject of future work. In addition, we limited the search space for topics for the 100 topics we obtained from clustering, we would like to generalize the search by enabling searches for any concept or keyword with different options to filter the search. 7. ACKNOWLEDGMENTS Figure 5: Stream graph showing how the effect of the main features on the relationship between Jeff Bezos and Elon Musk This work was supported by the euBusinessGraph (ICT-732003- is changing through time. IA) project [6]. 8. REFERENCES Finally, the set of all characteristics that affect the relationship is visualized in a tag cloud to give a big picture about it. Figure 6 shows the tag cloud of the same relationship mentioned above. [1] Gregor Leban, Blaz Fortuna, Janez Brank, and Marko Grobelnik. 2014. Event registry: learning about world events from news. In Proceedings of the 23rd International Conference on World Wide Web (WWW '14 Companion). ACM, New York, NY, USA, 107-110. DOI: https://doi.org/10.1145/2567948.2577024 [2] Marko Grobelnik and Dunja Mladenić. 2004. Visualization of news articles. Informatica 28. [3] Sandeep Kaur and Navdeep Kaur Khiva. 2016. Online news classification using Deep Learning Technique. IRJET 03/10 (Oct 2016). [4] Janez Brank, Gregor Leban and Marko Grobelnik. 2017. Annotating documents with relevant Wikipedia concepts. In Figure 6: Tag cloud illustrating a general view about all the Proceedings of siKDD2017. Ljubljana, Slovenia. characteristics that affects the relationship between Jeff Bezos and Elon Musk under the space topic. [5] Dmoz, open directory project, http://dmoz-odp.org/ (accessed in July, 2018) [6] euBusinessGraph project, http://eubusinessgraph.eu/ (accessed in July, 2018). 164 Cross-lingual categorization of news articles Blaž Novak Jožef Stefan Institute Jamova 39 Ljubljana, Slovenia +386 1 477 3778 blaz.novak@ijs.si ABSTRACT categories. We consider each document belonging to all In this paper we describe the experiments and their results categories that are explicitly stated, and all of their parents. We performed with the purpose of creating a model for automatic will compare the performance of model predictions on the same categorization of news articles into the IPTC taxonomy. We show language and in the cross-lingual setting, where we train the that cross-lingual categorization is possible using no training data model on the entire dataset available for one language, and from the target language. We find that both logistic regression and measure its performance on the other language. support vector machines are good candidate models, while Basic features of the dataset can be seen in the following 2 random forests do not perform acceptably. Furthermore, we show figures. Figure 1 shows the distribution of number of articles in that using Wikipedia-derived annotations provides more each category, and Figure 2 shows that most categories contain a information about the target class than using generic word roughly even number of articles in both languages, but there are features. some outliers. We ignored categories with less than 15 examples per language, which resulted in 308 categories. General Terms Algorithms, Experimentation Keywords News, articles, categorization, IPTC, Wikifier, SVM, Logistic regression, Random forests. 1. INTRODUCTION The JSI Newsfeed [1] system ingests and processes approximately 350.000 news articles published daily around the world, in over 100 languages. The articles are automatically cleaned up and semantically annotated, and finally stored and made available for downstream consumers. One of the annotation tasks that we would like to perform in the future is to automatically categorize articles into the IPTC “Media Topics” subject taxonomy [2]. IPTC – the International Press Figure 1. Number of articles in each category. Discrete Telecommunications Council – provides a standardized taxonomy categories on x axis are ordered by descending number of articles. of roughly 1100 terms, arranged into a 5 level taxonomy, describing subject matters relating to daily news. The vocabulary is accessible in a machine readable format – RDF/XML and RDF/Turtle – at http://cv.iptc.org/newscodes/mediatopic. There are two relations linking concepts in the vocabulary – the ‘broader concept’ taxonomical relation, and a ‘related concept’ sibling relation. The ‘related concept’ links concepts both to other concepts from the same taxonomy, and directly to external Wikidata [3] entities. The purpose of this work is to evaluate multiple machine learning algorithms and multiple sets of features with which we could automatically perform the categorization. As we would like to categorize articles in all the languages the Newsfeed system supports, but we only have example articles in English and Figure 2. Language imbalance for each category. Discrete French, the method needs to be language independent. categories on x axis are ordered from “mostly English” to “mostly French”. 2. EXPERIMENTAL SETUP We compare three different machine learning models – random The dataset that we have access consists of 30364 English and forests, logistic regression (LR), and Support Vector Machines 29440 French articles, each of which is tagged with 1 to 10 (SVM). 165 We try two different types of features, and their combinations. significantly worse. “Wiki-W” denotes the weighted version of Wikifier annotations, and “Wiki-K” the combination of KCCA- The first kind of a feature set we use is a projection of the bag-of- derived features and Wikifier annotations. Every second line in words representation of the document text into a 500 dimensional vector space. The KCCA [4] method uses an aligned multi-lingual the table is the standard deviation of the result when averaged corpus to find such a mapping, that words with similar meanings across all categories. map to a similar vector, regardless of their language. We represent a document as a sum of all word vectors. Table 1. ROC scores by model and feature type, cross- The second set of features we use is the output of the JSI Wikifier validation [5] system. The Wikifier links each word in a document to a set of Rand. Forest Log. Reg. SVM Wikipedia pages that might represent the meaning of that word. For each such annotation, we also get a confidence weight. EN FR EN FR EN FR We consider these annotations as a classical vector space model -- KCCA 0.75 0.71 0.96 0.95 0.95 0.94 as a bag-of-entities. We use two versions of the TF-IDF [7] (stdev) 0.11 0.11 0.04 0.04 0.05 0.04 scheme: in the first case, we use the number of times an entity Wiki 0.70 0.70 0.95 0.95 0.94 0.94 annotation is present for any word in a document as the TF (term (stdev) 0.12 0.12 0.04 0.04 0.05 0.04 frequency) factor, and in the second version, we use the sum of annotation weights of an entity across the document. In both Wiki-W 0.71 0.71 0.95 0.95 0.94 0.94 cases, we perform L1 normalization of the vector containing TF (stdev) 0.12 0.11 0.04 0.04 0.05 0.04 terms. For IDF terms, we use log �1 + 𝑁𝑁� where 𝑁𝑁 is the number 𝑛𝑛 Wiki+K 0.71 0.69 0.97 0.96 0.96 0.95 of all documents and 𝑛𝑛 the number of documents where an (stdev) 0.12 0.11 0.03 0.03 0.03 0.04 annotation was present at least once. Finally, we use a combination of both KCCA-derived and Wikifier-derived features as the last feature set option. Looking at the feature selections, we see almost no significant difference -- both kinds of features -- KCCA and Wikipedia For model training, we use Pythons scikit-learn [6] software annotations have useful predictive value. The combination of both package. In the case of logistic regression, we use L2 penalty, feature types slightly improves the ROC score. with automatic decision threshold fitting, using the liblinear library backend. Table 2 shows F1 cross-validation scores of all three models. Logistic regression scores much higher than SVM here, possibly For the SVM model, we use a stochastic gradient descent indicating that the SVM model would benefit from a post- optimizer. We performed a grid search for the optimal processing step of optimizing the decision threshold on a separate regularization constant 𝐶𝐶, but since there were no significant training set. accuracy changes, we used the default of 1.0 in all other experiments. Table 2. F1 scores by model and feature type, cross-validation For the random forest model, we used 4 different parameter Rand. Forest Log. Reg. SVM combinations: EN FR EN FR EN FR • default – 10 trees, splitting until only one class is in the KCCA 0.16 0.12 0.30 0.25 0.20 0.18 leaf (stdev) 0.21 0.18 0.21 0.20 0.21 0.19 • 30 trees, maximum tree depth of 10 Wiki 0.07 0.07 0.41 0.44 0.25 0.29 • 50 trees, maximum tree depth of 10 (stdev) 0.15 0.15 0.21 0.21 0.22 0.22 • 30 trees, maximu tree depth of 20 Wiki-W 0.08 0.08 0.40 0.43 0.24 0.28 In all cases, GINI index was used as the node splitting criterion. (stdev) 0.17 0.17 0.21 0.21 0.21 0.22 Since the majority of categories only have a small number of Wiki+K 0.09 0.07 0.44 0.46 0.27 0.30 documents, we automatically weighed training examples by the (stdev) 0.16 0.15 0.21 0.21 0.22 0.22 inverse of their class frequency. We also performed some experiments without this weighting scheme, but got useless models in all cases except for the couple largest categories. The combination of both feature sets performs significantly better than either alone, with generic word-based features providing the All reported results are the average of a 3-fold cross-validation. least amount of information. So far, we only created one-versus-all models for each category The feature usefulness changes when looking at cross-lingual independently, and only used the taxonomy information of classification performance. Table 3 shows the ROC score for all categories to select all examples from sub-categories when three models, when the model trained on English is used to predict training the more general category. categories of French articles, and vice versa. Decision trees give essentially a random result, and SVM scores somewhat higher than logistic regression. 3. RESULTS Table 1 shows ROC scores for cross-validation of all three models Table 3. ROC scores - cross-lingual classification on four sets of feature combinations, for English and French Rand. Forest Log. Reg. SVM separately. SVM and logistic regression are comparable in EN FR EN FR EN FR behavior and promising, while the random forest model performs 166 KCCA 0.50 0.50 0.50 0.50 0.50 0.51 (stdev) 0.00 0.00 0.01 0.03 0.04 0.08 Wiki 0.51 0.51 0.76 0.80 0.81 0.84 (stdev) 0.04 0.04 0.12 0.11 0.11 0.10 Wiki-W 0.51 0.52 0.78 0.82 0.82 0.84 (stdev) 0.04 0.05 0.11 0.10 0.10 0.10 Wiki+K 0.50 0.50 0.57 0.70 0.66 0.81 (stdev) 0.01 0.01 0.10 0.13 0.14 0.12 The biggest change here is the influence of KCCA cross-lingual word embedding: by itself it provides no informative value, as indicated by ROC value of 0.5 in all cases, and it even reduces the performance of the combined Wikifier + KCCA model. Figure 3. F1 score correlation for logistic regression In the Table 4, F1 scores from the same experiment are shown. Logistic regression still has a big advantage over SVM, as in the same-language categorization setting. The change from previous experiments is the influence of weighting of Wikipedia features -- it increases the performance of all models. Table 4. F1 scores - cross-lingual classification Rand. Forest Log. Reg. SVM EN FR EN FR EN FR KCCA 0.00 0.00 0.00 0.01 0.00 0.02 0.02 0.02 0.02 0.06 0.01 0.06 Wiki 0.03 0.04 0.48 0.44 0.30 0.26 0.10 0.11 0.21 0.20 0.22 0.22 Wiki-W 0.03 0.05 0.49 0.44 0.29 0.26 Figure 4. F1 score correlation for SVM 0.11 0.13 0.20 0.21 0.22 0.22 Wiki+K 0.00 0.00 0.18 0.40 0.20 0.23 0.04 0.04 0.22 0.22 0.19 0.21 An interesting observation is that the performance of the cross- lingual model is occasionally higher than that of the baseline cross-validation experiment. This anomaly however disappears for categories with large amount of positive training examples. It also disappears if we reduce the amount of training examples in the cross-lingual experiment by 1/3 – the effect seems to be caused by cross-validation reducing the training dataset size. KCCA cross-lingual word embedding feature generation used here was tested in other experiments and systems and gives a useful feature set for comparison of documents across languages, so its negative impact on the performance of these models needs Figure 5. ROC score correlation for logistic regression to be investigated in the future. As the weighted Wikipedia feature set appears to be the best for the stated goal of cross-lingual article categorization, the results of next experiments are shown only for it, but we performed the same experiments on all other combinations, and the results broadly follow the conclusions from the previous section. The following figures show correlation of testing and cross- lingual performance of logistic regression and SVM models. Both F1 score and area under ROC curve are shown for each of 308 categories in the experiment, since they provide complementary information. As the figures show, there is a good agreement between the cross-validation and the cross-lingual classification performance, giving us an ability to estimate cross-lingual performance based on the cross-validation score in the production environment. The difference between distributions for French and Figure 6. ROC score correlation for SVM English language models is consistent with the class imbalance for each of the categories. 167 The SVM model seems to have a more consistent behavior, so we will use it in the final application instead of logistic regression. Figures 7 through 10 show the F1 and ROC score behavior of logistic regression and SVM models for cross-validation and cross-lingual classification with regard to the number of positive examples in the category, separately for English and French language. While the SVM model underperforms on the F1 metric on average, it produces a better ranking of documents with respect to a category, as seen on ROC plots, especially for smaller categories. This further indicates the need for decision threshold tuning in the SVM model before we use its predictions. Figure 10 ROC score with respect to category size, cross- lingual prediction As expected, classification performance of all models improves with the number of training examples, but in cases of small categories, it appears that some are much easier to learn than others. 4. CONCLUSIONS AND FUTURE WORK We found that using a logistic regression model with weighted Wikifier annotations gives us a good enough result to use IPTC category tags as inputs for further machine processing in the Figure 7. F1 score with respect to category size, cross- Newsfeed pipeline. Before we can use this categorization for validation human consumption, we need to investigate automatic tuning of SVM decision thresholds on this problem, and add an additional filtering layer that takes into consideration interactions between categories beyond the sub/super-class relation. Additionally, the negative effect of KCCA-derived features for cross-lingual annotation needs to be examined. 5. ACKNOWLEDGEMENTS This work was supported by the Slovenian Research Agency as well as the euBusinessGraph (ICT-732003-IA) and EW-Shopp (ICT-732590-IA) projects. 6. REFERENCES [1] Trampuš M., Novak B., “The Internals Of An Aggregated Web News Feed” Proceedings of 15th Multiconference on Information Society 2012 (IS-2012). Figure 8. ROC score with respect to category size, cross- [2] https://iptc.org/standards/media-topics/ validation [3] https://www.wikidata.org/wiki/Wikidata:Main_Page [4] Rupnik, J., Muhič, A., Škraba, P. “Cross-lingual document retrieval through hub languages”. NIPS 2012, Neural Information Processing Systems Worshop, 2012 [5] Brank J., Leban G. and Grobelnik M. “Semantic Annotation of Documents Based on Wikipedia Concepts”. Informatica, 42(1): 2018. [6] Pedregosa, F., Varoquaux, G., Gramfort, A. et al. “Scikit- learn: Machine Learning in Python”. Journal of Machine Learning Research, 12. 2011, pp. 2825-2830. [7] K. Sparck Jones. "A statistical interpretation of term specificity and its application in retrieval". Journal of Documentation, 28 (1). 1972 Figure 9. F1 score with respect to category size, cross-lingual prediction 168 Transporation mode detection using random forest Jasna Urbančič Veljko Pejović Dunja Mladenić Artificial Intelligence Faculty of Computer and Artificial Intelligence Laboratory, Information science, Laboratory, Jožef Stefan Institute University of Ljubljana Jožef Stefan Institute Jamova 39, 1000 Ljubljana, Večna pot 113, 1000 Ljubljana Jamova 39, 1000 Ljubljana, Slovenia Slovenia Slovenia jasna.urbancic@ijs.si veljko.pejovic@fri.uni-lj.si dunja.mladenic@ijs.si ABSTRACT While the first attempts to recognize user activity were ini- This paper addresses transportation mode detection for a tiated before smart phones, the real effort in that direc- mobile phone user using machine learning and based on mo- tion begun with the development of mobile phones having bile phone sensor data. We describe our approach to data built-in sensors [10], including GPS and accelerometer sen- collection, preprocessing and feature extraction. We eval- sors. There are still some studies that use custom loggers uate our approach using random forest classification with to collect the data [11, 17] or use dedicated devices as well focus on feature selection. We show that with feature selec- as smart phones [5]. Although GSM triangulation and local tion we can significantly improve classification scores. area wireless technology (Wi-Fi) can be employed for the purpose of transportation mode detection, their accuracy is 1. INTRODUCTION relatively low compared to GPS [11], so latest state of the art In the recent years we have witnessed a drastic increase in research is focused on transportation mode detection based sensing and computational resources that are built in mo- on GPS tracks and/or accelerometer data. bile phones. Most of modern cell phones are equipped with a Machine learning approaches for transportation mode detec- set of sensors containing triaxial accelerometer, magnetome- tion often rely on statistical, time-based, frequency-based, ter, and gyroscope, in addition to having a Global Position- peak-based and segment-based [8] features, however in most ing System (GPS). Smart phone operating system APIs of- cases statistical features and features based in frequency are fer activity detection modules that can distinguish between used [10, 11, 16]. Feature domains are described in Table different human activities, for example: being still, walk- 1. Statistical, time-based, and spectral attributes are com- ing, running, cycling or driving in a vehicle [2, 3]. However, puted on a level of a time frame that usually covers a few sec- APIs cannot distinguish between driving in different kind onds, whereas peak-based features are calculated from peaks of vehicles, for example driving a car or traveling by bus or in acceleration or deceleration. On the other hand, segment- by train. Recognizing different kind of transportation, also based features are computed on the recordings of the whole known as transportation mode detection, is crucial for mo- trip, which means that they cover much larger scale. Statis- bility studies, for routing purposes in urban areas where pub- tical, time-based, and spectral features are able to capture lic transportation is often available, for facilitating the users the characteristics of high-frequency motion caused by user’s to move towards more environmentally sustainable forms of physical movement, vehicle’s engine and contact between transportation [1], or to inspire them to exercise more. wheels and surface. Peak-based features capture movement In this paper we discuss the use of random forest in trans- with lower frequencies, such as acceleration and breaking portation mode detection based on accelerometer signal. We periods, which are essential for distinguishing different mo- focus on torized modalities. Additionally, segment-based features de- 1. feature extraction, and scribe patterns of such acceleration and deceleration periods [8]. 2. feature analysis to determine the most meaningful fea- tures for this specific problem and the choice of classi- Machine learning methods that are most commonly used fier. in accelerometer based modality detection include support vector machines, decision trees and random forests, how- Our main contribution is feature analysis, which revealed ever naive Bayes, Bayesian networks and neural networks the impact of each feature to the classification scores. have been used as well [11, 12]. Often these classifiers are 2. RELATED WORK used in an ensemble [16]. The majority of algorithms addi- tionally use Adaptive Boosting or Hidden Markov Model to improve the performance of the methods mentioned above [16, 8, 11, 10]. In the last years, deep learning has also been used [6, 14]. Accelerometer-only approach where only statistical features have been used reported 99.8% classification accuracy, how- ever users were instructed to keep the devices fixed position during a trip. Furthermore, only 0.7% of data was labeled as train [11]. State of the art approach to accelerometer-only 169 Domain Description (1) Data (1a) Mobile Statistical These features are include mean, standard de- acquisition applications viation, variance, median, minimum, maximum, range, interquartile range, skewness, kurtosis, root (2b) mean square. (2) Pre- (2a) (2b) Gravity Resampling Filtering Time processing Time-based features include integral and double estimation integral of signal over time, which corresponds to speed gained and distance traveled during that (3) Feature recording. Other time-based features are for ex- extraction ample auto-correlation, zero crossings and mean crossings rate. (4a) (4b) Frequency Frequency-based features include spectral energy, (4) Feature Correlation Statistical spectral entropy, spectrum peak position, wavelet analysis analysis analysis entropy and wavelet coefficients. These can be computed on whole spectrum or only on spe- (5) Clas- (5a) (5b) cific parts, for example spectral energy bellow Defining Choosing sification 50Hz. Spectrum is usually computed using fast feature sets classifiers Fourier transform, whereas wavelet is a result of the Wavelet transformation. Entropy measures are Figure 1: Detailed work flow diagram of the based on the information entropy of the spectrum proposed approach. We stacked general, high- or wavelet [7]. level tasks common in other approaches vertically, Peak Peak-based features use horizontal acceleration whereas subtasks specific to our approach are pic- projection to characterize acceleration and decel- tured horizontally. eration periods. These features include volume, intensity, length, skewness and kurtosis. Split the signal Convolute with Segment Segment-based include peak frequency, stationary Signal on acceleration Find peaks a box window and deceleration duration, variance of peak features, and station- ary frequency. The latter two are similar to ve- locity change rate and stopping rate used by [17]. Count or compute Segment-based features are computed on a larger scale than statistical, time-based or frequency- Number of peaks based features. Table 1: Feature domains and their descriptions Mean Peak height adopted from [8]. Peak-based Standard Peak width features deviation transportation mode detection relies on long accelerometer Skewness Peak width height samples. It uses features from all five domains for classifica- tion with AdaBoost with decision trees as a weak classifier Peak area and achieves 80.1% precision and 82.1% recall [8]. Figure 2: Work flows for extraction of peak-based The performance of transportation mode detection systems features. depends on the effectiveness of handcrafted features designed by the researchers, researcher’s experience in the field af- We collect five second samples of sensor data and resam- fects the results. Thus, there have been approaches that use ple them to sampling frequency 100 Hz in the preprocessing deep learning methods, such as autoencoder or convolutional phase. Resampling ensures us that our samples all contain neural network, to learn the features used for classification. 500 measurements. The most prominent problem we face in Using a combination of handcrafted and deep features for preprocessing concerns the correlation of acceleration mea- classification with deep neural network resulted in 74.1% surements with the orientation of the phone in the three classification accuracy [15]. dimensional space. Practically this means that gravity is measured together with the dynamic acceleration caused by 3. PROPOSED APPROACH phone movements. To eliminate gravity from the samples we perform gravity estimation on raw accelerometer signal. Work flow of the proposed approach is visualized in Figure We follow an approach proposed by Mizell [9]. Gravity es- 1. The first task is data collection. To collect data we use timation splits the acceleration to static and dynamic com- NextPin mobile library [4] developed by the Artificial In- ponent. Static component represents the constant accelera- telligence Laboratory at Jožef Stefan Institute. Library is tion, caused by the natural force of gravity, whereas dynamic embedded into two free mobile applications. The first one is component is a result of user’s motion. Furthermore, using OPTIMUM Intelligent Mobility [1]. OPTIMUM Intelligent this approach we are able to calculate vertical and horizontal Mobility is a multimodal routing application for three Eu- components of acceleration. ropean cities — Birmingham, Ljubljana, and Vienna. The second one is Mobility patterns [4]. Mobility patterns is es- We use preprocessed signal to extract features for classifica- sentially a travel journal. Both applications send five second tion. Features are divided into five domains, based on their long accelerometer samples every time OS’s native activity meaning and method of computation. We have listed the do- recognition modules, Google’s ActivityRecognition API [2] mains in Table 1. We extract features from three domains — for Android and Apple’s CMMotionActivity API [3], de- statistical, frequency, and peak. We extract statistical fea- tect that the user is traveling in a vehicle. We use that tures (maximal absolute value, mean, standard deviation, accelerometer samples for fine-grained classification of mo- skewness, 5th percentile, and 95th percentile) from dynamic torized means of transportation. acceleration and its amplitude, horizontal acceleration and 170 Set Accele. Features Size Feature set CA RE PR F1 D-S Dynamic Statistical 54 D-S 0.48 0.41 0.39 0.37 D-SF Dynamic Statistical, Frequency 94 D-SF 0.60 0.41 0.41 0.39 D-SFP Dynamic Statistical, Frequency, Peak 172 D-SFP 0.46 0.39 0.40 0.35 H-S Horizontal Statistical 54 H-S 0.64 0.40 0.43 0.41 H-SF Horizontal Statistical, Frequency 94 H-SF 0.53 0.39 0.43 0.36 H-SFP Horizontal Statistical, Frequency, Peak 172 H-SFP 0.50 0.37 0.40 0.34 ALL 376 ALL 0.47 0.35 0.40 0.33 Table 2: Predefined feature sets used for classifica- Table 3: Classification metrics for classification with tion. random forest on predefined feature sets. Change model parameters the training set we use the data from [13], whereas validation and test sets were obtained during Optimum pilot testing in 2018. During validation step we are trying to maximize F1 (2) (1) score as our data set is imbalanced. We visualized the evalu- Validate Evaluate Train ation scenario in Figure 3, while the composition of the sets Join datasets in pictured in Figure 4. (3) (4) Test Use best parameters Join datasets Train + and 4. RESULTS Validate evaluate We trained random forest classifier on the predefined fea- ture sets from Table 2. Classification metrics we report on Figure 3: Schema of evaluation scenario. include classification accuracy (CA), recall (RE), precision its amplitude, amplitude of raw acceleration, and amplitude (PS) and F1 score (F1) Results are listed in Table 3. Ta- of vertical acceleration. From the same signals we extract ble 3 shows that we achieved the highest F1 score of 0.41 frequency-based features using fast Fourier transformation. using H-S feature set. This feature set consists of statisti- As frequency-based features we use the power spectrum of cal features calculated on the horizontal acceleration vector. the signal aggregated in 5 Hz bins. Pipeline for extraction of Classification accuracy in that case is also high, compared to peak-based features from dynamic and horizontal in acceler- other feature sets. The peak features seems to be the source ation is pictured in Figure 2. To extract peak-based features of noise in the data, as using peak features in combination we first smooth out the signal with convolution with a box with the other features sets decreases the performance, for window and split it into moments of acceleration and mo- example F1 drops from 0.39 for D-SF to 0.35 for D-SFP. ments of deceleration. Then we find peaks and compute F1 score and classification for dynamic acceleration increase peak heights, peak widths, peak width heights, and peak when we add frequency-based features, whereas these two areas. As there is usually more than one peak we aggregate measures decrease in case of similar action for horizontal ac- these values using mean, standard deviation, and skewness. celeration. This offers two possible interpretations. One is All together we extract 376 features. We organize features that frequency-based features of dynamic acceleration carry into seven predefined feature sets we use for classification. more information compared to frequency-based features of Feature sets are listed in Table 2. horizontal acceleration. The second one is that statistical To evaluate the capabilities and performance of the pro- features of horizontal acceleration are much better than sta- posed approach, we divide our dataset in 3 subsets — train, tistical features from dynamic acceleration. validation, and test set — based on the date the samples We noticed that smaller feature sets generally perform better were recorded on. By doing so we avoided using in this than larger so we focused on feature selection. We initially domain methodologically questionable random assignment train the model with all features and evaluate it on valida- of samples collected during the same trip to different sub- tion set. Then we remove each feature one by one, train the sets. The reason why we did not apply cross-validation is model, evaluate it on the validation set and compare all F1 similar. Using samples from the same trip in train and test scores. We eliminate the feature with the highest F1 score, set would result in significantly higher evaluation scores. For as this means that when the model was trained without that feature if performed better than when the eliminated feature was included. We repeat this procedure until the feature set consists of one feature. Similarly, we do feature addition — we start with an empty feature set and gradually add features one by one. Using the described process of forward feature selection and backward feature elimination we selected two feature sets that performed the best — in case of forward selection the best feature set has 10 features, whereas feature set pro- duced with backward elimination has 28 features. Feature set obtained by forward selection mostly contains statisti- cal features, followed by peak-based. Only one frequency- based features appears in that set. Additionally, features Figure 4: Distribution of modes in train, validation, are in vast majority extracted from dynamic acceleration. and test set. We also added joint train and valida- On the other hand feature set obtained by backward elim- tion set, which we use to train the final model. 171 Feature set CA RE PR F1 J. Urbančič. Optimum project: Geospatial data Forward selection (10) 0.70 0.50 0.47 0.48 analysis for sustainable mobility. In 24th ACM Backward elimination (28) 0.73 0.50 0.48 0.49 SIGKDD International Conference on Knowledge Table 4: Classification metrics for classification with Discovery & Data Mining Project Showcase Track. the selected features in feature selection. ACM, 2018. http://www.kdd.org/kdd2018/files/ project-showcase/KDD18_paper_1797.pdf. Forward selection Backward elimination [5] K.-Y. Chen, R. C. Shah, J. Huang, and L. Nachman. T \P Car Bus Train T \P Car Bus Train Mago: Mode of transport inference using the Car 0.78 0.27 0.05 Car 0.83 0.12 0.05 hall-effect magnetic sensor and accelerometer. Bus 0.51 0.40 0.09 Bus 0.55 0.35 0.10 Proceedings of the ACM on Interactive, Mobile, Train 0.47 0.21 0.32 Train 0.45 0.23 0.32 Wearable and Ubiquitous Technologies, 1(2):8, 2017. Table 5: Confusion matrix for classification with the [6] S.-H. Fang, Y.-X. Fei, Z. Xu, and Y. Tsao. Learning selected features in feature selection. transportation modes from smartphone sensors based ination contains more peak-based features than statistical, on deep neural network. IEEE Sensors Journal, again only one frequency-based feature appears. Dynamic 17(18):6111–6118, 2017. acceleration and horizontal acceleration appear in similar [7] D. Figo, P. C. Diniz, D. R. Ferreira, and J. M. proportions. We evaluated the models trained with that Cardoso. Preprocessing techniques for context feature sets against the test set. Results are listed in Ta- recognition from accelerometer data. Personal and ble 4. Both feature sets achieve better F1 scores than any Ubiquitous Computing, 14(7):645–662, 2010. previous feature sets. Confusion matrix in Table 5 reveals [8] S. Hemminki, P. Nurmi, and S. Tarkoma. what are the differences between these two feature sets. We Accelerometer-based transportation mode detection can see that in case of eliminating features there is less cars on smartphones. In Proceedings of the 11th ACM missclassified as buses and more buses missclassified as cars. Conference on Embedded Networked Sensor Systems, Classification of trains is fairly consistent. For buses and page 13. ACM, 2013. trains the largest part of samples is still missclassified as [9] D. Mizell. Using gravity to estimate accelerometer cars. orientation. In Proc. 7th IEEE Int. Symposium on 5. CONCLUSIONS Wearable Computers (ISWC 2003), page 252. Citeseer, 2003. We showed that while transportation mode with random for- est is possible, careful feature selection is necessary. Using [10] S. Reddy, M. Mun, J. Burke, D. Estrin, M. Hansen, feature selection we were able to improve classification scores and M. Srivastava. Using mobile phones to determine for at least 0.04, in some cases even over 0.10. Although clas- transportation modes. ACM Transactions on Sensor sification scores improved, most of non-car samples were still Networks (TOSN), 6(2):13, 2010. misclassified as cars. We observed that even though peak- [11] M. A. Shafique and E. Hato. Use of acceleration data based features did not perform as well in predefined feature for transportation mode prediction. Transportation, sets, they appeared consistently among selected features in 42(1):163–188, 2015. feature selection. That does not hold for frequency-based [12] L. Stenneth, O. Wolfson, P. S. Yu, and B. Xu. feature only one feature appeared among selected features. Transportation mode detection using mobile phones For the future work we suggest maximization of another clas- and gis information. In Proceedings of the 19th ACM sification score as we focused on maximization of F1 score. SIGSPATIAL International Conference on Advances in Geographic Information Systems, pages 54–63. 6. ACKNOWLEDGMENTS ACM, 2011. This work was supported by the Slovenian Research Agency [13] J. Urbančič, L. Bradeško, and M. Senožetnik. Near under project Integration of mobile devices into survey re- real-time transportation mode detection based on search in social sciences: Development of a comprehensive accelerometer readings. In Information Society, Data methodological approach (J5-8233), and the ICT program of Mining and Data Warehouses SiKDD, 2016. the EC under project OPTIMUM (H2020-MG-636160). [14] T. H. Vu, L. Dung, and J.-C. Wang. Transportation 7. REFERENCES mode detection on mobile devices using recurrent nets. [1] Optimum project - European Union’s Horizon 2020 In Proceedings of the 2016 ACM on Multimedia research and innovation programme under grant Conference, pages 392–396. ACM, 2016. agreement No 636160-2. [15] H. Wang, G. Liu, J. Duan, and L. Zhang. Detecting http://www.optimumproject.eu/, 2017. [Online; transportation modes using deep neural network. accessed 4-November-2017]. IEICE TRANSACTIONS on Information and [2] ActivityRecognition. https://developers.google. Systems, 100(5):1132–1135, 2017. com/android/reference/com/google/android/gms/ [16] P. Widhalm, P. Nitsche, and N. Brändie. Transport location/ActivityRecognition, 2018. [Online; mode detection with realistic smartphone sensor data. accessed 31-August-2018]. In Pattern Recognition (ICPR), 2012 21st [3] CMMotionActivity. https://developer.apple.com/ International Conference on, pages 573–576. IEEE, library/ios/documentation/CoreMotion/ 2012. Reference/CMMotionActivity_class/index.html#// [17] Y. Zheng, Q. Li, Y. Chen, X. Xie, and W.-Y. Ma. apple_ref/occ/cl/CMMotionActivity, 2018. [Online; Understanding mobility based on gps data. In accessed 31-August-2018]. Proceedings of the 10th international conference on [4] L. Bradeško, Z. Herga, M. Senožetnik, T. Šubic, and Ubiquitous computing, pages 312–321. ACM, 2008. 172 FSADA, an anomaly detection approach A modern, cloud-based approach to anomaly-detection, capable of monitoring complex IT systems Viktor Jovanoski Jan Rupnik Jozef Stefan International Postgraduate School Jozef Stefan Institute Jamova 39 Jamova 39 Ljubljana, Slovenia Ljubljana, Slovenia viktor@carvic.si jan.rupnik@ijs.si ABSTRACT huge volumes or just a few data points per day. Designing Modern IT systems are becoming increasingly complex and a system that can cope with such diverse situations can be inter-connected, spanning over a range of computing de- challenging. vices. As software systems are being split into modules and services, coupled with an increasing parallelization, de- Another important aspect is ”actionability” of the reported tecting and managing anomalies in such environments is anomalies. When human operator is presented with a new hard. In practice, certain localized areas and subsystems alert, the message as to what is wrong needs to be clear. The provide strong monitoring support, but cross-system error- operator must be able to immediately start addressing the correlation, root-cause analysis and prediction are an elusive problem. Sometimes all we need is a different presentation target. of the result, but most often the easy-to-describe algorithms and models are used - e.g. linear regression or nearest neigh- We propose a general approach to what we call Full-spectrum bour. anomaly detection - an architecture that is able to detect lo- cal anomalies on data from various sources as well as creating This high velocity of data (volume and rate) makes some high-level alerts utilizing background knowledge, historical of the algorithms less usable in such scenarios - specifically data and forecast models. The methodology can be imple- batch processing that requires random access to all past mented either completely or partially. data is not desired. Ideally, we would only use streaming algorithms - algorithms that live on the stream of incoming Keywords data, where each data point is processed only once and then discarded. Anomaly detection, Outlier detection, Infrastructure moni- toring, Cloud The contribution of this paper is a hollistic approach to anomaly detection system that clearly defines different parts 1. INTRODUCTION and stages of the processing, including active learning as a Modern IT systems need several key capabilities, apart from crucial part of the processing loop. The design addresses tracking and directing the underlying businesses. They need modern challenges in IT system monitoring and is suitable to manage errors and failures - predict them in advance, for cloud deployment. detect them in their early stages, help limit the scope of the damage and mitigate their consequences. All this is achieved by analyzing past and current data and detecting outliers in 2. ANOMALY-DETECTION it. The most common definition of an anomaly is a data point that is significantly different from the majority of other data Anomaly detection must happen in near-real time, while si- points. See [2] for a detailed explanation. This definition is multaneously analyzing potentially thousands of data points strictly analytical. But most often the users define it within per second. Incoming data that such a system can monitor the scope of their operation, such as finding abnormal engine is very diverse. Data can come in different shapes (numeric, performance in order to prevent catastrofic failure, flagging discrete or text), in regular time intervals or sporadically, in unexpected delays in manufacturing pipeline in order to pre- vent shipment bottlenecks, detecting unusual user behavior that indicates intrusion and identifying market sectors that exibit unusual trends to detect fraud and tax evasion. The anomaly-detection process is thus heavily influenced by the target domain. It also needs process-specific way of mea- suring the detection efficiency. For instance, in classification problems we can use several established measures such as accuracy, recall, precision or F 1. In anomaly detection, on the other hand, we often don’t have classes to work with 173 and secondly, we need strong user feedback to evaluate our 3. THE SYSTEM ARCHITECTURE results. Sometimes anomaly detection looks more like a fore- To create a system that is able to ingest such huge amount of casting and optimization problem. We measure how much different data streams, detect anomalies in them and present the current state of a complex system is different from the user with a manageable amount of actionable alerts we pro- optimal or predicted value. pose a reference architecture of such system (figure 1). The acronym FSADA stands for Full-Spectrum Anomaly Detec- 2.1 Actionability tion Architecture, is based on the Kappa architecture [5] and It is not sufficient for algorithms to just detect unusual pat- comprises the components described below. terns. Human operators that get notified about them must clearly understand the detected problems and be able to act • Storage module contains historical data (raw and upon them - we call this property of alerts actionability. For derived), background knowledge as well as generated instance, it is not enough to report “the euclidian distance alerts and incidents. between multi-dimensional vectors of regularized input val- ues is too big” - end-users will have no clue about what is • Stream-processing module performs incoming-data wrong here. Instead, the system should report something pre-processing, as well as signal- and incident-detection. like “The average processing time of customer orders is well above its usual values. This situation will very likely re- • Batch processing module calculates aggregations, sult in a significant drop of daily productivity.” Some algo- pattern discovery as well as background knowledge re- rithms produce models that are easier to translate into hu- fresh. man language than others. This feature needs to be taken • into account when an anomaly-detection system is being im- User-interface module (commonly abbreviated as plemented. GUI) displays raw-data, generated alerts along with feedback and active learning support. 2.2 Modern challenges In the era of big data there are many systems that produce 3.1 Terminology data and a lot of the generated data can be used to monitor, From now on we will be using the following terminology: maintain and improve the target system. The data volumes are staggering and need to be addressed properly within the anomalies - any kind of abnormal behavior inside the sys- system implementation. tem, regardless of the effect on the system performance. Users expect results to be available as soon as possible - signals - low-level anomalies that have been detected on within hours, sometimes even minutes or seconds. In cases single data-stream. where automated response in possible, this time-frame short- ens to miliseconds (e.g. stock trading, network intrusion). incidents - complex anomaly or a group of them with major impact on the system. Its time duration is usually limited Current systems for anomaly detection are developed as add- to several minutes or hours. They are closely related to the ons to the existing systems for collecting and processing way users perceive the system problems and outages. data. This makes sense, since they developed organically, during the usage by the competent users, which identified alerts - an anomaly that is reported to the user, self-contained areas that require advanced monitoring. We belive this pro- with explanation and basic context. vides necessary business validation of anomaly detection sys- tems. However, there are limitations of such approach. 3.2 Storage module The system needs to store several types of data that per- • Data that is available in one part of the system might form different functions. Each part of the storage layer can not be available in another part, where anomaly- be located in separate system that best matches the require- detection could greatly benefit from it. ments. • Data volume could prove to be too big for effective Measurement data represents raw values that were ob- anomaly detection analysis, because needed resources served and processed in order to monitor the system. This might not be available (e.g. computing power is needed data is strictly speaking not necessary when our algorithms for main processing and anomaly detection should not are designed to work on a stream, but they are required interfere with it). for batch algorithms, for model retraining, active learning • Anomaly detection has local scope as it only pro- and for ad-hoc analytics. Generated signals and inci- cesses data from one part of the system. The alerts dents are stored, additionally processed and viewed by the are thus not aware of the potential problems in other user. The storage needs to support flexible format of alerts, parts of the system, so resolving issues takes longer since each one of them is ideally an independent chunk of and involves more people from several departments to data that can be visualized without additional data retrieval. coordinate during problem escalations. The algorithms can use domain knowledge to guide their execution. To facilitate this, the data needs to be stored in • There is no systematic way of collecting the user feed- a storage system that provides fast searching, in order to be back that would guide and improve the anomaly de- used in stream processing steps for enrichment, routing and tection process. aggregation. The algorithms inside the system create and 174 Figure 1: The big picture - displays the main building layers such as stream processing and storage, as well as the flow of the data between different components of the system. update their models all the time, so this part of the stor- is required. This enables handling of previously unseen data age needs to support reliable and robust storing of possibly partitions as well as scalability in parallel processing. large binary files. These anomalies (signals) have simple models and conse- 3.3 Stream processing module quently alert explanations. But they are local in nature - their scope is most often very limited. They also operate This module contains the most important part of the system on single-data stream, so they don’t take into account the - the components that transport the data, run the processing anomalies in ‘‘the neighbourhood”. To overcome this defi- and generate alerts. ciencies, we propose the third step of stream processing, to which signals should be sent. 3.3.1 Incoming data pre-processing Incoming data that the system analyses arrives at different 3.3.3 Incident detectors volumes and speeds (high-velocity), as well as in many differ- This stage of the processing receives signals from different ent types and formats. This data needs to be pre-processed parts of the system, performing scoring of their importance, before it reaches any anomaly detection algorithms. combining them into incidents and thus achieving several advantages. Coping with such high-volume data stream requires special technologies. Streaming solutions such as Apache Kafka [4] The scoring algorithm provides option to assign user-guided have been developed and battle tested for processing millions subjective importance to signals - e.g. two statistically equally of data records per second in a distributed manner. This important anomalies can have completely difference per- step needs to perform several functions. ceived value to the user. This step can also can correlate data across data-streams, a step that is hard to achieve and Data formatting and enrichment - transform messages that proves to be very valuable. Given data from differnt from the input format into a common format that is accepted parts of the system it can create more complex constructs by the internal algorithms. Also, additional data fields can that better evaluate the impact of the current problem on be attached, based on background knowledge. the whole system. Data aggregation - sometimes we want to measure char- This level of abstraction is the main access point for end- acteristics of all the data within some time intervals (e.g. users - it more closely follows their way of addressing system average speed in the last 10 minutes). malfunctions (e.g ”if module A breaks, it will have impact on modules B and C, but module D should remain unaffected”). Data routing - send the transformed and aggregated data to relevant receivers. There may be several listeners for the 3.3.4 Background knowledge same type of input data. To help guide the algorithms during the signal detection we can provide additional background knowledge in differ- 3.3.2 Signal detectors ent forms, such as metadata, manual thresholds and rules, When data is ready for processing, it is routed to signal graphs and other structures. All this data can be used to detectors. They operate on a single data stream, often only perform various enhancements of basic algorithms, such as on a small partition of it - e.g. single stock, group of related creation of additional features in data pre-processing, up- stocks. They handle huge data volumes, so they need to be and down-voting of results (e.g. estimated impact of de- fast, using very little resources. To achieve great flexibility tected anomaly), pruning of search space in optimization regarding input data a dynamic allocation of such processors steps, estimation of affected entities for given anomaly 175 or support for complex simulations when current per- terval. Users feel comfortable with seeing the big picture formance is measured against historical values. These rules (an indcident) in then drill down into specific data (indi- and metadata can be acquired by analyzing historical data vidual signal). They reported this feature enables them to as well as collecting knowledge from end-user, e.g. manual cut down time for understanding the problem by an order feedback/sign-off and active learning. of magnitude (from hours to minutes). 3.4 Improving actionability Active-learning component was well received, as it made The system modules presented so far are mostly established manual work more efficient. The users noticed how the al- components that are used also in normal processing steps gorithm was choosing more and more complex learning ex- of modern, cloud-based systems (see [1]). We propose that amples for manual classification. This helped them feel pro- they should be upgraded with the following functionalities ductive and engadged. They also reported positive impact in order to achieve the goal of high-quality actionable alerts, of active learning on their problem understanding, as they empowering users to manage their complex systems in the were presented with some potentially problematic situations most efficient way. that went unnoticed in the past. Based on above observations were conclude that our pro- 3.4.1 Feedback posed approach has positively impact on the organization, Historical incidents are very valuable for learning of informa- both for technologies as well as human operators. Additional tive features that aid detection of anomalies. They are also ideas that were collected from users are listed under future used for calibrating scoring algorithm that assigns relevance work. scores to generated signals and incidents. It is common for the organization to require every major detected incident to 5. CONCLUSIONS AND FUTURE WORK be manually signed off - a relevance tag (e.g. high-relevant, The focus of our future work is on several advanced scenar- semi-relevant, not-relevant, noise, new-normal) has to be as- ios where a lot of added value for users is expected, mix- signed to it by the operators. These tags are used for train- ing anomaly detection, optimization and simulation. Main ing of incident-classification algorithms, but we can also con- gains are expected to come from feedback collection and ac- struct a more complex setting where a form of backtracking tive learning. Apart from monitoring IT systems, the target is used to calibrate signal detectors. domains are also manufacturing and smart cities. We also collected some features that users commonly inquired about: 3.4.2 Active learning The active learning approach [3] can be used to make the manual classification effort more efficient. The system pro- • Root-cause analysis - when a major incident occurs, vides untagged examples/incidents where the criteria func- many parts of the system get affected. To resolve is- tion returns the value that is the closest to the decision sues as quickly as possible, the operators should be boundary. The user then manually classifies the incident pointed to the origin of the problem. The anomaly de- and the classification model is re-trained with this new data. tection system should thus have a capability to point By guiding users in this way the system requires relatively to the first signal with high-impact on the final rele- small number of steps to cover the search space and obtain vance score. good learning examples. • Predictions - The goal for all monitoring systems is to detect problems as soon as possible. The system Our proposed approach incorporates this continuous activ- must that not only be able to detect signals, but also ity in several areas. GUI module should contain appropriate forecast them, based on past behavior. In order to do pages where user can enter his feedback and active-learning that, the algorithms require more metadata and struc- input. Storage module contains alerts historical data that ture to properly model inter-dependencies between sig- can be used for re-training of incident detectors. Storage nals. Mere observation is much easier than simulation module also contains old and new incident-detector mod- of a complex system with many moving parts. But it els that can be picked up automatically by the processing is possible and is what users expect from a modern AI- module. based system. Our future reserach will be oriented to- wards providing and efficiently integrating these func- 4. VALIDATION AND DISCUSSION tionalities into our anomaly-detection approach. Based on our extensive experience with practical anomaly detection implementation, we identified several new require- 6. REFERENCES ments for these systems. The presented approach satisfies [1] Anodot anomaly detection system. them by supporting big-data real-time analytics on one side http://www.anodot.com, 2018. and actionability via active-learning support on the other. [2] C. C. Aggarwal. Outlier Analysis. Springer New York, New York, New York, 2013. The system architecture is deployable to cloud environment [3] D. A. Cohn, Z. Ghahramani, and M. I. Jordan. Active by design. We also employ modern streaming and storage learning with statistical models. CoRR, cs.AI/9603104, technologies for transporting and storing of different input 1996. data and alerts. [4] N. Garg. Apache Kafka. Packt Publishing, 2013. We observed that users appreciate our notion of incidents [5] J. Lin. The lambda and the kappa. IEEE Internet - a grouping of alerts that occur in certain small time in- Computing, 21(5):60–66, 2017. 176 Predicting customers at risk with machine learning David Gojo Darko Dujič Jožef Stefan International Postgraduate School, Ceneje d.o.o., Jamova 39, 1000 Ljubljana, Slovenia Jožef Stefan International Postgraduate School, david.gojo@ijs.si Štukljeva cesta 40, 1000 Ljubljana, Slovenia darko.dujic@ceneje.si ABSTRACT 2. RELATED WORK Today’s market landscape is becoming increasingly competitive Improvements in tracking technology have enabled data driven as more advanced methods are used to understand customer’s industries to analyze data and create insights previously behavior. One of key techniques are churn mitigation tactics unavailable to the business. Data mining techniques have evolved which are aimed at understanding which customers are at risk to to now support the prediction of behavior of customers such risk leave the service provider and how to prevent this departure. This of leaving due to the attributes that are trackable [2]. The use of paper presents analyzes accounts renewal rates and uses easily data mining methods has been widely advocated as machine applicable models to predict which accounts will be decreasing learning algorithms, such as random-forest approaches have spend at the time when they are due to renew their existing several advantages over traditional explanatory statistical contract based on number of attributes. Key questions it tries to modeling [3]. explore is if customer behavioral or customer characteristic data Lack of predefined hypothesis makes algorithms excel in these (or combination of both) is better at predicting accounts that will tasks as it is making it less likely to overlook predictor variables renew at lower than renewal target amount (churn rate). or potential interactions that would otherwise be labelled Categories and Subject Descriptors unexpected [4]. Models are often labelled as business intelligence models aimed at finding customers that are about to switch to F.2.1 [Numerical Algorithms and Problems]: Data mining, competitors or leave the business [5]. Structured prediction Key classifications are observed in work related to churn that we General Terms will use in our data set for review [6]: Algorithms, Management, Measurement, Documentation, - Behavioral data - will consist of attributes that we have Performance historically observe that play a role in whether the account will renew or not: product utilization, activity Keywords levels of the account, number of successful actions in Data Mining, Analysis, Churn prediction. the account and number of upsells done ahead of renewal. 1. INTRODUCTION - Characteristic attributes - will consist of size of the The main issue of business is how to make educated decision with account in terms of spend, number of members in the support of analysis that dissect complex decisions on addressable company, number of active users of the products in the problems using measurements and algorithms. Where there are company, payment method and how they renew the many disciplines are researching methodological and operational contract, geography and what level of support the aspects of decision making, at the main level, we distinguish product is given (number of sales visits and interactions between decision sciences and decision systems [1]. With with the customer). increasing number of companies trying to use machine learning to assist in their decision-making process we examined how decision science can be supplemented by applying machine learning models to the company’s customer data. We partnered 3. DATA ACQUISITION with the medium sized B2B business operating in Europe and 3.1 Data understanding Africa with the aim to help them better understand their ‘customers at risk’ segment of clients. Working with the customer we have arranged a set of interviews with the leadership to better understand their business and To this end we developed two easily applicable performance challenges they are experiencing together with ambitions they algorithms designed to highlight customers at risk and company have in the business. After the interview rounds we focused on can address to mitigate their risk of leaving as clients. reviewing 2 hypotheses flagged in the examination process: The paper has the following structure: in section 2 we are - What is driving churn numbers: behavior of the customers or presenting related work to the area recorded historically. Next, better structure of the base? data acquisition is explained in section 3 followed by results - Does acquisition of new accounts represent a risk in churn acquired from the tested algorithms in section 4. We then number with historic observation of accounts renewing lower / conclude our observations in section 5. not renewing in their first-year renewal? 177 3.2 Data pre-processing or if no descriptor can be found that would result to the The data we used derives from company’s internal customer information gain. bookings and customer databases we consolidated. As customers Random Forest. We assume that the user knows about the are on yearly renewals we have taken the renewal and the data on construction of single classification trees. Random Forests grows the account before the renewal as the key building block for many classification trees. To classify a new object from an input analysis. vector, put the input vector down each of the trees in the forest. Each tree gives a classification, and we say the tree "votes" for 3.3 Feature engineering that class. The forest chooses the classification having the most We enriched the data using SQL joins on the customer numbers votes (over all the trees in the forest) [7]. Both methods were to include key characteristics of accounts, tenure of the client, applied to imported dataset numerous times with continuous products utilization information, behavior of the customer before testing of parameters to improve performance. the renewal and their usage of the product. In terms of regional split of the market the dataset consists of 4 4.2 Application of J48 key geo and segment regions in Europe and Africa: - Medium-business segment Working with Weka on the dataset of the customer we tried to - UK & Ireland market launch the model to tune the parameters. Key modifications: - Europe Enterprise segment - “10-fold cross validation” used to improve accuracy - Eastern Europe, Middle-East and Africa - Minimum number of objects moved to 100 Through feature engineering and reviewing descriptive statistics As Figure 2 shows this reduced the number of leaves to 16 which key attributes we nominalized are 11. was something comprehendible enough. For the machine learning purposes for the calls we have selected 3 possible outcomes related to the outcome of customer spend Summary of results below: after it’s renewal: - Customer_Renew (Not renew, Partial renew, Full renew) 3.4 Data Set Statistics We selected bookings period from 2016 to end of 2017 including 23,043 instances in above selected renewal of 12,872 accounts. The attributes that were nominalized are listed below: - (nom) Has main product – has product 1 - (nom) Has_assisting_product – has product 2 - (nom) Has_media_product – product 3 - (nom) Account_potential – size and potential of the account Figure 1: J-48 model error estimates - (nom) Is_Auto_Renew – auto renewal option enabled 4.3 Application of Random forest - (nom) First_renewal – is the client renewing first time We ran several tests on Random forest vs Random trees. When - (nom) Upsold_Before_renewal – was there an upsell before tuning parameters Random tree tended to not respond well to - (nom) JS_Utilization – utilization of product 2 - indicator pruning so Random forest was a preferred option. Like J48, - (nom) Score_Engagement – engagement of the recruiter application with key modifications was focused on validation and - (nom) LRI_Score – savviness of the user of the product additionally on setting maximum depth of the random forest: 4. RESULTS - “10-fold cross validation” - Max. depth set at 3 4.1 Brief description of the methods used Summary of results below: Where multiple algorithms were used during the testing due to important feature that the result needed to have at least one interpretable model, so we went in the direction of nominalizing attributes and decided to use J-48 model and Random forest classifier on the data set. J48. Decision trees C4.5 (J48 in Weka) algorithm: deals with continuous attributes as observed in the related work. Where the method is classification-only the main machine learning method applied is J48 pruned tree or WEKA-J48 machine learning method. Tree tries to partition the data set into Figure 2: Random forest model error estimates subsets by evaluating the normalized information gain from choosing a descriptor for splitting the data. The training process stops when the resulting nodes contain instances of single classes 4.4 Comparisons of models Overall the J48 model has surprisingly 0.7pp points higher Classification Accuracy than the Random forest model. 178 Validation Measures J48 Random Forest J48 provided a significantly better interpretability and classification accuracy than the Random forest or any test on the Classification Accuracy 72.9% 72.2% Random tree model. Some additional tests were done on Naïve Bayes model and J48 was superior in the results. Key area it accelerated was in predicting accounts that will not renew. Where Mean absolute error 0.276 0.280 the precision is just above 38% this is almost double comparing to Random forest model. Table 1. Baseline benchmark validation measures 3 key takeaways observed that the company found the most Key observation analyzing the data was that neither model was insightful were: predicting any partially churned accounts after we forced their trees to be pruned. - One of the new features designed by the product team that encouraged the auto-renew of their clients played the most important at predicting the renewal rate J48 predictions: - Customer behavior is a better signal for probability of a b c <-- classified as renewal vs general account characteristics - Account potential which is the predictor of account potential 0 2745 285 | a = PARTIAL_RENEW and size plays the role only after product utilization and 0 1528 789 | b = FULL_RENEW engagement of the account with our products 0 2434 1504 | c = NOT_RENEWED potent al potent al 5. CONCLUSION AND FUTURE WORK Figure 3: The J48 decision tree For the acceleration of performance, the decision tree is of paramount importance and value. Insight that Auto renew as a Random forest predictions: feature is one of the key predictors if the account will renew fully a b c <-- classified as is truly remarkable based on the simplicity of the models and how easily applicable they are. 0 2857 173 | a = PARTIAL_RENEW 0 15591 483 | b = FULL_RENEW Applications of this models will be of great foundation for driving the discussion on different account features and metrics. This is 0 2894 1044 | c = NOT_RENEWED especially true as it is tackling one of the key challenges observed in hypothesis as in how important ‘account potential’ is for the account ahead of the renewal. Even though Random forest has a lower classification accuracy J48 offers significantly higher interpretability with tree pruning Observing the attributes added into the analysis scope and offering valuable insights, short description below and discussed optimizing them for the J48 we were able to get valuable insight in evaluation of models. which account characteristics vs account behaviors ahead of the renewal are the best predictors for the account to renew at the full amount. 179 6. REFERENCES [5] M. Óskarsdóttir, B. Baesens and J. Vanthienen, "Profit- Based Model Selection for Customer Retention Using Individual Customer Lifetime Values," Big Data, vol. [1] M. Bohanec, Decision Making: A Computer-Science 6, no. 1, pp. 53-65, 3 2018. and Information-Technology Viewpoint, vol. 7, 2009, pp. 22-37. [6] S. Kim, D. Choi, E. Lee and W. Rhee, "Churn prediction of mobile and online casual games using [2] A. Rodan, A. Fayyoumi, H. Faris, J. Alsakran and O. play log data," PLOS ONE, vol. 12, no. 7, p. e0180735, Al-Kadi, "Negative correlation learning for customer 5 7 2017. churn prediction: a comparison study.," TheScientificWorldJournal, vol. 2015, p. 473283, 23 3 [7] J. Hadden, A. Tiwari, R. Roy and D. Ruta, "Computer 2015. assisted customer churn management: State-of-the-art and future trends," Computers & Operations Research, [3] A. K. Waljee, P. D. R. Higgins and A. G. Singal, "A vol. 34, no. 10, pp. 2902-2917, 10 2007. Primer on Predictive Models," Clinical and Translational Gastroenterology, vol. 5, no. 1, pp. e44- [8] A. K. Meher, J. Wilson and R. Prashanth, "Towards a e44, 2 1 2014. Large Scale Practical Churn Model for Prepaid Mobile Markets," Springer, Cham, 2017, pp. 93-106. [4] Y. Zhao, B. Li, X. Li, W. Liu and S. Ren, "Customer Churn Prediction Using Improved One-Class Support [9] M. Ballings, D. Van den Poel and E. Verhagen, Vector Machine," Springer, Berlin, Heidelberg, 2005, "Improving Customer Churn Prediction by Data pp. 300-306. Augmentation Using Pictorial Stimulus-Choice Data," Springer, Berlin, Heidelberg, 2012, pp. 217-226. 180 Text mining MEDLINE to support public health João Pita Costa, Luka Stopar, Raghu Santanam, Paul Carlin Michaela Black, Flavio Fuart, Marko Grobelnik Chenlu Sun South Eastern Health and Jonathan Wallace Jožef Stefan Institute, Ljubljana Arizona State University, USA Social Care Trust, UK Ulster University, UK Quintelligence, Ljubljana, Slovenia ABSTRACT demonstration paper focuses on this large open dataset, and the exploration of its structured data. Today’s society is data rich and information driven, with access to numerous data sources available that have the potential to provide new insights into areas such as disease prevention, personalised medicine and data driven policy decisions. This paper describes and demonstrates the use of text mining tools developed to support public health institutions to complement their data with other accessible open data sources, optimize analysis and gain insight when examining policy. In particular we focus on the exploration of MEDLINE, the biggest structured open dataset of biomedical knowledge. In MEDLINE we utilize its terminology for indexing and cataloguing biomedical information – MeSH – to maximize the efficacy of the dataset. Categories and Subject Descriptors Figure 1. MIDAS platform dashboard, composed of visualisation modules customized to the public health data sourced in each H.4 [Information Systems Applications]: Miscellaneous governamental institution, and combined with open data. General Terms 2. THE MEDLINE BIOMEDICAL OPEN DATA Measurement, Performance, Health. SET AND IT’S CHALLENGES. Keywords 2.1. MEDLINE DATASET. Big Data, Public Health, Healthcare, Text Mining, Machine Learning, MEDLINE, MeSH Headings. With the accelerating use of big data, and the analytics and visualization of this information being used to positively affect the daily life of people worldwide, health professionals require 1. MEANINGFUL BIG DATA TOOLS TO more and more efficient and effective technologies to bring SUPPORT PUBLIC HEALTH added value to the information outputs when planning and delivering care. The day-to-day growth of online knowledge The Meaningful Integration of Data, Analytics and Service requires that the high quality information sources are complete, [MIDAS], Horizon 2020 (H2020) project [1] is developing a high quality and accessible. A particular example of this is the big data platform that facilitates the utilisation of healthcare PubMed system, which allows access to the state-of-the-art in data beyond existing isolated systems, making that data medical research. This tool is frequently used to gain an amenable to enrichment with open and social data. This overview of a certain topic using several filters, tags and solution aims to enable evidence-based health policy decision- advanced search options. PubMed has been freely available making, leading to significant improvements in healthcare and since 1997, providing access to references and abstracts on life quality of life for all citizens. Policy makers will have the sciences and biomedical topics. MEDLINE is the underlying capability to perform data-driven evaluations of the efficiency open database [7], maintained by the United States National and effectiveness of proposed policies in terms of expenditure, Library of Medicine (NLM) at the National Institutes of Health delivery, wellbeing, and health and socio-economic (NIH). It includes citations from more than 5200 journals inequalities, thus improving current policy risk stratification, worldwide journals in approximately 40 languages (about 60 formulation, implementation and evaluation. MIDAS enables languages in older journals). It stores structured information on the integration of heterogeneous data sources, provides privacy- more than 27 million records dating from 1946 to the present. preserving analytics, forecasting tools and visualisation About 500,000 new records are added each year. 17.2 million of modules of actionable information (see the dashboard of the these records are listed with their abstracts, and 16.9 million first prototype in Figure 1). The integration of open data is articles have links to full-text, of which 5.9 million articles have fundamental to the participatory nature of the project and core full-text available for free online use. In particular, it includes ideology, that heterogeneity brings insight and value to 443.218 full-text articles with the key-words string “public analysis. This will democratize, to some extent, the health”. contribution to the results of MIDAS. Moreover, it enables the 2.2. MEDLINE STRUCTURE. MIDAS user to profit from the often powerful information that exists in these open datasets. A point in case is MEDLINE, the The MEDLINE dataset includes a comprehensive controlled scientific biomedical knowledge base, made publicly available vocabulary – the Medical Subject Headings (MeSH) – that through PubMed. The set of tools described in this 181 delivers a functional system of indexing journal articles and public health policy making, a suitable MIDAS PubMed books in the life sciences. It has proven very useful in the search repository had to be developed. This repository has to allow of specific topics in medical research, which is particularly exploration of a wide range of different visualisation techniques useful for researchers conducting initial literature reviews in order to evaluate their applicability to policy-making tasks before engaging in particular research tasks. Humans annotate within the policy cycle. Therefore, there was a need for a most of the articles in MEDLINE with MeSH Heading selection of a powerful, semi-structured text index, that would descriptors. These descriptors permit the user to explore a allow free text searches, but also allow the creation of complex certain biomedical related topic, which relies on curated queries based on available metadata. An obvious choice is information made available by the NIH. MeSH is composed of elasticSearch, which combines features provided by NoSQL 16 major categories (covering anatomical terms, diseases, drugs, databases with standard full text indexes, as it is based on the etc) that further subdivide from the most general to the most Apache Lucene Index. The main design challenge when specific in up to 13 hierarchical depth levels. choosing this particular toolset was that querying based on arrays or parent-child relations are not supported, meaning that for complex use-cases different indexes based on the same source dataset have to be created. Nevertheless, excellent results, particularly with regards to the area of performance have been obtained. 2.4. MEDLINE DASHBOARD. One of the identified needs motivating this work is assuring the availability of a dynamic dashboard that permits the user to explore data visualisation modules, representing the queries to the MEDLINE dataset through pie charts, bar charts, etc [5]. The dashboard that we made available (in Figure 2) feeds on that dataset through the elasticSearch index earlier discussed. It is composed of several interactive visualisation modules that utilises the mouse hover when interacting and provide information through pop-up messages on several aspects of the data based on particular queries of interest (e.g. a pie chart representing the “public health” citations that talk about “childhood obesity” during a selected period of time; or a bar chart showing different concepts included in the articles related to “mental health” in Finnish scientific journals). The MEDLINE dataset is mostly in the English language but includes a significant volume of translated abstracts of scientific articles that were written in several other languages. The open source data visualisation Kibana is a plugin to elasticSearch that supports the described dashboard. Thus, it is the data visualisation dashboard of choice for elasticSearch-based Figure 2. MEDLINE data visualisation tool enabling exploration of that indexes, such as the one we present here. It is used in the context open dataset in its full potential, based on data representations easy to understand and to communicate. It provides an interactive public of MIDAS for fast prototyping and support of part of MIDAS instance that can be managed at the dashboard management tool use-cases. While the dashboard itself serves the less technical (below) for which the visualisation modules are constructed (in the user to explore the data available (over a subset of the data center) based on the queries made to the MEDLINE dataset (above). generated by a topic of interest), other options are available that permit more control of the data by the data scientists at a more 2.3. MEDLINE INDEX. operational level. These are: (i) the management dashboard, This paper demonstrates the interactive data visualisation text- where the technical user can perform the appropriate mining tools that enable the user to extract meaningful subsampling based on the topics of interest and the required information from MEDLINE. To do that we are using the advanced options over the available data features; and (ii) the underlying ontology-like structure MeSH. MEDLINE data, visual modules creator permitting the technical user to easily together with the MeSH annotation, that is indexed with create new interactive visualisation modules. Moreover, this tool ElasticSearch and made available to data analytics and enables one to query large datasets and produce different types visualisation tools. This will be discussed in more detail in the of visualisation modules that can be later integrated into next section. customized dashboards. The flexibility of such dashboards The manipulation and visualization of such a complete data permits the user to profit from data visualisations that feed on source brings challenges, particularly in the efficient search, his/her preferences, previously set up as filters to the dataset. review and presentation when choosing appropriate scientific The MIDAS data visualisation tools permit the user to explore knowledge. The manipulation and visualisation of complex text the potential of the MEDLINE dataset, based on pie charts and data is an important step in extracting meaningful information other representations that are easy to comprehend, interact with, from a dataset such as MEDLINE. Although powerful, the and to communicate. It also enables a public instance based on a online search engine provided by the NLM does not provide particular query to the dataset, which includes different types of suitable tools for in-depth analysis and the emergence of data visualisation modules that can later integrate a customised scientific information. As one of the main goals of MIDAS is to dashboard, designed in agreement with the workflows and experiment with advanced visualisation techniques in support of preferences of the end-user. This live dashboard can easily be 2 182 integrated through an iframe in any website, not showing the could be transformed as D1 = (1, 1, 1, 0) and D2 = (1, 1, 0, 1). customization settings but maintaining the interaction capability Then the documents are clustered into k groups G1, G2, ..., Gk and the real-time update. It permits a complete base solution to using the k-means algorithm. For each group we compute the further explorer the MEDLNE index and the associated dataset "average" document (centroid), which is the representative of [6]. the group. The most frequent words in the "average" document are drawn in the word cloud - the central grey word cloud is the "average" of all the documents in S. We can calculate how similar a particular document d is to a group Gi by calculating the cosine of the angle between the vector representation of d and the "average" document (centroid) of the group Gi. By dragging the red SearchPoint ball over the word-groups, we provide the relevance criteria to the search result, thus bringing to the top results the articles we are most interested in (see Figure 4). When that ball is moved, for each document, we calculate the similarity to each of the word-groups and combine it with the distance between the ball and the group. The result is used as the ranking weight where the document with the highest cumulative weight is ranked first. When having the mouse over the word-clouds we get a combination of the most frequent words shown in the tag clouds that change based on how close the ball is to a particular group. After getting to a position with the SearchPoint over the word cloud highlighting “primary Figure 4. A screenshot of the MEDLINE SearchPoint, with groups of care”, a qualitative study in primary care on childhood obesity keywords (on the right) extracted from the search results, represented that occupied the position 188 is now in the first position. The by different colors, and on the left the reindexed search results user can read its title and first lines of abstract, and when themselves with the number that they appear in the original index [6]. clicking on it, the system opens the article in the browser at its PubMed URL location. 4. MEDLINE SEARCHPOINT. 3. MeSH CLASSIFIER The efficient visualisation of complex data is today an important step in obtaining the research questions that describe the This rich data structure in the MEDLINE open set is annotated problem that is extracted from that data. The MEDLINE by human hand (although assisted by semi-automated NIH SearchPoint is an interactive exploratory tool refocused from the tools) and therefore is not available in the most recent citations. proprietary open source technology SearchPoint [8] (available at However, in the context of MIDAS we made available an searchpoint.ijs.si) to support health professionals in the search automated classifier based on [2] that is able to suggest the for appropriate biomedical knowledge. It exhibits the clustered categories of any health related free text. It learns over the part keywords of a query, after searching for a topic. When we use of the MEDLINE dataset that is already annotated with MeSH, indexing services (such as standard search engines) to search for and is be able to suggest categories to the submitted text information across a huge amount of text documents – the snippets. These snippets can be abstracts that do not yet include MEDLINE index described in Section 2 being an example – we MESH classification, medical summary records or even health usually receive the answer as a list sorted by a relevance criteria related news articles. To do that we use a nearest centroid defined by the search engine. The answer we get is biased by classifier [3] constructed from the abstracts from the MEDLINE definition. Even by refining the query further, a time consuming dataset and their associated MeSH headings. Each document is process, we can never be confident about the quality of the embedded in a vector space as a feature vector of TF-IDF result. This interactive visual tool helps us in identifying the weights. For each category, a centroid is computed by averaging information we are looking for by reordering the positioning of the embeddings of all the documents in that category. For higher the search results according to subtopics extracted from the levels of the MeSH structure, we also include all the documents results of the original search by the user. For example, when we from descendant nodes when computing the centroid. To enter a search term ‘childhood obesity, the system performs an classify a document, the classifier first computes its embedding elasticSearch search over the MEDLINE dataset, extracts groups and then assigns the document to one or more categories whose of keywords that best describe different subgroups of results centroids are most similar to the document’s embedding. We (these are most relevant, and not most frequent terms). This tool measure the similarity as the cosine of the angle between the gives us an overview of the content of the retrieved documents embeddings. Preliminary analysis shows promising results. For (e.g. we see groups of results about prevention, pregnancy, instance when classifying the first paragraph of the Wikipedia treatments, etc.) represented by: (i) a numbered list of 10 page for “childhood obesity”, excluding the keyword “childhood MEDLINE articles with a short description extracted from the obesity” from the text, the classifier returns the following MeSH first part of the abstract; (ii) a word-cloud representing the k- headings: means clusters of topics in the articles that include the searched keywords; (iii) a pointer that can be moved through the word- "Diseases/Pathological Conditions, Signs and cloud and that will change the priority of the listed articles. The Symptoms/Signs and Symptoms/Body Weight/Overweight", word-cloud in (ii) is done by taking a set of MEDLINE "Diseases/Pathological Conditions, Signs and documents S and transforming them into vectors using TF-IDF, Symptoms/Signs and Symptoms/Body where each dimension represents the "frequency" of one Weight/Overweight/Obesity". particular word. For example, lets say that we have document The demonstrator version of the MeSH classifier is already D1: "psoriasis is bad" and document D2: "psoriasis is good". This available through a web app, as well as through a REST API 3 183 using a POST call, and is at the moment under qualitative the research topic over time windows that enable filtering to evaluation. This is being done together with health professionals avoid known unreliable results. with years of practical experience in using MeSH themselves through PubMed. In addition, we aim to further explore the In line with this work we have been developing research to potential of the developed classifier in several public health contribute to the smart automation of the production of related contexts including non classified scientific articles of biomedical review articles. This collaborative research lead by three types: (i) review articles; (ii) clinical studies; and (iii) Raghu T. Santanam at Arizona State University, aims to provide standard medical articles. The potential impact of this a wide knowledge over a restricted topic over the wider technology will also include electronic health records and the knowledge available at MEDLINE. We utilize the deep learning monitoring health related news sources. We believe that his algorithm Doc2vec [4] to create similarity measures between approach will address an identified recurrent need of health articles in our corpus. In that we built a balanced test dataset and departments to enhance the biomedical knowledge, and motivate three different representations of the corpus, and compared the a step change in health monitoring. performance between them. The implementation currently builds a matrix of similarity scores for each article in the corpus. In the next steps, we will compare similarity of documents from our implementation against the baseline for a randomly chosen set of articles in the corpus. The further development of the MeSH classifier will consider the feedback of the usability of health professionals working in partner institutions, profiting of their years of experience with MEDLINE and MeSH itself, to tune the system to ensure the best usability in the domain. Furthermore, we will use the outcomes of the final version of this classifier to label health related news with the MeSH Headings descriptors, potentiating a new approach on the processing and monitoring of population health, population awareness of certain diseases, and the general public acceptance of public health decisions through news. Figure 3. A screenshot of the web app to the MEDLINE classifier, when ACKNOWLEDGMENTS requesting the automated MeSH annotation of a scientific review article abstract extracted from PubMed (in the body of text above) and the We thank the support of the European Commission on the results as MeSH headings descriptors including their tree path in the H2020 project MIDAS (G.A. nr. 727721). MeSH ontology-like structure (center), their similarity measure (right) REFERENCES and their positioning in the classification (left). [1] B. Cleland et al (2018). Insights into Antidepressant 5. CONCLUSION AND FUTURE WORK Prescribing Using Open Health Data, Big Data Research, doi.org/10.1016/j.bdr.2018.02.002 To further extend the usability of the MEDLINE SearchPoint, [2] L. Henderson, Lachlan (2009). Automated text classification we are developing a data visualisation tool that permits the user in the dmoz hierarchy. TR. to plot the top results mostly related with a topic of interest, as [3] C. Manninget al (2008), “Introduction to Information explored with SearchPoint. Based on the choice of a time Retrieval,” Cambridge Univ. Press, 2008, pp. 269-273. window and a certain topic, such as “mental health”, the user is [4] T. Mikolov et al (2013). Efficient Estimation of Word able to view the clustered MEDLINE documents, rolled over the Representations in Vector Space, arXiv:1301.3781. plot or click to view the plotted points, each of which represents [5] J. Pita Costa et al (2017). Text mining open datasets to an article in PubMed. This will be done through support public health. In Conf. Proceedings of WITS 2017. multidimensional scaling, plotting the articles in the subsample [6] J. Pita Costa et al (2018). MIDAS MEDLINE Toolset Demo. using cosine text similarity. The difficulties to plot large datasets http://midas.quintelligence.com (accessed in 28-8-2018). using these methods, and the lack of potential in the outcomes of [7] F. B. Rogers, (1963). Medical subject headings. Bull Med that heavy computation, provided a focus for the team to only Libr Assoc. 51: 114–6. plot the first hundred results of the explorations done within [8] L. Stopar, B. Fortuna and M. Grobelnik (2012). Newssearch: MEDLINE SearchPoint. With this extended tool the healthcare Search and dynamic re-ranking over news corpora. In Conf. professional will be able to: (i) explore a certain area of research Proceedings of SiKDD2012. with the aim of a more accessible scientific review, in identifying the evidence base for a medical study or a diagnostic decision; (ii) identify areas of dense scientific research corresponding to searchable topics (e.g. the evaluation of the coverage of certain rare diseases that need more biomedical research, or the identification of alternative research paths to overpopulated but inefficient research); and (iii) exploration of 4 184 Crop classification using PerceptiveSentinel Filip Koprivec Matej Čerin Klemen Kenda Jožef Stefan Institute Jožef Stefan Institute Jožef Stefan Institute Jamova 39, 1000 Ljubljana, Jamova 39, 1000 Ljubljana, Jožef Stefan International Slovenia Slovenia Postgraduate School filip.koprivec@ijs.si matej.cerin@ijs.si Jamova 39, 1000 Ljubljana, Slovenia klemen.kenda@ijs.si ABSTRACT data labels, which will become apparent when interpreting Efficient and accurate classification of land cover and land results. usage can be utilized in many different ways: ranging from natural resource management, agriculture support to legal Another class of problems is posed by the spatial resolu- and economic processes support. In this article, we present tion of images. Since satellite images provided by the ESA an implementation of land cover classification using the Per- Sentinel-2 mission have a resolution of 10 m × 10 m on most ceptiveSentinel platform. Apart from using base 13 bands, granular bands and 60 m × 60 m on bands used for atmo- only minor feature engineering was performed and different spheric correction, land cover irregularities falling in this classification methods were explored. We report an F1 and order of magnitude might not be detected and correctly accuracy score (80-90%) in range of state of the art when learned on. This problem is especially prevalent in smaller using pixel-wise classification and even comparable to time and more diverse regions, where individual fields are smaller series based land cover classification. and land usage is more fragmented. The current state of the art land classification focuses heav- Keywords ily on the temporal dimension of acquired data [1], [13], remote sensing, earth observation, machine learning, classi- [14]. The time-based analysis offers clear advantages since fication it considers growth cycles of sample crops, enables continu- ous classification etc., and generally produces better results, 1. INTRODUCTION with reported F1 scores for crop/no-crop classification in a range from 0.80-0.93 [14]. One major drawback of time- Specific aspects of earth observation (EO) data (huge amount based classification is the problem of missing data. In our of data, widespread usage, many different problem settings test drive scenario, 70% of images are heavily obscured by etc.), coupled with the recent launch of ESA Sentinel mission clouds [9], a problem which removes a lot of the advantages that provides a huge volume of data relatively frequently (ev- of time-based classification and demands major compensa- ery 5-10 days), present an environment that is suitable for tions with missing data imputation. current machine learning approaches. In this paper, we present a possible approach on a land cover Efficient and accurate land cover classification can provide classification of single time image acquired using the Percep- an important tool for coping with current climate change tiveSentinel 1 platform, using multiple classification meth- trends. Classification of crops, their location and potentially ods for tulip field classification in Den Helder, Netherlands. their yield prediction provide various interested parties with information on crop resistance, adapting to changes in tem- perature and water level changes. Along with direct help, 2. PERCEPTIVESENTINEL PLATFORM accurate crop classification tools can be used in a variety of 2.1 Data other programs, from government based subsidies to various Data used in this article is provided by ESA Sentinel-2 mis- insurance schemes. sion. The Sentinel-2 mission comprises a constellation of two polar-orbiting satellites placed in the same orbit, phased at Along with previously highly promising features of EO data, 180◦ to each other [2]. Sentinel-2A was launched on 23rd data acquisition and processing pose some specific challenges. June 2015, while the second satellite was launched on 7th Satellite acquired data is highly prone to missing data due March 2017. Revisit time for equator is 10 days for each to various reasons; mostly cloud coverage, (cloud) shadows, satellite, so since the launch of the second satellite, each atmospheric refraction due to changes in atmospheric con- data point is sampled at least every 5 days (a bit more fre- ditions. Additionally, correct training data, either for clas- quently when away from the equator). sification or regression, is hard to come by, must be rela- tively recent, and regularly updated due to changes in land Each satellite collects data in 13 different wavelength bands use. Furthermore, correct labels and crop values are almost presented in figure 1, with varying granularity. Data ob- impossible to verify and usually self-reported, which often tained for each pixel is firstly preprocessed by ESA where means that quality of training data is not perfect. Valero et al. [13] raise the problem of incorrect (or partially correct) 1http://www.perceptivesentinel.eu/ 185 atmospheric reflectance and earth surface shadows are cor- 3. METHODOLOGY rected [4]. 3.1 Sample Data For purpose of this article, a sample patch of fields in Den Helder, Netherlands, with coordinates: (4.7104, 52.8991), (4.7983, 52.9521) was analyzed. Three different datasets were considered: tulip fields in year 2016 and 2017 and arable land in 2017. For each of these datasets, the first ob- served date with no detected clouds was selected and binary classification (tulips vs no-tulips and arable vs non-arable land) was performed on the image from that date. The date selection was based on the fact that tulips’ blooms are most apparent during late April and beginning of May [9]. 3.2 Feature Vectors Three additional earth observation indices that were used as features are presented in Table 1 as suggested by [8]. Figure 1: Sentinel 2 spectral bands [12] Name Formula 2.2 Data Acquisition Satellites provide around 1TB of raw data per day, which B08 − B04 NDVI is provided for free on Amazon AWS. Images are then pro- B08 + B04 cessed and indexed by Sinergise and subsequently provided for free along with their SentinelHub [11] library. As part 2.5(B08 − B04) EVI of the PerceptiveSentinel project, a sample platform was de- (B08 + 6B04 − 7.5B02 + 1) veloped on top of SH library, which eases data acquisition, cloud detection and further data analysis on acquired data. B08 − B04 SAVI (1 + 0.5) B08 + B04 + 0.5 The whole dataset currently consists of images captured from the end of June 2015 till August 2018. Images are avail- able for use in a few hours after being taken. Since working Table 1: Additional indices with data for the whole world is impractical, smaller geo- graphical regions are usually queried and analyzed on their For each selected image, all 13 Sentinel2-bands were consid- own. One important aspect when analyzing larger regions ered as feature vectors for each pixel, in the second experi- that must be taken care of is the fact that EO data is ac- ment, additional land cover based classification indices from quired in swaths with the width of approximately 290 km Table 1 were added. [3]. Since the swaths overlap a bit, care must be taken when sampling larger areas (in a size of small state), as the area 3.3 Experiment might be chopped into a few irregular tiles covering only The experiment was conducted in the Den Helder region part of an area of interest. to asses the effectiveness of classification and improvement with additional features. The same region is also used as a Corrected images are available using the SentinelHub li- test drive location for the PerceptiveSentintel project. One brary. PerceptiveSentinel platform provides an easy to use important characteristic to keep in mind is the fact that framework that combines satellite data acquisition, subse- classification classes are not uniformly distributed. Tulip quent cloud detection enables an easy way to pipeline ma- fields constitute 17.1% and 17.7% of all pixels in the year chine learning framework. They also provide an easy way 2016 and 2017 respectively, while arable land accounts for to integrate (vectorized or rasterized) geopedia layers as a 64% of pixels in 2017 data set. Care must, therefore, be source of ground truth for classification. taken when assessing the predictive power of a model. 2.3 Data Preprocessing For each dataset, multiple classification algorithms were tested Most of the preprocessing is already done by ESA (atmo- on base band feature vectors and feature vectors enriched spheric reflectance, projection . . . ). The data is mostly clean with calculated indices from Table 1. Experiments were and presented as a pixel array with dimensions H×W×B, carried out using python library scikit-learn and default where W and H are image dimensions (in our case 589 and parameters were used for each type of classifier. For each 590) and B is number of bands selected (in our case 13, but data set and each classifier (Ada Boost, Logistic regression, we may individually preconfigure the Sentinelhub library to Random Forest, Multilayer perceptron, Gradient Boosting, return variable number of bands and even custom calcula- Nearest neighbors and Naive Bayes), 3-fold cross-validation tions based on other bands). was performed. Folds were generated on a non-shuffled dataset with balanced classes ratios. When preprocessing we only need to consider one part of problematic data, namely clouded parts of images. ESA 4. RESULTS provides some sort of cloud detection, but our experiments Results of selected classifiers are presented in Tables 2–4 (ind proved it unsatisfactory, so we used the s2cloudless library column indicates additional indices as features) are compa- developed by Sinergise for this task [10]. rable with results from related works [5], [6] which report 186 accuracy results from 60-80%, although our experimental Alg. Ind Prec Rec Acc F1 T dataset was quite small and homogeneous, which might of- Logistic No 0.853 0.913 0.843 0.882 2.7 fer some advantage over larger plots of land. Regression Yes 0.854 0.908 0.841 0.880 3.2 Decision No 0.878 0.868 0.837 0.873 9.6 Alg. Ind Prec Rec Acc F1 T Tree Yes 0.885 0.868 0.842 0.876 14.5 Logistic No 0.895 0.551 0.912 0.682 2.8 Random No 0.928 0.889 0.884 0.908 17.3 Regression Yes 0.877 0.564 0.912 0.686 3.6 Forest Yes 0.934 0.891 0.889 0.912 26.3 Decision No 0.640 0.697 0.881 0.667 7.9 ML No 0.929 0.932 0.911 0.931 522.4 Tree Yes 0.629 0.698 0.878 0.662 11.3 Perceptron Yes 0.926 0.940 0.913 0.933 586.2 Random No 0.870 0.675 0.927 0.760 15.0 Gradient No 0.899 0.921 0.883 0.910 82.6 Forest Yes 0.867 0.680 0.927 0.762 21.7 Boosting Yes 0.905 0.926 0.890 0.915 118.4 ML No 0.875 0.720 0.935 0.790 184.2 Naive No 0.823 0.830 0.776 0.827 0.4 Perceptron Yes 0.835 0.740 0.931 0.784 241.3 Bayes Yes 0.814 0.806 0.757 0.810 0.6 Gradient No 0.878 0.657 0.926 0.751 84.8 Boosting Yes 0.856 0.657 0.923 0.743 120.6 Table 4: Arable land in 2017 results Naive No 0.343 0.800 0.704 0.480 0.4 Bayes Yes 0.316 0.808 0.669 0.454 0.6 Table 2: Tulip fields in 2016 results Alg. Ind Prec Rec Acc F1 T Logistic No 0.514 0.561 0.829 0.537 2.8 Regression Yes 0.545 0.615 0.841 0.578 4.0 Decision No 0.574 0.633 0.852 0.602 7.3 Tree Yes 0.565 0.634 0.849 0.598 11.2 Random No 0.786 0.599 0.900 0.680 13.8 Forest Yes 0.788 0.604 0.901 0.683 20.5 ML No 0.790 0.673 0.911 0.727 375.9 Perceptron Yes 0.780 0.693 0.911 0.734 419.8 Gradient No 0.786 0.613 0.902 0.689 84.4 Boosting Yes 0.785 0.614 0.902 0.689 120.3 Naive No 0.330 0.861 0.666 0.477 0.4 Bayes Yes 0.318 0.858 0.649 0.464 0.6 Table 3: Tulip fields in 2017 results For each test, precision, recall, accuracy, and F1 score were reported along with the timing of the whole process. As seen from the tables, multilayer perceptron achieved best results when comparing F Figure 2: Graphical representation of errors in ML 1 score across all data sets, but its training was considerably slower than all other classification perceptron classification of tulip fields in 2017 (TP methods (in fact, it’s training time was comparable to all in purple, FP in blue, FN in red) other classification times combined). Multilayer perceptron was followed closely by random forest, which achieved just marginally worse results, but was way less expensive to train seen, that classification produced quite satisfactory results. and evaluate, while still retaining score that was higher or An important thing to notice when inspecting Figure 2 is comparable with related works. that the true positive pixels were classified in densely packed groups with clear and sharp edges, which correspond nicely Adding additional features to feature vector did not signif- to field boundaries seen with the naked eye (both RF and icantly improve classification score and has in some cases Gradient boosting decision trees produced visually very sim- even hampered performance while having a significant im- ilar results). This might suggest that algorithms have de- pact on the training time. tected another culture similar to tulips and classified it as tulips (or conversely, that the ground truth might not be Using classifier trained on 2016 tulips data and predicting that accurate). A lot of FN pixels can also be spotted on data in 2017 yielded an F1 score of 0.57, while classifier field boundaries, which may correspond to different quality trained on 2017 data yielded an F1 score of 0.67 on 2016 or mixing of different plant cultures near field boundaries. data, indicating the robustness of the classifier. Similarly, observing results of arable land classification, one Graphical representation of classification errors can be seen immediately notices small (false positive) blue patches and in Figure 2 and 3 which show true positive (TP) pixels in some red patches. Most notably, a long blue line is spotted purple color, false positive (FP) in blue color and false neg- in the left part of the image (near the sea), which may in- ative (FN) in red. Looking at the images it can easily be dicate some sort of wild culture near the sea that was not 187 [2] ESA. Satellite constellation / Sentinel-2 / Copernicus / Observing the Earth / Our Activities / ESA. https://www.esa.int/Our_Activities/Observing_ the_Earth/Copernicus/Sentinel-2/Satellite_ constellation. Accessed 13 August 2018. [3] ESA. Sentinel-2 - Missions - Resolution and Swath - Sentinel Handbook. https://sentinel.esa.int/web/ sentinel/missions/sentinel-2/ instrument-payload/resolution-and-swath. Accessed 13 August 2018. [4] ESA. User Guides - Sentinel-2 MSI - Level-2 Processing - Sentinel Online. https: //sentinel.esa.int/web/sentinel/user-guides/ sentinel-2-msi/processing-levels/level-2. Accessed 13 August 2018. [5] Guida-Johnson, B., and Zuleta, G. A. Land-use land-cover change and ecosystem loss in the espinal ecoregion, argentina. Agriculture, Ecosystems & Environment 181 (2013), 31 – 40. [6] Gutiérrez-Vélez, V. H., and DeFries, R. Annual multi-resolution detection of land cover conversion to oil palm in the peruvian amazon. Remote Sensing of Environment 129 (2013), 154 – 167. Figure 3: Graphical representation of errors in ML [7] Gómez, C., White, J. C., and Wulder, M. A. perceptron classification of arable land in 2017 (TP Optical remotely sensed time series data for land cover in purple, FP in blue, FN in red) classification: A review. ISPRS Journal of Photogrammetry and Remote Sensing 116 (2016), 55 – 72. included in the original mask. Further manual observation [8] Jiang, Z., Huete, A. R., Didan, K., and Miura, of misclassified red patch in the middle of arable land sug- T. Development of a two-band enhanced vegetation gests that this field is barren (easily seen in Figure 2) and index without a blue band. Remote Sensing of possibly wrongly classified as arable land in training data. Environment 112, 10 (2008), 3833 – 3845. [9] Kenda, K., Kažič, B., Čerin, M., Koprivec, F., 5. CONCLUSIONS Bogataj, M., and Mladenić, D. D4.1 Stream Learning Baseline Document. Reported 30. April 2018. In our work, we have examined the use of different classifica- tion methods and additional features for land cover classifi- [10] Sinergise. sentinel-hub/sentinel2-cloud-detector: cation problem on a single image. Our results are compara- Sentinel Hub Cloud Detector for Sentinel-2 images in ble with results from the related literature. We propose that Python. https://github.com/sentinel-hub/ classification strength and adaptability be further improved sentinel2-cloud-detector. Accessed 14 August by considering time series and stream aggregates for each 2018. pixel as researched in [14] [7]. Additionally, pixels might be [11] Sinergise. sentinel-hub/sentinelhub-py: Download grouped together into logical objects to enable object (field) and process satellite imagery in Python scripts using level classification as proposed by [13]. Sentinel Hub services. https://github.com/sentinel-hub/sentinelhub-py. Furthermore, results have shown, that correct ground truth Accessed 14 August 2018. mask is essential for good classification performance. As [12] Spaceflight 101. Sentinel-2 Spacecraft Overview. seen from our results, even seemingly correct labels might http://spaceflight101.com/copernicus/ miss some cultures or classify empty straits of land as crops. wp-content/uploads/sites/35/2015/09/8723482_ orig-1024x538.jpg. Accessed 14 Aug. 2018. [13] 6. ACKNOWLEDGMENTS Valero, S., Morin, D., Inglada, J., Sepulcre, G., Arias, M., Hagolle, O., Dedieu, G., Bontemps, This work was supported by the Slovenian Research Agency S., Defourny, P., and Koetz, B. Production of a and the ICT program of the EC under project PerceptiveSen- dynamic cropland mask by processing remote sensing tinel (H2020-EO-776115). The authors would like to thank image series at high temporal and spatial resolutions. Sinergise for it’s contribution to sentinelhub and cloudless Remote Sensing 8(1) (2016), 55. library along with all help with data analysis. [14] Waldner, F., Canto, G. S., and Defourny, P. Automated annual cropland mapping using 7. REFERENCES knowledge-based temporal features. ISPRS Journal of [1] Photogrammetry and Remote Sensing 110 (2015), 1 – Belgiu, M., and Csillik, O. Sentinel-2 cropland mapping using pixel-based and object-based 13. time-weighted dynamic time warping analysis. Remote Sensing of Environment 204 (2018), 509 – 523. 188 Towards a semantic repository of data mining and machine learning datasets Ana Kostovska Sašo Džeroski Panče Panov Jožef Stefan IPS & Jožef Stefan Institute & Jožef Stefan Institute & Jožef Jožef Stefan Institute Jožef Stefan IPS Stefan IPS Ljubljana, Slovenia Ljubljana, Slovenija Ljubljana, Slovenia ana.kostovska@ijs.si saso.dzeroski@ijs.si pance.panov@ijs.si ABSTRACT The benefits of publishing FAIR data are manifold. It spe- With the exponential growth of data in all areas of our lives, eds up the process of knowledge discovery and reduces the there is an increasing need of developing new approaches for consumption of resources. When the FAIR-compliant data effective data management. Namely, in the field of Data Mi- at hand does not contain all the information needed it can be ning (DM) and Knowledge Discovery in Databases (KDD), easily integrated with data from external sources and boost scientists often invest a lot of time and resources for collec- the overall KDD performance [12]. ting data that has already been acquired. In that context, by publishing open and FAIR (Findable, Accessible, Interope- Semantic data annotation, being very powerful technique, rable, Reusable) data, researchers could reuse data that was is massively used in some domains, i.e. medicine, but it is previously collected, preprocessed and stored. Motivated by sill in the early phases in the domain of data mining and this, we conducted extensive review on current approaches, machine learning. To the best of our knowledge, there are data repositories and semantic technologies used for annota- no semantic data repositories that adhere to the FAIR prin- tion, storage and querying of datasets for the domain of ma- ciples. We recognize the ultimate benefits of having one and chine learning (ML) and data mining. Finally, we identify we are going in depths of the research covering semantic data the limitations of the existing repositories of datasets and annotation, ontology usage, storing and querying of data. propose a design of a semantic data repository that adheres to FAIR principles for data management and stewardship. 2. BACKGROUND AND RELATED WORK The Semantic Web (Web 3.0) is an extension of the World 1. INTRODUCTION Wide Web in which information is given semantic meaning, One of the main use of data is in the process of knowledge enabling machines to process that information. The aim of discovery, where scientist employ ML and DM methods and the Semantic Web initiative is to enhance web resources with try to solve various real-life problems from diverse fields, highly structured metadata, known as semantic annotations. from systems biology and medicine, to ecology and enviro- When one resource is semantically annotated, it becomes a nmental sciences. In order to obtain their objectives, they source of information that is easy to interpret, combine and need high-quality data. The quality of the data is crucial to a reuse by the computers [13]. In order to achieve this, the DM project’s success. Ultimately, no level of algorithmic so- Semantic Web uses the concept of Linked Data. Linked data phistication can make up for low-quality data. On the other is build upon standard web technologies [7] including HTTP, hand, progress in science is best achieved by reproducing, RDF, RDFS, URIs, Ontologies, etc. reusing and improving someone else’s work. Unfortunately, datasets are not easily obtained, and even if they are, they For uniquely identifying resources across the whole Linked come with limited reusability and interoprability. Data, each resource is given a Unified Resource Iden- tifier (URI). The resources are then enriched with terms A key-aspect in advancing research is making data open from controlled vocabularies, taxonomies, thesauruses, and and FAIR. FAIR are four principles that have been recen- ontologies. The standard metadata model used for logical tly introduced to support and promote good data manage- organization of data is called Resource Description Fra- ment and stewardship [17]. Data must be easily findable mework (RDF). Its basic unit of information is the triplet (Findability) by both humans and machines. This me- compiled from a subject, a predicate, and an object. These ans data should be semantically annotated with rich meta- three components define the concepts and relations, the bu- data and all the resources must be uniquely identified. The ilding blocks of an ontology. metadata should always be accessible (Accessibility) by standardized communication protocols such as HTTP(S) or In the context of computer science, ontology is “an expli- FTP, even when the data itself is not. Data and metadata cit formal specifications of the concepts and relations among from different data sources can be automatically combined them that can exist in a given domain” [3]. As computational (Interoperabiliy). To do so, the benefits of formal voca- artifacts, they provide the basis for sharing meaning both bularies and ontologies should be exploited. Data and me- at machine and human level. When creating an ontology, tadata is released with provenance details and data usage there are multiple languages to choose from. RDF Shema licence, so that humans and machines know whether data (RDFS) is ontology language with small expressive power. can be replicated and reused or not (Reusabiliy). It provides mechanisms for creating simple taxonomies of 189 concepts and relations. Another commonly used ontology There are numerous repositories of ML datasets available language is the Web Ontology Language (OWL). OWL online. The UCI repository3 is the most popular reposi- supports creation of all ontology components: concepts, in- tory of ML datasets. Each dataset is annotated with several stances, properties (or relations). Finally, SPARQL1 is descriptors such as dataset characteristics, attribute charac- standard, semantic query language used for querying fast- teristics, associated task, number of instances, number of growing private or public collections of structured data on attributes, missing values, area, etc. Similarly, Kaggle Da- the Web or data stored in RDF format. tasets4, Knowledge Extraction based on Evolutionary Le- arning (KEEL), and Penn Machine Learning Benchmarks There are diiferent technologies for storing data and meta- (PMLB)5 are well-known dataset repository that provide data. The most broadly used are relational databases, users with data querying based on the descriptors attached digital databases based on the relational model of data or- to the datasets. OpenML6 is an open source platform desi- ganized in tables, forming entity-relational model. Another gned with the purpose of easing the collaboration of resear- approach that became popular with the appearance of Big chers within the machine learning community [14]. Resear- Data are NoSQL databases [5], which are flexible databases chers can share datasets, workflows and experiments in such that do not use relational model. Triplestores are specific a way that they can be found and reused by others. When type of NoSQL databases, that store triples instead of rela- the data format of the datasets is supported by the platform, tional data. Triplestores use URIs and can be queried over the datasets are annotated with measurable characteristics trillions of records, which makes them very applicable. [15]. These annotations are saved as textual descriptors and are used for searching through the repository. Data in an information system can reside in different hete- rogeneous data sources, both internal and external to the In contrast to the above mentioned repositories, there are organization. In this setting, the relevant data from the frameworks in other domains that offer advanced techniques diverse sources should be integrated. Accessing disparate for describing, storing and querying datasets. One cutting- data sources has been a difficult challenge for data analysts edge framework in the domain of neuroscience is Neurosci- to achieve in modern information systems, and an active re- ence Information Framework(NIF) [4]. Its core objec- search area. OBDA [1, 11] is much longed-for method that tive is to create a semantic search engine that benefits from addresses this problem. It is a new paradigm, based on a semantic indexes when querying distributed resources by three-level architecture constituted of the ontology, the data keywords. The Gene Ontology Annotation (GOA), sources, and the mappings between the two (see Figure 1). is a database that provides high-quality annotations of ge- With this approach, OBDA provides data structure descrip- nome data [2]. The annotations are based on GO, a voca- tion, as well as semantic description of the concepts in the bulary that defines concepts related to gene functions and domain of interest and roles between them. relation among them. Large part of the annotations are ge- nerated electronically by converting existing knowledge from the data to GO terms. Electronic annotations are associated with high-level ontology terms. The process of generating more specific annotations can hardly be automated with the current technologies, therefore it is done manually. 3. CRITICAL ASSESSMENT Figure 1. The OBDA architecture In this section, we conduct critical assessment of the cur- rent research based on the review presented in the previous section. In the context of semantic ML data repository, we group ontologies in three categories, i.e., ontologies for describing Semantic Web technologies. The whole stack of seman- machine learning and data mining, ontologies for provenance tic technologies provide ways of making the content readable information, and domain ontologies. OntoDM ontology de- by machines. The metadata that describes the content can scribes the domain of data mining. It is composed of three be used not only to disregard useless information, but also sub-ontologies: OntoDT [10] - generic ontology for repre- for merging results to provide a more constructed answer. sentation of knowledge about datatypes; OntoDM-core [8] - A major drawback of this process of giving data a semantic ontology of core data mining entities (e.g., data, DM task, meaning is that it is time consuming and requires great amo- generalizations, DM algorithms, implementations of algori- unt of resources, thus people sometimes feel unmotivated to thms, DM software); OntoDM-KDD [9] - ontology for repre- do it. Another point to make is that semantic annotations senting the knowledge discovery process following CRISP- cannot solve the ambiguities of the real world. DM process model. The Data Mining OPtimization Ontology (DMOP) [6] has been designed to support au- Technologies for storing data and metadata. The tomation at various choice points of the DM process, i.e., data in relational databases is stored in a very structured choosing algorithms, models, parameters. The PROV On- way, making them a good choose for applications that relay tology (PROV-O)2 and Dublin Core vocabulary [16] 3 facilitate the discovery of electronic resources by providing a https://archive.ics.uci.edu/ml/ 4 base for describing provenance information about resources. https://www.kaggle.com/datasets 5https://github.com/EpistasisLab/ 1https://www.w3.org/TR/rdf-sparql-query/ penn-ml-benchmarks 2https://www.w3.org/TR/prov-o/ 6https://www.openml.org/ 190 on heavy data analysis. Moreover the referential integrity the approaches and technologies. Each of the proposed ar- guarantees that transactions are processed reliably. While chitectures has positive and negative sides, so there will be relational databases are a suitable choice for some applica- trade-off when choosing one. tions, they have difficulties dealing with large amounts of data. On the other hand, NoSQL databases were designed The common part of the three designs is that DM and ML primarily for big data and can be run on cluster architectu- datasets will be annotated through a semantic annotation res. Non-relational databases store unstructured data, with engine. The semantic query engine will receive SPARQL no logical schema. They are flexible, but this comes with query as input, and it will bring back results in form of set the price of potentially inconsistent data. of RDF triples. There will be SPARQL endpoint through which users can specify the query used as input in the se- Describing data and metadata. OntoDM is an ontology mantic query engine. Another open possibility is to enable that describes the domain of DM, ML and KDD with a great users to query data and metadata by simply writing key- level of detail. Because it covers a wide area, some parts words. Later, the system itself generates SPARQL query would be irrelevant for our application. DMOP is ontology based on those keywords. The anotation schema used by built with the special use case of optimizing the DM process. the semantic annotation engine will be based on three di- Nevertheless, both of them can be used for describing ML fferent types of ontologies such as ontologies for DM and and DM datasets. DC vocabulary and PROV-O define a ML (e.g., OntoDT, OntoDM-core, Onto-KDD, DMOP), do- wide range of provenance terms, therefore both of them can main ontologies, and ontologies and schemes for describing be employed in the provenance metadata generation. provenance information (e.g., Dublin Core ontology, PROV- O). Part of the annotations will be generated automatically, Repositories of machine learning datasets. The UCI e.g., annotations related to datatypes, while others will be repository offers a wide range of datasets, but they are not semi-automatically because they require concept mapping, available through a uniform format or API. Although it also e.g., annotations based on domain ontologies. provides data descriptors for searching the data, a major setback is that none of the descriptors is based on any vo- We plan to build a web-based user interface that will enable cabulary or ontology, which certainly limits interoperabi- users to search and query both datasets and metadata anno- lity. Kaggle Datasets, KEEL, PMLB also provide similar tations. Users will be given a chance of uploading new data- meta annotations, but they all lack semantic interpretabi- sets in CSV or ARFF fromat. Besides the dataset, users will lity. Another shortcoming of the UCI repository, KEEL and be expected to specify some additional information about it PMLB is that they don’t allow uploading new datasets. All such as data mining task they plan to execute on the data, datasets stored in the OpenML repository can be downlo- domain, provenance information, descriptions of the attri- aded in CSV or ARFF format. The annotation are based butes, etc. Since the whole process of semantic annotation on Exposé ontology, and they can be downloaded in JSON, can’t be automatic, when new dataset is uploaded, it won’t XML or RDF format. A major weakness of this repository be immediately available on the site. First it must be cura- is that annotations are not stored, but they are calculated ted, and only when the complete set of metadata annotati- on-the-fly and can not be used for semantic inference. ons is generated, the metadata will be published online. The dataset itself will be released under clear data usage licence. Frameworks for describing, storing and querying do- main datasets. The NIF framework is very progressive in The three architectural designs differ in the way of storing terms of semantic annotation, storing, and querying. Its ad- the datasets. The metadata annotations will be RDF tri- vantages come from providing domain experts with the abi- plets and they will be stored in triplestore that optimizes lity to contribute to the ontology development, by adding physical storage. Next, we briefly explain the differences new terms through the use of Interlex. It has a powerful between storing the datasets and what are the effects on search engine, and it follows the OBDA paradigm. Hetero- querying. geneous data is stored in its original format. The user defi- ned, keyword query is mapped to ontological terms to find Proposal I. The simplest approach of storing a dataset synonyms, and then translated to a query relevant to the in- would be to store it in RDF format in the same triplestore dividual data store. With respect to the genomics domain, as the metadata. The datasets from their original format, GOA database is favourable because of its high-quality an- will be converted to RDF triples. Having only one triplestore notations. Curators put extreme efforts in generating ma- will ease querying, but it will require more storage capacity nual annotations. To speed up the query execution it uses (see Figure 2). the Solr document store. Another superiority of GOA data- Proposal II. The second option is to store the datasets in base is that it provides advanced filtering of the annotations, a relational database and the metadata in RDF triplestore. for downloading customized annotation sets. The deficiency Datasets from CSV or ARFF format will be translated into of NIF and GOA database is that they are not able to query a relational database. Here, querying becomes more compli- and access the annotations in RDF format, which is an emer- cated, for which we will need a federated query engine. A ging standard for representing semantic information federated query engine allows simultaneous search on multi- ple data sources. A user makes a single query request, which 4. PROPOSAL FOR SEMANTIC is distributed across the management systems participating REPOSITORY OF DM/ML DATASETS in the federation and translated to a query written in a lan- guage relevant to the individual system. We will have two In this section, we propose three possible architecture desi- data stores, one for the data itself and one for the metadata. gns of the semantic data repository for the domain of ML For querying the two data stores, we will still use the same and DM. The proposals are based on the critical review of 191 querying of ML and DM datasets. We also examined speci- fic implementations of frameworks in the domain of neuro- science and genomics. Taking into consideration the critical assessment of the current state-of-the-art we will construct semantic data repository for ML and DM datasets. The semantic repository would be utilized for easy access of se- mantically rich annotated datasets and semantic inference. This, will improve the reproducibility and reusability in ML and DM research area. Moreover, annotating the datasets with domain ontologies will facilitate the process of under- standing the analyzed data. As of now, we have three pro- posed architectural designs for the semantic data repository Figure 2. Architectural design I that differ in the way of storing the datasets. We will either store both data and metadata in a triplestore, or we will have multiple data stores which will require usage of tools RDF query language, SPARQL. In order to query the rela- and methods from the ontology based data access paradigm. tional database with SPARQL, it will be mapped to virtual RDF graph (see Figure 3). Acknowledgements The authors would like to acknowledge the support of the Slovenian Research Agency through the projects J2-9230, N2-0056 and L2-7509 and the Public Scholarship, Development, Disability and Maintenance Fund of the Republic of Slovenia through its scholarship program. 6. REFERENCES [1] Mihaela A Bornea et al. Building an efficient rdf store over a relational database. In Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, pages 121–132. ACM, 2013. [2] Gene Ontology Consortium. Gene ontology consortium: going forward. Nucleic acids research, 43(D1):D1049–D1056, 2014. [3] Thomas R Gruber. Toward principles for the design of ontologies used for knowledge sharing? International journal of human-computer studies, 43(5-6):907–928, 1995. [4] Amarnath Gupta et al. Federated access to heterogeneous information resources in the neuroscience information framework (nif). Neuroinformatics, 6(3):205–217, 2008. [5] Jing Han et al. Survey on nosql database. In Pervasive Figure 3. Architectural design II computing and applications (ICPCA), 2011 6th international conference on, pages 363–366. IEEE, 2011. [6] C Maria Keet et al. The data mining optimization ontology. Web Semantics: Science, Services and Agents on the World Proposal III. Instead of mapping the relational database Wide Web, 32:43–53, 2015. to virtual RDF graph, we can use the OBDA methodology [7] Brian Matthews. Semantic web technologies. E-learning, 6(6):8, 2005. and federated querying to use a combination of SQL que- [8] Panče et al. Panov. Ontology of core data mining entities. Data ries and SPARQL queries. Metadata will be queried with Mining and Knowledge Discovery, 28(5-6):1222–1265, 2014. SPARQL queries, but for the datasets, they will be mapped [9] Panče Panov et al. Ontodm-kdd: ontology for representing the to SQL queries. The integrated results are brought back to knowledge discovery process. In International Conference on Discovery Science, pages 126–140. Springer, 2013. the user (see Figure 4). [10] Panče Panov et al. Generic ontology of datatypes. Information Sciences, 329:900–920, 2016. [11] Antonella Poggi et al. Linking data to ontologies. In Journal on data semantics X, pages 133–173. Springer, 2008. [12] Petar Ristoski and Heiko Paulheim. Semantic web in data mining and knowledge discovery: A comprehensive survey. Web semantics: science, services and agents on the World Wide Web, 36:1–22, 2016. [13] Gerd Stumme et al. Semantic web mining: State of the art and future directions. Web semantics: Science, services and agents on the world wide web, 4(2):124–143, 2006. [14] Jan N Van Rijn et al. Openml: A collaborative science platform. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 645–649. Springer, 2013. [15] Joaquin Vanschoren et al. Taking machine learning research online with openml. In Proceedings of the 4th International Conference on Big Data, Streams and Heterogeneous Source Figure 4. Architectural design III Mining, pages 1–4. JMLR. org, 2015. [16] Stuart Weibel. The dublin core: a simple content description model for electronic resources. Bulletin of the Association for 5. CONCLUSION Information Science and Technology, 24(1):9–11, 1997. [17] Mark D Wilkinson et al. The fair guiding principles for scientific We have conducted a literature overview of research be- data management and stewardship. Scientific data, 3, 2016. ing done in the field of semantic annotation, storage, and 192 Towards a semantic store of data mining models and experiments Ilin Tolovski Sašo Džeroski Panče Panov Jožef Stefan International Jožef Stefan Institute & Jožef Jožef Stefan Institute & Jožef Postgraduate School & Jožef Stefan International Stefan International Stefan Institute Postgraduate School Postgraduate School Ljubljana, Slovenia Ljubljana, Slovenia Ljubljana, Slovenia ilin.tolovski@ijs.si saso.dzeroski@ijs.si pance.panov@ijs.si ABSTRACT sible, Interoperable, Reusable) data principles, introduced Semantic annotation provides machine readable structure to by Wilkinson et al. [9]. Implementing these principles for the stored data. We can use this structure to perform seman- the annotation, storing, and querying of data mining models tic querying, based on explicitly and implicitly derived infor- and experiments will provide a solid ground for researchers mation. In this paper, we focus on the approaches in seman- interested in reproducing and reusing the results from the tic annotation, storage and querying in the context of data previous research on which they can build and improve. mining models and experiments. Having semantically anno- tated data mining models and experiments with terms from In the literature, there exist some approaches that address domain ontologies and vocabularies will enable researchers some of these problems. In both ontology engineering and to verify, reproduce, and reuse the produced artefacts and data mining community, there are approaches that aim to- with that improve the current research. Here, we first pro- wards describing the data mining domain, as described in vide an overview of state-of-the-art approaches in the area of Section 2. Furthermore, Vanschoren et al. [5] developed the semantic web, data mining domain ontologies and vocabu- OpenML system, a machine learning experiment database laries, experiment databases, representation of data mining for storing various segments of a machine learning experi- models and experiments, and annotation frameworks. Next, ment such as datasets, flows (algorithms), runs, and com- we critically discuss the presented state-of-the-art. Further- pleted tasks. more, we sketch our proposal for an ontology-based system for semantic annotation, storage, and querying of data min- In other domains, such as life sciences, storing annotated ing models and experiments. Finally, we conclude the paper data about experiments and their results is a common prac- with a summary and future work. tice. This is mostly due to the fact that the experiments are more expensive to conduct, and require specific prepara- tions. From the perspective of annotation frameworks, there 1. INTRODUCTION are significant advances in these domains, such as The Cen- Storing big amounts of data from a specific domain comes in ter for Expanded Data Annotation and Retrieval (CEDAR) hand with several challenges, one of them being to seman- workbench [8] , and the OpenTox framework [11]. tically represent and describe the stored data. Semantic representation enables us to infer new knowledge based on This paper is organized as follows. First, we make an overview the one that we assert, i.e. the description and annotation of the state-of-the-art approaches in annotating, storing, and of the data. This can be done by providing semantic annota- querying of models and experiments. Next, we critically as- tions of the data with terms originating from a vocabulary or sess these approaches and sketch a proposal for a system for ontology describing the domain at hand. In computer and annotating, storing and querying data mining models and information science, ontology is a technical term denoting experiments. Finally, we provide a summary and discuss an artifact that is designed for a purpose, which is to en- the possible approaches for further work. able the modeling of knowledge about some domain, real or imagined [15]. Ontologies provide more detailed description of a domain, first by organizing the classes into a taxonomy, 2. BACKGROUND AND RELATED WORK and further on by defining relations between classes. With The state-of-the-art in semantic annotation of data min- semantic annotation we attach meaning to the data, we can ing models and experiments provides very diverse research, infer new knowledge, and perform queries on the data. ranging from domain-specific data mining ontologies, exper- iment databases, to new languages for deploying annotations Data mining and machine learning experiments are con- in unified format. Here, we provide an introduction to the ducted with faster pace than ever before, in various settings state-of-the-art in semantic web, ontologies and vocabular- and domains. In the usual practice of conducting data min- ies, representations of data mining models and experiments, ing experiments, almost none of the settings are recorded, experiment databases, and annotation frameworks. nor the produced models are stored. These predicaments make for a research that is hard to verify, reproduce and up- Semantic technologies. The Semantic Web is defined grade. This is also in line with the FAIR (Findable, Acces- as an extension of the current web in which information is 193 given well-defined meaning, enabling computers and people for (semi) automatically or manually annotating data, there to work in cooperation [14]. The stack of technologies con- are several solutions that exist outside of the data min- sists of multiple layers, however, in this paper we will focus ing domain, which provide innovative approaches and good on the ones essential for our scope of research. Resource foundation for development in the direction of creating a Description Framework (RDF) represents a metadata data software to enable ontology-based semantic annotation of model for the Semantic Web, where the core unit of informa- models and experiments, their storage and querying. The tion is presented as a triple. A triple describes the subject by CEDAR Workbench [13] provides an intuitive interface for its relationship, which is what the predicate resembles, with creating templates and metadata annotation with concepts the object. RDF files are stored in triple store (typically or- defined in the ontologies available at BioPortal4. On the ganized as relational or NoSQL databases [12]), on which we other hand, OpenTox [11] represents domain specific frame- can perform semantic queries, by using querying languages work that provides unified representation of the predictive such as SPARQL. Finally, ontology languages, such as Re- modelling in the domain of toxicology. source Description Framework Schema (RDFS) and Ontol- ogy Web Language (OWL), are formal languages used to 3. CRITICAL ASSESSMENT construct ontologies. RDFS provides the basis for all ontol- ogy languages, defining basic constructs and relations, while In this section, we will critically assess the presented state- OWL is far more expressive enabling us to define classes, of-the-art in Section 2 in the context of semantic annota- properties, and instances. tion, storage and querying of data mining models and ex- periments. Ontologies & vocabularies. Currently, there are several ontologies that describe the data mining domain. These The state-of-the-art in ontology design for data mining pro- include the OntoDM ontology [16], DMOP ontology [7], Ex- vides well documented research with various ontologies that pose [4], KDDOnto [1], and KD ontology [10]. MEX [2] is an thoroughly describe the domain from different aspects and interoperable vocabulary for annotating data mining mod- can be used in various applications. OntoDM provides uni- els and experiments with metadata. In addition there have fied framework of top level data mining entities. Building been developments in formalism for representing scientific on this, it describes the domain in great detail, containing experiments in general, such as the EXPO ontology [6]. definitions for each part of the data mining process. Because of the wide reach, it lacks a particular use case scenario. On Representation of models. With the constant devel- the other hand, this same property makes this ontology suit- opment of new environments for developing data mining able for wide range of applications where there is a need of software, it is necessary to have a unified representation describing a part of the domain. of the constructed data mining models and the conducted experiments. The first open standard was the Predictive Ontologies like EXPO and Exposé have a essential meaning Model Markup Language (PMML). For a period of time it in the research since the first one describes a very wide and provided transparent and intuitive representation of data important interdisciplinary domain, while the latter uses it mining models and experiments. However, due to the as a base for defining a specific sub-domain. DMOP ontol- fast growth in the development of new data mining meth- ogy describes the process of algorithm and model selection in ods, PMML was unable to follow the pace and extend its the context of semantic meta mining. Both the KD ontology more and more complicated specification. Its successor, the and KDDOnto describe the knowledge discovery process in Portable Format for Analytics (PFA), was developed having the context of constructing knowledge discovery workflows. the PMML’s drawbacks as guidelines for improvement. They differ mainly in the key concepts on which they were built. At the same time, the MEX vocabulary provides a Experiment and model databases. Storing already con- lightweight framework for automating the metadata gener- ducted experiments in a well structured and transparent ation. Since it is tied with Java environment, it provides manner is essential for researchers to have available, veri- a library which only uses the MEX API and can also be fiable, and reproducible results. An experiment database is implemented in other programming languages. designed to store large number of experiments, with detailed information on their environmental setup, the datasets, algo- All in all, the current state of the art in ontologies for data rithms and their parameter settings, evaluation procedure, mining provides a good foundation for development of ap- and the obtained results [3]. The state-of-the-art in storing plications which will be based on one or several of these setups and results is abundant with approaches and solu- ontologies. Given the wide of coverage they can be easily be tions in different domains. For example, OpenML1 is the combined in a manner to suit the application at hand. biggest machine learning repository of data mining datasets, tasks, flows, and runs, the BioModels2 repository stores In the area of descriptive languages for data mining models more than 8000 experiments and models from the domains and experiments, one can see the path of progress in re- of systems biology, and ModelDB3 is an online repository search. PMML was the first, ground-breaking, XML-based for storing computational neuroscience models. descriptive language. However, with the expansion of the data mining domain, several weaknesses of PMML emerged. Annotation frameworks. When it comes to frameworks The language was not extensible, users could not create chains of models, and it was not compatible with the dis- tributed data processing platforms. Therefore, the same 1https://www.openml.org/ community started working on a new, extensible, portable 2http://www.ebi.ac.uk/biomodels/ 3https://senselab.med.yale.edu/modeldb/ 4https://bioportal.bioontology.org/ 194 language. Since its inception, the PFA format was intended need to have complete information about the conditions in to fill the small gaps that PMML had. Made up of analytic which that experiment was conducted. Namely, we need to primitives, implemented in Python and Scala, it provides the have an annotated dataset, annotation of the algorithm and users with more customizable framework, where they can its parameter settings for the specific run of the experiment. create custom models, model chains, and implement them Since one experiment usually consists of multiple algorithm in a parallelized setting. runs we annotate each run separately, as well as each of the results from each of them. For annotating the results, we use Storing and annotating experiments is of great significance the definitions of the performance metrics formalized in the in multiple scenarios. First, in domains where conducting data mining ontologies. A sketched example of the proposed the experiment is not a trivial task, i.e. the physical or solution is shown in Figure 1. financial conditions challenge the process, there needs to be a database where the setup and the findings of the experiment The proposed system for ontology-based annotation, stor- will be saved. For example, in BioModels.net we have two age, and querying of data mining experiments and models groups of experiments: Manually curated with structured will consist of several components. The users will interact metadata, and experiments without structure. The main with the system through an user interface enabling them drawback with this type of storage is the need for manual to run experiments on a data mining software, which will curation of the metadata. It is repetitive, time consuming export models and experiment setups to a semantic anno- task for which there is a strong need to be automated. tation engine. For example, for testing purposes we plan to use CLUS5 software for predictive clustering and structured In the domain of neuroscience, ModelDB provides an online output prediction, which generates different types of models service for storing and searching computational neuroscience and addresses different data mining tasks. models. In this database, alongside the files that constitute the models, researchers also need to upload the code that In the semantic annotation engine, the data mining mod- defines the complete specification of the attributes of the els and experiments will be annotated with terms from the biological system represented in the model, together with extended OntoDM ontology and then stored in a database. files that describe the purpose and application of the model. Once stored, the users will be able to semantically query Therefore, researchers can search the database for models the models and experiments in order to infer new knowl- with specific applications describing biological systems. edge. This will be done through a querying engine based on the SPARQL language, accessible through a user interface. OpenML provides a good framework for storing and anno- tating data mining datasets, experimental setups and runs, In order to perform annotation, we will extend the exist- as well as algorithms. One particular drawback of OpenML ing OntoDM ontology by adding a number of new terms, is that it does not store the actual models that are produced linking it to other domain ontologies, such as Exposé and from each experimental run, and one can not query the mod- EXPO. Linking OntoDM to these ontologies will extend the els. Furthermore, it’s founded on relational-database which domain of OntoDM towards connecting the data mining en- can not provide execution of semantic queries. tities that it already covers with new entities that describe the experimental setup and principles. With this we will All in all, these three examples show significant advances in obtain a schema for annotation of data mining models and storing and annotating models and experiments. However, experiments. The schema will then be used to annotate the there is also a significant room for improvement in the di- data mining models and experiments through a semantic an- rection of storing the models and experiments into NoSQL notation engine. The engine will have to read the models databases that are better suited for this task. and experiments from a data mining software system, anno- tate them with terms from developed schema and produce Finally, in the context of annotation tools the CEDAR Work- an RDF representation of the annotated data. bench and the OpenTox Framework provide a good insight in annotation frameworks. CEDAR enables the user to ex- Furthermore, the RDF graphs will be stored in a triple store ecute the annotations in modular manner by creating tem- database. Since the data mining models and experiments plates and adding elements to them. After curating the differ a lot in their structure, we have yet to decide on the annotations, they can export the schemas either in JSON, type of database in which we will store them. The data JSON-LD, or RDF file. OpenTox [11] is also based on on- stored in this way is set for performing semantic queries tology terms and represents a complete framework that de- on top of it. Therefore, we will develop a SPARQL-based scribes the predictive process in toxicology, starting with querying enigne so the users can perform predefined or cus- toxicity structures and ending with the predictive modelling. tom semantic queries on top of the storage base. Finally, the format of the results is another point where we 4. A PROPOSAL FOR SEMANTIC STORE need to decide whether the results will be presented as RDF OF MODELS AND EXPERIMENTS graphs, or in a different format (such as JSON) that is easier to interpret. This software package along with the storage After analysing the previous and current research, we can will then be added as a module to the CLUS software, de- conclude that despite the great achievements, there is a wide veloped at the Department of Knowledge Technologies. area for improvement in which we will contribute in the up- coming period by developing an ontology-based framework for storage and annotation of data mining models and exper- iments. In order to annotate a data mining experiment, we 5http://sourceforge.net/projects/clus 195 Annotation Schema Semantically RDF Triples Domain Extends Annotated OntoDM Storage of DM Ontology 2 Experiment Ontology experiments SPARQL Query SPARQL Query Domain Extends Storage of Querying Ontology 1 DM models RDF Triples Engine Semantically Experiment Annotated Semantic Model Annotation User defined query Model Engine Runs experiments Results Data Mining User interface Software Figure 1. Schema of the proposed solution 5. CONCLUSION & FURTHER WORK and the Public Scholarship, Development, Disability and Maintenance In this paper, we presented the state-of-the-art in annota- Fund of the Republic of Slovenia through its scholarship program. tion, storage and querying in the light of designing a se- mantic store of data mining models and experiments. We 6. REFERENCES first gave an overview of semantic web technologies, such as [1] Claudia Diamantini et al. KDDOnto: An ontology for discovery RDF, SPARQL, RDFS, and OWL that provide a complete and composition of kdd algorithms. Towards Service-Oriented Knowledge Discovery (SoKD’09), pages 13–24, 2009. foundation for annotation and querying of data. [2] Diego Esteves et al. MEX Vocabulary: a lightweight interchange format for machine learning experiments. In Furthermore, we critically reviewed the state-of-the-art on- Proceedings of the 11th International Conference on tologies and vocabularies for describing the domain of data Semantic Systems, pages 169–176. ACM, 2015. [3] Hendrick Blockheel et al. Experiment databases: Towards an mining provide detailed description of the domain of data improved experimental methodology in machine learning. In mining and machine learning (OntoDM, Expose, KD On- European Conference on Principles of Data Mining and tology, DMOP and KDDOnto, MEX). Next, we focused on Knowledge Discovery, pages 6–17. Springer, 2007. experiment databases as repositories where the experiment [4] Joaqin Vanschoren et al. Exposé: An ontology for data mining experiments. In Towards service-oriented knowledge discovery datasets, setups, algorithm parameter settings, and the re- (SoKD-2010), pages 31–46, 2010. sults are available for the performed experiments in various [5] Joaqin Vanschoren et al. Taking machine learning research domains. Furthermore, we saw that annotation frameworks online with OpenML. In Proceedings of the 4th International provide environments for (semi) automatically or manually Conference on Big Data, Streams and Heterogeneous Source Mining, pages 1–4. JMLR. org, 2015. annotating data, by discussing two frameworks from the do- [6] Larisa N Soldatova et al. An ontology of scientific experiments. mains of biomedicine and toxicology in order to analyze best Journal of the Royal Society Interface, 3(11):795–803, 2006. practices present in those domains. [7] Maria C Keet et al. The Data Mining OPtimization Ontology. Web Semantics: Science, Services and Agents on the World Wide Web, 32:43–53, 2015. Finally, given the performed analysis of the state-of-the-art, [8] Mark A Musen et al. The Center for Expanded Data we outlined our proposal for an ontology-based framework Annotation and Retrieval. Journal of the American Medical for annotation, storage, and querying of data mining mod- Informatics Association, 22:1148–1152, 2015. els and experiments. The proposed framework consists of an [9] Mark D. Wilkinson et al. The FAIR guiding principles for scientific data management and stewardship. Scientific Data, annotation schema, a semantic annotation engine, and stor- 3, 2016. age for data mining models and experiments with a querying [10] Monika Záková et al. Automating knowledge discovery engine, all of which will be controlled from an user interface. workflow composition through ontology-based planning. IEEE Transactions on Automation Science and Engineering, It will allow users to semantically query their data mining 8:253–264, 2011. models and experiments in order to infer new knowledge. [11] Olga Tcheremenskaia et al. OpenTox predictive toxicology framework: toxicological ontology and semantic media In the future, we plan to adapt this framework for the needs wiki-based openToxipedia. In Journal of biomedical semantics, page S7, 2012. of research groups or companies that conduct high volume of [12] Olivier Curé et al. RDF database systems: triples storage and data mining experiments, enabling them to obtain a queryable SPARQL query processing. Morgan Kaufmann, 2014. knowledge base consisting of annotated metadadata for all [13] Rafael S Gonçalves et al. The CEDAR Workbench: An experiments and produced models. This will enable them Ontology-Assisted Environment for Authoring Metadata that Describe Scientific Experiments. In International Semantic to reuse existing models on new data for testing purposes, Web Conference, pages 103–110. Springer, 2017. infer knowledge based on past experimental results, all while [14] Tim Berners-Lee et al. The semantic web. Scientific American, saving time and computational resources. 284:34–43, 2001. [15] Tom Gruber. Ontology. Encyclopedia of database systems, pages 1963–1965, 2009. Acknowledgements [16] Panče Panov. A Modular Ontology of Data Mining. PhD The authors would like to acknowledge the support of the Slovenian thesis, Jožef Stefan IPS, Ljubljana, Slovenia, 2012. Research Agency through the projects J2-9230, N2-0056 and L2-7509 196 A Graph-based prediction model with applications [Extended Abstract] ∗ András London József Németh Miklós Krész University of Szeged, Institute University of Szeged, Institute InnoRenew CoE of Informatics of Informatics University of Primorska, IAM Poznan University of University of Szeged, Institute Economics, Department of of Applied Sciences Operations Research ABSTRACT and later it appeared in many areas from social network We present a new model for probabilistic forecasting using analysis to optimization in technical networks (e.g. road graph-based rating method. We provide a “forward-looking” and electric networks) [16]. type graph-based approach and apply it to predict football game outcomes by simply using the historical game results Making predictions in general, and especially in sports as data of the investigated competition. The assumption of our well, is a difficult task. The predictions generally appear in model is that the rating of the teams after a game day cor- the form of betting odds, that, in the case of “fixed odds”, rectly reflects the actual relative performance of them. We provide a fairly acceptable source of expert’s predictions re- consider that the smaller the changing of the rating vector – garding sport games outcomes [21]. Thanks to the increasing contains the ratings of each team – after a certain outcome quantity of available data the statistical ranking, rating and in an upcoming single game, the higher the probability of prediction methods have become more dominant in sports that outcome. Performing experiments on European foot- in the last decade. A key question is that how accurate ball championships data, we can observe that the model per- these evaluations are, more concretely, the outcomes of the forms well in general and outperforms some of the advanced upcoming games how accurately can be predicted based on versions of the widely-used Bradley-Terry model in many the statistics, ratings and forecasting models in hand. cases in terms of predictive accuracy. Although the appli- cation we present here is special, we note that our method Statistics-based forecasting models are used to predict the can be applied to forecast general graph processes. outcome of games based on some relevant information of the competing teams and/or players of the teams. A detailed Categories and Subject Descriptors survey of the scientific literature of rating and forecasting I.6 [Simulation and Modeling]: Applications; I.2 [Artificial methods in sports is beyond the scope of this paper, we Intelligence]: Learning refer only some important and recent results in the topic. For some papers with detailed literature overview and sport applications of the the celebrated Bradley-Terry model [3], 1. INTRODUCTION see e.g. [5, 7, 24]). Other popular approach is the Poisson The problem of assigning scores to a set of individuals based goal-distribution based analysis. For some references, see on their pairwise comparisons appears in many areas and ac- for instance [10, 15, 20]. In these models the goals scored tivities. For example in sports, players or teams are ranked by the playing teams follow a Poisson distribution with pa- according to the outcomes of games that they played; the rameter that is a function of attack and defense “rate” of impact of scientific publications can be measured using the the respective teams. A large family of prediction models relations among their citations. Web search engines rank only consider the game results win, loss (and tie) and usu- websites based on their hyperlink structure. The centrality ally uses some probit regression model, for instance [11] and of individuals in social systems can also be evaluated accord- [13]. More recently, well-known data mining techniques, like ing to their social relations. Ranking of individuals based artificial neural networks, decision trees and support vector on the underlying graph that models their bilateral relations machines have also become very popular; some references - has become the central ingredient of Google’s search engine without being exhaustive - see e.g [8, 9, 14, 18].Based on ∗Corresponding author, email: london@inf.u-szeged.hu the huge literature it can be concluded that the prediction accuracy strongly depends on the investigated sport and the feature set of the machine learning algorithms used. A no- table part of prediction models based on the historical data of game results use the methodology of ranking and rat- ing. Some recent articles in the topic are e.g. [2, 6, 12, 17, 23]. Specifically highlighting [2] the authors analyzed the he predictive power of eight sports ranking methods using only win-loss and score difference data of American major sports. They found that the least squares and random walker meth- 197 ods have significantly better predictive accuracy than other to a successful bettor is less than that represented by the methods. Moreover, utilizing score-differential data are usu- true chance of the event occurring. This means mathemat- ally more predictive than those using only win-loss data. ically that 1/odds(i) + 1/odds(j) is more than one. This profit expected by the agency is known as the “over-round In contrast to those techniques that use the actual respective on the book”. strength of the two competing teams, we provide a graph- based and forward-looking type approach. The assumption 2.2 The Bradley-Terry Model of our model is that if a rating of the teams after a game day The Bradley-Terry model [3] is a widely-used method to as- correctly reflects the actual relative performance, then the sign probabilities to the possible outcomes when a set of smaller the change in that rating after a certain result occurs n individuals are repeatedly compared with each other in (in an upcoming single game) the higher the probability of pairs. For two elements i and j, the probability that i beats that event occur. j defined as The structure of this paper is follows. After presenting πi Pr(i beats j) = , the classical approaches (“Betting Odds” and “The Bradley- πi + πj Terry Model”), our new model is introduced. Then in Sec. 3 where πi > 0 is a parameter associated to each individual i = we present our preliminary experimental results, and finally 1, . . . , n, representing the overall skill, or “intrinsic strength” in Sec. 4 we conclude and discuss some possible research of it. Equivalently, πi/πj represents the odds in favor i beats directions. j, therefore this is a “proportional-odds model”. Suppose that i and j played Nij games against each other with i 2. MODELS winning Wij of them, and all games are considered to be Let V = (1, . . . , n) be the set of n teams (or players) and independent. The likelihood is given by let R be the number of game days in a competition among W N the teams in V . A rating is a function φr : V → n ij ij −Wij R that Y πi πj L(πi, . . . , πn) = . assigns a score to each team after each game day r (r = πi + πj πi + πj i Sj ) = Pr(Si − Sj > 0) = 1 − 1 + elogπi−logπj if j wins, then the bettor loses his $1. We can calculate the πi probabilities of the respective events as = . πi + πj 1/odds(i) Pr(i beats j) = 1/odds(i) + 1/odds(j) Extension with Home advantage and Tie. A natu- and ral extension of the Bradley-Terry model with “home-field 1/odds(j) advantage”, according to [1], say, is to calculate the proba- Pr(j beats i) = . 1/odds(i) + 1/odds(j) bilities as ( θπ We should note here that odds provided by betting agen- i , if i is at home Pr(i beats j) = θπi+πj cies do not represent the true chances (as imagined by the πi , if j is at home πi+θπj bookmaker) that the event will or will not occur, but are the amount that the bookmaker will pay out on a winning bet. where θ > 0 measures the strength of the home-field advan- The odds include a profit margin meaning that the payout tage (or disadvantage). Considering also a tie as a possible 198 final result of a game, the following calculations, proposed node home-i with weight x and an edge from node home- in [22], can be used : i to node away-j with weight y are added to the graph, π respectively. Our assumption is that if an outcome x : y i Pr(i beats j) = , has a high probability and it occurs, then it causes a small πi + απj change in the PageRank vector; hence δxy will be small. To simplify the notations let {δ1, . . . , δm} be the distance val- (α2 − 1)πiπj Pr(i ties j) = ues obtained by considering different results {E1, . . . , Em} (πi + απj )(απi + πj ) of the upcoming game between i and j. The goal now is where α > 1. Combining them is straightforward. In our to calculate the probability that a certain result occurs if experiments, we used the Matlab implementations found at {δ1, . . . , δm} is given. To do this, we use the following sim- ple statistics-based machine learning method. Let f +() be http://www.stats.ox.ac.uk/~caron/code/bayesbt/ using the expectation maximization algorithm, described in detail the probability density function of δi random variable where in [7]. the event (game result) Ei occurred. In our implementa- tion Ei ∈ {0 : 0, 1 : 0, 1 : 1, . . . , 5 : 5}, assuming that the probability of other results equals 0. Similarly, let f −() be 2.3 Rating-based Model with Learning the probability density function of δi random variable in Our new model is designed as follows. We will use the term which case the event (game result) Ei did not occur. To “game day” in each case when at least one match is played approximate the f +() and f −() functions, for each game on the given day. For any game day in which we make we use the training data set contains all results and related a forecast, we consider the results matrix that contains all δi (i = 1, . . . , m) values of the preceding T = 40 game days the results of the previous T = 40 game days. For the 40 of the considered game. In our experiments, the gamma dis- game days time window, the entries of the results matrix S tribution (and its density function) turned out to be a fairly are defined as Sij = #{scores team home-i achieved against good approximate for f +(δ) and f −(δ). team away-j}. To take into account the home-field effect, for each team i we distinguish team home-i and team away-i. Assuming that δ1, . . . , δm are independent, using the Bayes Thus, we define a 2n × 2n results matrix, which, in fact, theorem and the law of total probability, we can calculate describes a bipartite graph where each team appears both that in the home team side and the away team side of the graph. For rating the teams, a time-dependent PageRank method f +(δi) Q f −(δ k6=i k ) Pr(Ei|{δ1, . . . , δm}) = . is used. The PageRank scores are calculated according the P f +(δ f −(δ ` `) Qk6=l `) time-dependent PageRank equation We should note here that in this way we assign probabilities λ φ = Π = [I − (1 − λ)St to concrete game final results, which is another novelty of N mod(l1t)−1]−11, (1) our model. Then, for the upcoming game between i and j, defined in [19]. The damping factor is λ = 0.1, while we may the outcome probability of the event “i beats j” is calculated multiply each entry of S with the exponential function 0.98α as to consider time-dependency and obtaining S X mod, where α Pr(i beats j) = Pr(Ek|{δ1, . . . , δm}), denotes the number of game days elapsed since a given result k: Ek encodes a result occurred (and stored in S). Note, that a home team and an of team-i win away team PageRank values are calculated for each team. where we sum over those Ek results for which i beats j (i.e. We would like to establish a connection between team home- 1:0, 2:0, 2:1, 3:0, 3:1, etc.). The probabilities Pr(i ties j) i and team away-i using the assumption that home-i is not and Pr(j beats i) can be calculated in a similar way. weaker than away-i. In our implementation we assumed that home-i had a win 2 : 1 against away-i to give a positive bias for home-i at the beginning. In our experiments this setup 3. EXPERIMENTAL RESULTS performed well, but it was not optimized precisely. To measure the accuracy of the forecasting we calculate the mean squared error, which is often called Brier scoring rule Using the above-defined results matrix S and the PageR- in the forecasting literature [4]. The Brier score measures the ank rating vector φ, we assign probabilities to the outcomes mean squared difference between the predicted probability {home team win, tie, away team win} of an upcoming game assigned to the possible outcomes for event E and the actual in game day r between home-i and away-j as follows. Be- outcome oE. Suppose that for a single game g, between i and fore the game day in which we make the forecast, let the j, the forecast is pg = (pgw, pg, pg) contains the probabilities t l calculated PageRank rating vector be φr−1(V ). We use δr of i wins, the game is a tie and i loses, respectively. Let 40 xy to measure how the rating vector of the teams changes if the actual outcome of the game be og = (ogw, og, og), where t l the result of an upcoming game between teams i and j exactly one element is 1, the other two are 0. Noting that is x : y, where x, y = 0, 1, . . . are the scores achieved by the number of games played (and predicted) is N , BS is team i and team j, respectively1. We define δrxy as the Eu- defined as clidean distance between φr−1(V ) and φr 40 40(V ) that is the N rating vector for the new results matrix obtained by adding 1 X BS = ||pg − og||22 x to S N ij and y to Sn+j,i. In the results graph interpreta- g=1 tion this simply means that an edge from node away-j to N 1 X 1We should note here that if the result is 0 : 0, then x = = [(pg − og)2 + (pg − og)2]. N w − og w )2 + (pg t t l l y = 1/2 is used. g=1 199 The best score achievable is 0. In the case of three pos- 6. REFERENCES sible outcomes (win, lost, tie) we can easily see that the [1] A. Agresti. Categorical data analysis. John Wiley & forecast pg = (1/3, 1/3, 1/3) (for each game g and any N ) Sons, New York, 1996. gives accuracy BS = 2/3 = 0.666. We consider this value [2] D. Barrow, I. Drayer, P. Elliott, G. Gaut, and B. Osting. Ranking rankings: an empirical comparison as a worst-case benchmark. One question of our investiga- of the predictive power of sports ranking methods. tion is that how better BS values can be achieved using our Journal of Quantitative Analysis in Sports, method, and how close we can get to the betting agencies’ 9(2):187–202, 2013. fairly good predictions. [3] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired The data set we used contained all final results of given comparisons. Biometrika, 39(3-4):324–345, 1952. seasons of some football leagues, listed in the first two col- [4] G. W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, umn of Table 1. We tested our method as it was described 78(1):1–3, 1950. in Sec. 2.3. We start predicting games starting from the [5] K. Butler and J. T. Whelan. The existence of 41th game day; for each single game predictions are made maximum likelihood estimates in the Bradley-Terry using the results of the previous 40 game day before that model and its extensions. arXiv preprint game. The Brier scores were calculated using all predic- math/0412232, 2004. tions we made. Our initial results are summarized in Ta- [6] T. Callaghan, P. J. Mucha, and M. A. Porter. ble 1. To calculate the betting odds probabilities we used Random walker ranking for NCAA division IA football. American Mathematical Monthly, the betting odds provided by bet365 bookmaker available 114(9):761–777, 2007. at http://www.football-data.co.uk/. We could see that [7] F. Caron and A. Doucet. Efficient bayesian inference these predictions gave the best accuracy score (BS) in each for generalized Bradley–Terry models. Journal of case. We highlighted the values where the difference between Computational and Graphical Statistics, the Bradley-Terry method and the PageRank method was 21(1):174–196, 2012. higher than 0.01. Although we can see that slightly more [8] A. C. Constantinou, N. E. Fenton, and M. Neil. than half of the cases the Bradley-Terry model gives a better Pi-football: A bayesian network model for forecasting accuracy, the results are still promising considering the fact association football match outcomes. Knowledge-Based Systems, 36:322–339, 2012. that the parameters of our method and the implementation [9] D. Delen, D. Cogdell, and N. Kasap. A comparative are far from being optimized. analysis of data mining methods in predicting NCAA bowl outcomes. International Journal of Forecasting, 28(2):543–552, 2012. 4. CONCLUSIONS [10] M. J. Dixon and P. F. Pope. The value of statistical We presented a new model for probabilistic forecasting in forecasts in the UK association football betting sports, based on rating methods, that simply use the histor- market. International Journal of Forecasting, 20(4):697–711, 2004. ical game results data of the given sport competition. We [11] D. Forrest, J. Goddard, and R. Simmons. Odds-setters provided a forward-looking type graph based approach. The as forecasters: The case of English football. assumption of our model is that the rating of the teams after International Journal of Forecasting, 21(3):551–564, a game day is correctly reflects their current relative perfor- 2005. mance. We consider that the smaller the changing in the [12] R. Gill and J. Keating. Assessing methods for college rating vector after a certain result occurs in an upcoming football rankings. Journal of Quantitative Analysis in single game, the higher the probability that this event will Sports, 5(2), 2009. occur. Performing experiments on results data sets of Eu- [13] J. Goddard and I. Asimakopoulos. Forecasting football results and the efficiency of fixed-odds betting. ropean football championships, we observed that this model Journal of Forecasting, 23(1):51–66, 2004. performed well in general in terms of predictive accuracy. [14] A. Joseph, N. E. Fenton, and M. Neil. Predicting However, we should note here, that parameter fine tuning football results using bayesian nets and other machine and optimizing certain parts of our implementation are tasks learning techniques. Knowledge-Based Systems, of future work. 19(7):544–553, 2006. [15] D. Karlis and I. Ntzoufras. Analysis of sports data by We emphasize, that our methodology can be also useful to using bivariate Poisson models. Journal of the Royal compare different rating methods by measuring that which Statistical Society: Series D (The Statistician), 52(3):381–393, 2003. one reflects better the actual strength (rating) of the teams [16] A. N. Langville and C. D. Meyer. Google’s PageRank according to our interpretation. Finally we should add that and beyond: The science of search engine rankings. the model is general and may be used to investigate such Princeton University Press, 2011. graph processes where the number of nodes is fixed and edges [17] J. Lasek, Z. Szlávik, and S. Bhulai. The predictive are changing over time; moreover it also has a potential to power of ranking systems in association football. link prediction. International Journal of Applied Pattern Recognition, 1(1):27–46, 2013. [18] C. K. Leung and K. W. Joseph. Sports data mining: 5. ACKNOWLEDGMENTS Predicting results for the college football games. Procedia Computer Science, 35:710–719, 2014. This work was partially supported by the National Research, [19] A. London, J. Németh, and T. Németh. Development and Innovation Office - NKFIH, SNN117879. Time-dependent network algorithm for ranking in sports. Acta Cybernetica, 21(3):495–506, 2014. Miklós Krész acknowledges the European Commission for [20] M. J. Maher. Modelling association football scores. funding the InnoRenew CoE project (Grant Agreement #739574) Statistica Neerlandica, 36(3):109–118, 1982. under the Horizon2020 Widespread-Teaming program. 200 Table 1: Accuracy results on football data sets. The values where the difference between the Bradley-Terry method and the PageRank method was higher than 0.01 are shown in bold. League Season Betting odds error Bradley-Terry error PageRank method error 2011/12 0.58934 0.60864 0.59653 Premier League 2012/13 0.56461 0.59744 0.58166 2013/14 0.54191 0.55572 0.59406 2014/15 0.55740 0.60126 0.60966 2011/12 0.58945 0.59994 0.59097 Bundesliga 2012/13 0.57448 0.59794 0.58622 2013/14 0.55724 0.57803 0.60125 2014/15 0.57268 0.60349 0.60604 2011/12 0.54598 0.57837 0.58736 La Liga 2012/13 0.56417 0.58916 0.60205 2013/14 0.57908 0.58016 0.60473 2014/15 0.52317 0.55888 0.56172 [21] P. F. Pope and D. A. Peel. Information, prices and efficiency in a fixed-odds betting market. Economica, pages 323–341, 1989. [22] P. Rao and L. L. Kupper. Ties in paired-comparison experiments: A generalization of the Bradley-Terry model. Journal of the American Statistical Association, 62(317):194–204, 1967. [23] J. A. Trono. Rating/ranking systems, post-season bowl games, and ’the spread’. Journal of Quantitative Analysis in Sports, 6(3), 2010. [24] C. Wang and M. L. Vandebroek. A model based ranking system for soccer teams. Research report, available at SSRN 2273471, 2013. 201 202 Zbornik 21. mednarodne multikonference INFORMACIJSKA DRUŽBA – IS 2018 Zvezek D Proceedings of the 21st International Multiconference INFORMATION SOCIETY – IS 2018 Volume D Mednarodna konferenca o visokozmogljivi optimizaciji v industriji, HPOI 2018 International Conference on High-Performance Optimization in Industry, HPOI 2018 Uredila / Edited by Bogdan Filipič, Thomas Bartz-Beielstein http://is.ijs.si 8. oktober 2018 / 8 October 2018 Ljubljana, Slovenia 203 204 PREDGOVOR Z optimizacijskimi problemi se v realnem svetu, zlasti pa v industriji, srečujemo vsakodnevno. Visokozmogljiva optimizacija temelji na združevanju računske moči in naprednih optimizacijskih algoritmov in se je pojavila kot odgovor na izzive, ki jih predstavljajo zahtevni optimizacijski problemi, ki so lahko visokodimenzionalni, multimodalni, šumni, dinamični, večkriterijski ali pa njihovo reševanje vključuje časovno zahtevne simulacije. Mednarodna konferenca o visokozmogljivi optimizaciji v industriji ( High-Performance Optimization in Industry, HPOI 2018) je mišljena kot forum za predstavitev primerov uporabe in izmenjavo izkušenj med akademskimi in industrijskimi partnerji o uvajanju visokozmogljive optimizacije. Poleg tega spodbuja nadaljnje širjenje metodologije in neposredno sodelovanje med akademskimi ustanovami in industrijo. Konferenca je aktivnost projekta Synergy for Smart Multiobjective Optimization (SYNERGY, http://synergy-twinning.eu) iz programa Twinning v Obzorju 2020. Eden od ciljev tega projekta je prenesti znanje, ki so ga pridobili partnerji v konzorciju, na druge raziskovalne ustanove in v industrijo, zlasti podjetja, ki sodelujejo v Slovenski strategiji pametne specializacije (S4). Pri doseganju tega cilja so člani projekta že predstavili svoje dosežke v visokozmogljivi optimizaciji na specializirani delavnici na Gospodarski zbornici Slovenije, nekatere pa predstavljajo tudi na tej konferenci. Program konference obsega 11 predstavitev, vsi prispevki pa so objavljeni v konferenčnem zborniku. Prispevalo jih je 21 (so)avtorjev, od katerih je večina sodelavcev projekta SYNERGY. Obravnavane teme vključujejo optimizacijsko metodologijo, pristope k premoščanju vrzeli med akademskimi ustanovami in industrijo ter študije primerov s področij transporta, avtomobilske industrije, inženirstva in proizvodnje. Zahvaljujemo se avtorjem za oddajo in predstavitve njihovih del, članom programskega odbora za ocenjevanje prispevkov, Institutu »Jožef Stefan« kot gostitelju srečanja in organizatorjem 21. Mednarodne multikonference Informacijska družba (IS 2018), katere del je tudi HPOI 2018, za organizacijsko podporo. Bogdan Filipič, Thomas Bartz-Beielstein 205 FOREWORD Optimization problems are met in the real world, and particularly in industry, on a daily basis. High-performance optimization (HPO) is founded on the coupling of high computing power and advanced optimization algorithms, and has emerged in response to the challenges posed by hard optimization problems that can be high-dimensional, multimodal, noisy, dynamic, multiobjective or involve time-consuming simulations in order to be solved. The International Conference on High-Performance Optimization in Industry (HPOI 2018) is meant as a forum for presenting use cases and exchanging experience among academic and industrial partners on deploying HPO. Apart from that, it stimulates further proliferation of the methodology and direct collaboration between academia and industry. The conference is an activity of the Horizon 2020 Twinning project “Synergy for Smart Multiobjective Optimization” (SYNERGY, http://synergy-twinning.eu). One of the objectives of this project is to spread the knowledge gained by the consortium partners to other research institutions and the industry, in particular to the companies participating in the Slovenian Smart Specialization Strategy (S4). Pursuing this goal, the project members have already presented their achievements in HPO at a specialized workshop at the Chamber of Commerce and Industry of Slovenia, and some of them are also being presented at this conference. The conference program consists of 11 presentations and the related papers are published in the proceedings. They were contributed by 21 (co)authors, most of them being the SYNERGY project members. The topics discussed include the optimization methodology, approaches to bridging the gap between academia and industry, and case studies from the domains of transportation, automotive industry, engineering, and manufacturing. We are grateful to the authors for submitting and presenting their work, the program committee members for reviewing the papers, the Jožef Stefan Institute for hosting the event, and the staff of the 21st International Multiconference on Information Society (IS 2018) that HPOI 2018 is part of for organizational support. Bogdan Filipič, Thomas Bartz-Beielstein 206 PROGRAMSKI ODBOR / PROGRAMME COMMITTEE Bogdan Filipič (chair) Thomas Bartz-Beielstein (chair) Erik Dovgan Peter Korošec Nouredine Melab Marjan Mernik Boris Naujoks Gregor Papa Jörg Stork El-Ghazali Talbi Tea Tušar Martin Zaefferer Jernej Zupančič This event has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 692286. 207 208 On Using Real-World Problems for Benchmarking Multiobjective Optimization Algorithms Tea Tušar Department of Intelligent Systems Jožef Stefan Institute Ljubljana, Slovenia tea.tusar@ijs.si ABSTRACT ment. The artificial test problems that are being consis- Although the motivation to study multiobjective optimiza- tently used for benchmarking EMO algorithms have charac- tion algorithms comes from practice, there are only a few teristics that are not representative of real-world problems. challenging real-world problems freely available to the re- They also fail to incorporate the peculiarities of real-world search community. Because of this, algorithm benchmarking problems, which means that the algorithms need additional is performed primarily on artificial test problems. The most adjustments before they can be applied to real-world prob- popular artificial test problems have characteristics that are lems [8]. Furthermore, most studies do not investigate the not well-represented in real-world problems. This and the influence of the problem dimension on the performance of predominant inadequate performance assessment methodol- the algorithms and the performance assessment is often done ogy widen the gap between theory and practice in the field only at a predefined number of evaluations. This makes it of multiobjective optimization. The paper suggests to in- hard to predict which algorithm will perform best on a par- stead compare the algorithms with the anytime performance ticular real-world problem when less evaluations are allowed benchmarking approach of COCO (the Comparing Continu- than the (high) numbers usually used in the studies. ous Optimizers platform) on more realistic artificial problem suites as well as suites with diverse real-world problems. By The COCO platform [2, 10] resolves many of these issues listing the benefits of sharing the real-world problems with by providing an alternative to the overused test suites and a the community, the paper hopes to encourage domain ex- more rigorous approach to algorithm benchmarking. How- perts to embrace this practice. ever, in order to bridge the gap between theory and prac- tice, multiobjective optimization algorithms should be stud- Keywords ied and compared not only on well-understood and easy-to- multiobjective optimization, real-world problems, algorithm compute artificial functions, but also on real-world problems benchmarking with various characteristics. Currently, only a small num- ber of challenging real-world problems are freely available 1. INTRODUCTION to the EMO community, which hinders the development of algorithms that could be used ‘off the shelf’. Most real-world optimization problems found in science and engineering are inherently multiobjective. For example, the The purpose of this paper is to show the advantages of task of many engineering design problems is to find solutions benchmarking algorithms on real-world problems and to en- of high quality and low cost. Such problems seldom have a courage domain experts to share their hardest problems with single solution (called the ideal solution) that would opti- the researchers to their mutual benefit. mize all objective simultaneously. Rather, they have (possi- bly infinitely) many Pareto-optimal solutions that represent In the remainder of the paper, we first recall the purpose of different trade-offs among the objectives. These solutions algorithm benchmarking (Section 2). Then, we review the form the so-called Pareto set in the decision space and Pareto existing practice of benchmarking multiobjective optimiza- front in the objective space. tion algorithms on artificial test problems and remind of an available alternative in the form of the COCO platform (Sec- Evolutionary Multiobjective Optimization (EMO) [4] is one tion 3). Next, we mention some real-world problems that of the most active research areas that deal with multiobjec- have been made publicly available, discuss the benefits of tive problems. It studies algorithms that make no assump- sharing real-world problems and give recommendations for tions on the properties of the optimization problems, such proposing new real-world problems and performing bench- as linearity, continuity and unimodality, and are therefore marking with them (Section 4). We conclude with some applicable to a variety of problems, including black-box op- closing remarks (Section 5). timization ones. EMO algorithms have successfully solved numerous challenging real-world optimization problems [3]. 2. THE PURPOSE OF ALGORITHM Nevertheless, there is a large gap between theory and prac- BENCHMARKING tice in the EMO field (stemming from the one in Evolution- The no free lunch theorem implies that no optimization algo- ary Computation [18]), which is widened by the dominat- rithm performs best for all possible problems [22]. The ob- ing (inadequate) paradigm of algorithm performance assess- served differences in performance are due to the (more/less) 209 successful adaptation of the algorithms to the problem land- ample the C-DTLZ test suite [15], there is no established scapes [12]. It is therefore crucial that the test problems used test suite containing mixed-integer problems with multiple in comparison studies have characteristics that are represen- objectives. tative of real-world problems. Furthermore, although the problems from the mentioned Algorithm benchmarking, either when comparing variants suites are scalable in the number of variables (the problem of the same algorithm or a novel algorithm to an established dimension) and the number of objectives, performance stud- one, can be used to gain an understanding of the algorithms ies rarely investigate the scaling of the algorithms with the at hand. However, the ultimate purpose of algorithm bench- problem dimension. This is usually simply fixed to a value marking is to find the algorithm that is expected to perform (often 30), while the number of objectives is being changed. best for a specific target problem—a real-world problem of Such an approach to performance assessment is problematic interest. This entails that we have as it disregards one of the most defining characteristics of a problem—its dimension. (a) some knowledge about the characteristics of the target problem, Finally, most studies compare the performance of the al- (b) information on the performance of a number of algo- gorithms only at a specific point in time, determined by rithms on test problems with similar characteristics as the number of function evaluations. Because they provide those of the target problem, and no data on the performance of the algorithms prior to that (c) an understanding of what best is, i.e., we can define moment, the findings of such studies cannot be used to in- and measure the desired algorithm performance. fer algorithm performance when less evaluations are avail- able, making them effectively useless for the main purpose Then, machine learning methods can be used to select the of benchmarking mentioned earlier. most appropriate algorithm for the given target problem [16]. 3.2 Benchmarking with the COCO Platform 3. USING ARTIFICIAL PROBLEMS FOR COCO (Comparing Continuous Optimizers) [2, 10] is an ALGORITHM BENCHMARKING open-source platform for benchmarking black-box optimiza- Benchmarking multiobjective algorithms on artificial opti- tion algorithms. It implements different test problem suites mization problems has several advantages. The evaluations and provides an anytime performance assessment method- are cheap (computed instantaneously), the characteristics ology that is in line with the purpose of benchmarking as of the problems can be controlled, and the problems can be described in Section 2. Furthermore, COCO incorporates implemented in any programming language. If constructed the results of various optimization algorithms on its tests with care, the artificial problems can be scaled in the num- suites that are regularly being collected at BBOB (Black- ber of decision variables, constraints and objectives, and the Box Optimization Benchmarking) workshops [1] and can be Pareto sets and fronts can be known, which considerably readily used for comparisons with new algorithms. facilitates performance assessment. In addition to singleobjective test suites, such as the estab- The main question when using artificial test problems for lished bbob suite [11], COCO currently provides two test benchmarking algorithms is whether they are good repre- suites with biobjective problems, bbob-biobj with 55 func- sentatives of real-world problems. tions and its extended version bbob-biobj-ext with 92 func- tions [21], each instantiated in six dimensions (n ∈ {2, 3, 5, 10, 20, 40}) and ten instances (small alterations of the func- 3.1 Issues with the Prevailing Benchmarking tion, such as shifts, etc.). Every biobjective function is con- Methodology structed using two separate bbob functions—one for each Since the introduction of the DTLZ [6] and WFG [13] test objective. This approach is motivated by the nature of real- suites in 2001 and 2006, respectively, the vast majority of world multiobjective problems, where each objective corre- studies in EMO have been comparing algorithms on one or sponds to a separate singleobjective function. It is there- both of these two suites. In fact, they have been overused fore closer to real-world conditions than the constructions to such a degree that we can speculate on overfitting of op- with distance and position variables used by the DTLZ and timization algorithms to these problems. This is especially WFG test suites. However, this approach results in unknown concerning because they have some properties that are ben- Pareto sets and fronts, which is not convenient for perfor- eficial when designing test suites, but are not likely to be mance assessment purposes. In order to alleviate this issue, found in real-world problems. For example, in order have COCO provides approximations of the Pareto fronts for all a known Pareto set and a controllable shape of the Pareto problems, collected during several runs of various EMO al- front, the problems are parameterized by two sorts of vari- gorithms. These can be used in plots to showcase the charac- ables: distance variables, which indicate the distance of a so- teristics of the Pareto fronts and to compute the best known lution from the Pareto front, and position variables, which hypervolume [23] values for these problems. indicate the position of a solution along the Pareto front. The resulting Pareto sets and fronts are much easier to work The anytime performance assessment approach from COCO with than the irregularly shaped real-world ones. is based on the notion of runtime, i.e., the number of function evaluations needed to achieve a target hypervolume (see [9] Many real-world problems have additional difficulties, such and [21] for more details). This makes it possible to study as constraints or a mixed-integer decision space. While there the results for each problem separately as well as aggregate are some multiobjective test suites with constraints, for ex- them over all problems in a suite. For example, the plot 210 1.0 bbob-biobj f1-f55, 5-D HMO-CMA-ES Loshc tion three that are of different nature, but are very demand- 58 targets: 1..-1.0e-4 best 2016 10 instances ing and therefore suitable for algorithm benchmarking: UP-MO-CMA-ES Kra 0.8 RM-MEDA Auger bb • The Radar Waveform problem has an integer decision DEMO Tusar bbob- SMS-EMOA-SA Wess space that can be scaled from four to 12 decision vari- 0.6 NSGA-II-MATLAB A ables, and nine objectives [14]. SMS-EMOA-DE Auge SMS-EMOA-PM Auge • The HBV Benchmark Problem consists of calibrating RANDOMSEARCH-5 A the HBV rainfall–runoff model [19]. It has 14 real- 0.4 MO-DIRECT-HV-Ran valued decision variables and four objectives. RANDOMSEARCH-4 A MO-DIRECT-ND Won • The recently proposed Mazda Benchmark Problem [17] 0.2 MO-DIRECT-Rank W is a car structure design optimization problem with RANDOMSEARCH-100 222 integer decision variables, two objectives and 54 Fraction of function,target pairs MAT-DIRECT Al-Du constraint functions that make it hard to find a feasible 0.0 v2.2.1.417, hv-hash=ff0e71e8cd978373 MAT-SMS Al-Dujai 0 2 4 6 solution. log10(# f-evals / dimension) There are multiple reasons why only a few black-box real- world problems are being publicly shared. Sometimes, the Figure 1: Bootstrapped empirical cumulative dis- companies that have such problems hide them to protect tribution of the number of objective function their trade secrets. Other times, the reasons are of an im- evaluations divided by dimension for 58 targets plementation nature, for example because some proprietary with target precision in {100, 10−0.1, . . . , 10−4.9, 10−5, 0, software is needed to perform the evaluations. It is also pos- −10−5, −10−4.8, . . . , −10−4.2, −10−4} for 16 algorithms sible that people do not make their problems public simply on all 5-D functions of the bbob-biobj test suite. because they see no benefit in doing so. Most of these issues can be amended. If the domain ex- in Figure 1 shows the proportion of targets (on the y axis) perts wish to keep the details of the problem hidden, this that an algorithm is expected to achieve given the number can be achieved by sharing an executable program without of function evaluations (divided by the problem dimension, the source code. If the companies fear that their competitors on the x axis). The plot presents the results aggregated over could retrieve useful information already from how the prob- all instances of the 5-D functions of the bbob-biobj suite. lem is defined, a simple linear transformation can be used Note that such plots allow to compare the performance of to transform a box-constrained continuous decision space to algorithms that were run using a different budget of function [0, 1]n without affecting the nature of the problem landscape evaluations (up to the minimal common budget). (an integer or mixed-integer decision space can be handled in a similar way). Although the least noteworthy, some im- The COCO platform could similarly be used to benchmark plementation issues can be hardest to bypass. The best way real-world problems. might be to use freely available software instead of the pro- prietary one (this, of course, might not always be possible). 4. USING REAL-WORLD PROBLEMS FOR If conceivable, time-consuming evaluations using specialized ALGORITHM BENCHMARKING software can be replaced by surrogate models as was done, for example, in [17]. 4.1 Availability of Real-World Problems Real-world problems can be separated into those whose ob- jectives and constraints can be given in an analytic form and 4.2 Benefits of Sharing Real-World Problems others that are truly black-box problems, for example those Suppose a real-world problem is interfaced with the COCO that require complex computations or simulations to evalu- platform and used in the BBOB workshops to benchmark ate the functions and constraints of the problem. Note that multiobjective algorithms. This means that the researchers as soon as one function or constraint behaves like a black not only run their algorithms on the problem, but also sub- box, the entire problem is considered to be a black box. mit their results to COCO for use in future comparisons. The first and most obvious benefit of such a setting is that There are quite a few multiobjective real-world problems the interested EMO community would most likely find bet- of the first type, i.e., with a known analytic form. See for ter solutions to the problem in question than a single team example the problems from [5], [7] and [20]. Similarly to of researchers. Next, if the problem has some characteris- the artificial problems, they can be evaluated quickly and tics that are not well-represented in artificial test problems, implemented in any programming language. However, as such as a mixed-integer decision space, sharing such a prob- recently shown in [20], many such problems are not chal- lem will motivate the researchers to adapt their algorithms lenging enough to distinguish between algorithms and can to its characteristics. This means that in time, there will therefore be useful for benchmarking purposes only in test be more versatile algorithms for these kinds of problems to suites containing other, harder problems. choose from. Finally, it is likely that in the future, the same experts who shared this problem, will face another problem On the other hand, there are also many black-box real-world of similar nature. Then, the algorithms that performed best problems from various domains, but only a few of them are on the original problem might be readily used on the future freely available to EMO researchers. Here, we briefly men- alternative versions of this problem. 211 4.3 Recommendations [9] N. Hansen, A. Auger, D. Brockhoff, D. Tušar, and When proposing real-world benchmark problems, domain T. Tušar. COCO: Performance assessment. ArXiv experts should try to make them as flexible as possible. Ide- e-prints, arXiv:1605.03560, 2016. ally, it should be possible to instantiate them in a few differ- [10] N. Hansen, A. Auger, O. Mersmann, T. Tušar, and ent dimensions and also to create some instances of the same D. Brockhoff. COCO: A platform for comparing problem (minor modifications that do not change the nature continuous optimizers in a black-box setting. ArXiv of the problems). In addition to providing better grounds e-prints, arXiv:1603.08785, 2016. for performance assessment, this might also help to better [11] N. Hansen, S. Finck, R. Ros, and A. Auger. understand the problems in question. Real-parameter black-box optimization benchmarking 2009: Noiseless functions definitions. Research Report When benchmarking EMO algorithms, artificial test suites RR-6829, INRIA, 2009. with properties reflective of the real-world problems should [12] Y. Ho and D. Pepyne. Simple explanation of the be used in order to gain understanding about the algorithms. no-free-lunch theorem and its implications. Journal of In addition, the algorithms should also be tested on real- Optimization Theory and Applications, world problems to show their applicability in practice. Since 115(3):549–570, 2002. real-world problems come from various domains and might [13] S. Huband, P. Hingston, L. Barone, and R. L. While. have particular characteristics, the algorithms should be run A review of multiobjective test problems and a on suites of real-world problems from different domains. scalable test problem toolkit. IEEE Transactions on Evolutionary Computation, 10(5):477–506, 2006. 5. CONCLUSIONS [14] E. J. Hughes. Radar waveform optimisation as a This paper reviewed the many drawbacks of the existing many-objective application benchmark. In Proceedings practice of benchmarking multiobjective algorithms with the of EMO 2007, pages 700–714, 2007. over-used DTLZ and WFG test suites. Using the COCO [15] H. Jain and K. Deb. An evolutionary many-objective platform most can be amended, but the performance assess- optimization algorithm using reference-point based ment is still being done solely on artificial problem functions. nondominated sorting approach, part II: Handling The paper proposes to benchmark algorithms using COCO’s constraints and extending to an adaptive approach. anytime performance assessment on suites of real-world al- IEEE Transactions on Evolutionary Computation, gorithms in addition to the artificial ones. Some benefits of 18(4):602–622, 2014. sharing real-world problems with the EMO community are [16] P. Kerschke and H. Trautmann. Automated algorithm presented in hope to encourage greater exchange of knowl- selection on continuous black-box problems by edge between academia and industry. combining exploratory landscape analysis and machine learning. ArXiv e-prints, arXiv:1711.08921, 2017. 6. ACKNOWLEDGMENTS [17] T. Kohira, H. Kemmotsu, A. Oyama, and The author acknowledges the financial support from the T. Tatsukawa. Proposal of benchmark problem based Slovenian Research Agency (project No. Z2-8177). on real-world car structure design optimization. In Companion Proceedings of GECCO 2018, pages 7. REFERENCES 183–184, 2018. [1] The BBOB workshop series. [18] Z. Michalewicz. Quo vadis, evolutionary computation? https://numbbo.github.io/workshops/index.html, On a growing gap between theory and practice. In 2018. Proceedings of WCCI 2012, pages 98–121, 2012. [2] COCO: Comparing Continuous Optimizers. [19] P. M. Reed, D. Hadka, J. D. Herman, J. R. Kasprzyk, https://github.com/numbbo/coco, 2018. and J. B. Kollat. Evolutionary multiobjective [3] C. A. C. Coello and G. B. Lamont, editors. optimization in water resources: The past, present, Applications of Multi-Objective Evolutionary and future. Advances in Water Resources, 51:438–456, Algorithms. World Scientific, Singapore, 2004. 2013. [4] K. Deb. Multi-Objective Optimization Using [20] R. Tanabe and A. Oyama. A note on constrained Evolutionary Algorithms. John Wiley & Sons, multi-objective optimization benchmark problems. In Chichester, UK, 2001. Proceedings of CEC 2017, pages 1127–1134, 2017. [5] K. Deb and A. Srinivasan. Innovization: Innovative [21] T. Tušar, D. Brockhoff, N. Hansen, and A. Auger. design principles through optimization. KanGAL COCO: The bi-objective black-box optimization report number 2005007, Indian Institute of benchmarking (bbob-biobj) test suite. ArXiv e-prints, Technology, Kanpur, 2005. arXiv:1604.00359, 2016. [6] K. Deb, L. Thiele, M. Laumanns, and E. Zitzler. [22] D. H. Wolpert and W. G. Macready. No free lunch Scalable test problems for evolutionary multi-objective theorems for optimization. IEEE Transactions on optimization. KanGAL report number 2001001, Indian Evolutionary Computation, 1(1):67–82, 1997. Institute of Technology, Kanpur, 2001. [23] E. Zitzler, L. Thiele, M. Laumanns, C. M. Fonseca, [7] EMO’2017 Real-World Problems, Black Box and V. Grunert da Fonseca. Performance assessment Optimization Competition. of multiobjective optimizers: An analysis and review. https://bbcomp.ini.rub.de/, 2017. IEEE Transactions on Evolutionary Computation, [8] B. Filipič and T. Tušar. Challenges of applying 7(2):117–132, 2003. optimization methodology in industry. In Companion Proceedings of GECCO 2013, pages 1103–1104, 2013. 212 Bridging Theory and Practice Through Modular Graphical User Interfaces Frederik Rehbach Jörg Stork Thomas Bartz-Beielstein TH Köln TH Köln TH Köln Institute for Data Science, Institute for Data Science, Institute for Data Science, Engineering and Analytics Engineering and Analytics Engineering and Analytics Steinmüllerallee 1 Steinmüllerallee 1 Steinmüllerallee 1 51643 Gummersbach, 51643 Gummersbach, 51643 Gummersbach, Germany Germany Germany frederik.rehbach@th-koeln.de joerg.stork@th-koeln.de thomas.bartz-beielstein@th- koeln.de ABSTRACT rion (e.g. best model function value) is evaluated on the State-of-the-art evolutionary algorithms and related search expensive objective function and further used to update the heuristics are well suited to solve problems from industry. model. The process is repeated in an iterative fashion. A Unfortunately, easy to use graphical user interfaces (GUI) more in-depth explanation of SMBO and its applications can are not available for many algorithms. We claim that the be found in [5] and [2]. availability of well-designed GUIs might increase the accep- tance of these algorithms in the real-world domain. The SPOT has been further improved and developed for many spotGUI R-package, which is introduced in this paper, pro- years. Today the package provides a vast set of different vides a GUI for the already well-established SPOT package. models, optimizers, and sampling schemes, each of which It includes state-of-the-art algorithms and modeling tech- can be configured to user specific requirements. The system niques that can be used without the requirement of opti- was initially targeted to parameter optimization tasks, but mization or programming knowledge. Using the spotGUI is well suited to any costly to evaluate optimization prob- in industry, as well as education, delivered first promising lem. The availability of these methods together with their results. respective documentation in the R-package is a first step towards an easy to use modular optimization tool. How- Keywords ever, SPOT remains a high-level toolbox, which requires SPOT, Graphical User Interface, Real-World Applications user experience and some R programming skills. Further- more, since R is rarely used by engineers in industry, this 1. INTRODUCTION again leads to problems (a) and (b) as previously discussed. The presented spotGUI tries to address these problems by Industrial problems are highly complex and challenging for making the tools included in SPOT accessible to everyone even the most advanced state-of-the-art algorithms. How- through an easy to use graphical interface. ever, the difficulty in solving such problems is often not their high complexity, but rather the challenge for a non-expert The rest of this paper is structured as follows: Section 2 user to apply a suitable algorithm. For a significant subset gives an overview of the basic functionality and some con- of the existing optimization problems in industry, suitable ceptual ideas of the spotGUI. In Section3 two practical ex- state-of-the-art algorithms already exist. Yet, they are often ample applications for the spotGUI applied in industry are still not applied because they are presented. One of which is the Electrostatic Precipitator (ESP) Problem, a current, costly-to-evaluate, discrete opti- a) not known to the field specialist or mization problem from industry. Lastly, the software, future b) no simple implementation is available. opportunities, and room for improvements are discussed in Section 4. This paper presents a simple to use GUI that bridges the gap between existing algorithms and real-world problems. 2. WORKFLOW The core of the new package relies on the Sequential Param- eter Optimization Toolbox (SPOT) [1]. SPOT provides a 2.1 Availability modular structure for combining sampling methods, mod- The spotGUI package shall give more users easy access to eling techniques and optimizers for an all-in-one Surrogate SPOT. All stable versions are available on CRAN. Develop- Model-Based Optimization (SMBO) toolbox. In SMBO, a ment versions are published on GitHub. One of the primary data-driven surrogate model is fitted to the data of an ex- goals of the spotGUI is to allow non-R-users and even non- pensive to evaluate objective function, e.g., a complex sim- programmers to use SPOTs model-based optimization tech- ulation or a real-world experiment. Under the assumption niques. Additionally, it can benefit experienced SPOT users that the surrogate is cheap to evaluate, an extensive search by enabling a faster setup and even code generation which on the model becomes feasible. The predicted candidate will be covered in more detail in Section 2.5. The spotGUI solution, which best fulfills some user-specified infill crite- is developed in the R extension Shiny [4]. It is divided into 213 I: Setup II: Parameters III: Experiment IV: Save Objective Function Spot Config Run Spot Exports Variables and Model and Run, Evaluate R-Code Dimensions Optimizer Setup and Visualize Log Figure 1: Typical optimization workflow for SMBO in the spotGUI four separate tabs, arranged in a typical workflow order as presented in Figure 1 and Algorithm 2.1. Each of the tabs is explained in more detail in the following. Algorithm 2.1: Surrogate Model-based Optimization 1 step I: setup 2 select and parametrize objective function 3 begin Figure 2: Screenshot illustrating the objective func- 4 step II: parameters tion setup in the spotGUI. The user has to define the 5 select and parametrize surrogate model function as well as it’s dimensionality and variable 6 select and parametrize experimental design types. 7 step III: experiment 8 generate design points 9 evaluate design points with objective function some real-world experiments by entering / importing the 10 build initial surrogate model experiment results back into the spotGUI. The only con- 11 while not termination-condition do figuration required in this scenario is to insert information 12 search for optimum on surrogate model on the problem dimensions. Each dimension is configured 13 evaluate new point on the objective function with a type (numeric/integer/factorial), as well as upper 14 update surrogate model and lower bounds. If there are multiple dimensions with the 15 end same upper and lower bounds, the convenience option ”am- 16 step IV: save ntDimensions” can be used to specify that the same bounds 17 end are required multiple times. 2.2 Setup 2.3 Parameters The objective function is specified and parametrized on the One of the main benefits of the spotGUI becomes evident first setup tab. A screenshot of the configuration window is during the setup of SPOT itself. As previously mentioned shown in Figure 2. Additionally to having an option to insert SPOT features a wide variety of different models and op- any function through the R-Environment and supporting timizers, each of which again provides a variety of config- manual result input, the spotGUI provides a broad set of uration options. In the spotGUI, these are conveniently preconfigured test functions. selectable from drop-down menus. Showing each available option together with simple explanations through tooltips, The set of provided test functions is loaded from the ’smoof’ tackles the requirement of any documentation reading for R-package [3], which provides an interface to many single- the user. The settings are arranged in four categories cov- and also multi-objective test functions. Of these, the spot- ering a general setup, modeling setup, optimizer setup and GUI only includes the current set of single-objective func- lastly design setup. Skipping the ’Spot Config’ tab alto- tions, totaling in 76 test functions. Each of these functions gether results in a robust default setup for SPOT. is loaded with its respective bounds as well as dimensional- ity. Scalable functions are loaded as 2-dimensional functions 2.4 Experiment and can then be adapted by the user to any desired dimen- The previously configured processes are executed in the ”Run sionality. The ’smoof’ package also allows the user to filter Spot” tab. The available options include creating a DOE, the functions by specific tags such as ”separable”, ”differen- fitting a model, running a model-based optimization, and tiable” or ”weak-global-structure”. This makes it possible to more. In the following, these methodologies will be briefly test a given optimizer on a particular type of test function explained. In many expensive real-world applications, an that should behave somewhat similar to a real-world prob- initial screening for variable importance and interaction is lem that shall be solved. Different settings for SPOT and desired. The spotGUI provides the option to do so with its tools can quickly be tested by using the spotGUI with a configured sampling method to build a design of experi- the given set of test functions. ments. Depending on the objective function configuration, the generated experiments can be evaluated automatically or The possibility to manually input evaluation results enables manually, e.g. a real-world experiment. Such manual results non-programmers to use the spotGUI without any require- can either be imported into the spotGUI or directly entered ments for an objective function definition in code. Thus into the result table. A surrogate-model is fitted to the for example making it possible to use SPOT to optimize given data making interactive 3D-visualizations available. 214 3. EXAMPLE APPLICATIONS 3.1 Applying the Manual Mode The spotGUI offers a couple of functionalities to be easily usable and applicable to problems where real-world experi- ments are required. We can imagine the following example where the user is not too affine with software programming: A machine engineer who needs to set up a new metal hard- ening machine to deliver good performance. Through the machine’s interface, he is allowed to control two temperature parameters which define a temperature curve that the machine runs through in the hardening process. Additionally, he can change two time parameters which de- fine the duration of the heating as well as the cooling phase in the hardening process. He is looking for the set of opti- mized parameters which result in the hardest end product. However, each test requires to run the hardening machine for a few hours and involves material costs. In this scenario, the manual mode of the spotGUI could help the engineer in this parameter optimization problem. First of all, by using Figure 3: Auto generated plot showing the fitted the spotGUI in the manual mode, no coded fitness function surrogate model. Red dots indicate evaluated can- is used. Instead, parameter settings are proposed by SPOT, didate solutions. Hovering the mouse over the plot manually evaluated on the hardening machine and inserted results in the black info box showing more detailed into the results table by the engineer. information for the given plot location. The tool- bar above the plots provides features for easy plot The detailed workflow is as follows: After an initial setup exports. in the spotGUI, defining the bounds and types of the in- put parameters, a DOE (Design of Experiments) is built. This is quickly done via the ’createDOE’ button in the ’run- The graphics are generated through plotly, an R-library for Mode’ tab. A model can be fitted, and a visualization of it creating web-based graphs [7]. The availability of interactive is available. With the now to him available information, the 3D plots enables the user to learn more about the landscape engineer could continue in a few different ways. He could of their objective function intuitively and gives a deeper in- straightforward accept the best solution found in the DOE. sight into variable behavior. After a model fit, it is easily However, this should not be done if resources for more ma- possible to run an optimizer on the model to propose a sin- chine tests exist. Continuing with a more in-depth DOE, gle next candidate solution, thus enabling SMBO even to a he could increase the DOE budget and optionally shrink the manual user / non-programmer. parameter bounds to an area that is considered as promis- ing by the fitted model. The second option to spend the Further options are again aimed at enhancing the automatic remaining test budget is to run an optimizer on the fitted evaluation and optimization of a configured objective func- model via the ’propose new point’ functionality. This ad- tion. As sometimes even just a few objective function evalu- ditional point is the model optimum for some configured ations might take a long time, the spotGUI execution can be infill-criterion. This criterion might be the best-predicted interrupted and restarted from the last completed function point, but depending on the model, it could for example evaluation. For users who only want to use the spotGUI as also be the point with the highest expected improvement as a quick setup tool for their code, another option exists. By utilized in EGO [6]. After evaluating the proposed point on entering the ’Log Only’ mode, all computations that would the machine, the model can be refitted to include the new usually be applied to the objective function are skipped. In- data point. After that, the ’propose new point’ functionality stead, the actions are only written to the code log. From is usable again. Therefore, by using this feature, surrogate there they can be exported and used in any R-Script, en- model-based optimization is available in a manual use case, abling an extra fast setup for new SPOT projects. making a well-known and powerful optimization technique available to a broader audience. Lastly, the configuration 2.5 Save of the spotGUI can easily be changed during the optimiza- Each action that is executed in the spotGUI is written into tion process, allowing for a more interactive optimization an exportable R-Code log. The log is accessible on the ’Ex- approach. port’ tab of the GUI, it can easily be exported or copied to the clipboard through the provided button. The resulting R-Code can be run standalone (given the spotGUI library 3.2 The Electrostatic Precipitator Problem is installed) and generates the same results as previously Electrostatic precipitators (ESP)s are large scale electrical shown in the experimentation tab. This also ensures re- filtering/separation devices. They are used to remove solid producibility of any work that was done with the help of particles from gas streams, such as from the exhaust gases spotGUI. of coal-burning power plants. An overview of the struc- ture of an ESP can be seen in Figure 4. The illustrated separator has three central separation zones in which the 215 4. SUMMARY The SPOT package has been available for many years. It has been continuously updated and grew to a very large and useful platform. However, through the growing amount of possible configurations and use cases it simultaneously be- came more complex to dig through all settings and find the best ones for each problem. The here introduced spotGUI package reduces the configuration complexity back down to a level where any beginner can use the package. It was suc- cessfully applied to industry use cases as well as in student courses. Thus, demonstrating its ease of use and capability to provide easy to access visual information. The playful style with which different optimization methods can be ap- plied makes the software a useful tool in education. One of the most significant drawbacks of the current ver- sion of the spotGUI is its dependency on R. Till now, the Figure 4: Electrostatic precipitator with 3 separa- spotGUI can only be published as a web application avail- tion zones. This figure was kindly provided by Stein- able through a browser or started directly in R. Future work müller Babcock Environment GmbH. on the spotGUI will, therefore, concentrate on making the software available as a standalone executable without the requirement of starting it through R. Additionally, more fea- particles are separated from the gas flow by the precipita- tures are planned or even already are under construction, in- tor. Gas streams in from piping through the inlet hood and cluding: Parallelization support for SPOT, more DOE and exits through an outlet hood. The entrance and exit pip- analysis functionality, additional exports, and report gener- ing of the separator has a much smaller cross-section and ation. therefore a higher gas velocity than desired in the separa- tor. Without additional measures the fast gas stream would 5. ACKNOWLEDGEMENT rush through the center of the precipitator, resulting in very Parts of this work were supported by the ”Ministerium für low separation efficiency. The primary optimization target Kultur und Wissenschaft des Landes Nordrhein-Westfalen” is the so-called gas distribution system (GDS). The GDS is (FKZ: 005-1703-0011). mounted directly behind the flue gas inlet of the precipita- tor. It is used to distribute the gas flow from the small inlet 6. REFERENCES cross-section to the much larger cross-section of the precip- [1] T. Bartz-Beielstein and M. Zaefferer. A Gentle itation zones. The GDS in the given application consists of Introduction to Sequential Parameter Optimization. 49 configurable slots. Each of these slots can be filled with CIplus 1/2012, Fakultät 10 / Institut für Informatik, different types of metal plates, porous plates, angled plates, 2012. or be left completely empty. Increasing the separators effi- [2] T. Bartz-Beielstein and M. Zaefferer. Model-based ciency by achieving a more evenly distributed gas flow al- methods for continuous and discrete global lows a smaller overall separator. A reduced separator size, optimization. Applied Soft Computing, 55:154 – 167, together with lowered operating costs would accumulate to 2017. multiple millions of euro in cost reduction. [3] J. Bossek. smoof: Single- and Multi-Objective Optimization Test Functions. The R Journal, Two central factors reveal a complex to solve optimization 9(1):103–113, 2017. problem: [4] W. Chang, J. Cheng, J. Allaire, Y. Xie, and J. McPherson. shiny: Web Application Framework for a) The amount of configurable slots together with the R, 2018. R package version 1.1.0. amount of available configurations per slot leads to ≈ 1041 possible configurations for the overall system [5] A. Forrester, A. Keane, et al. Engineering design via b) Each objective function evaluation requires a costly surrogate modelling: a practical guide. John Wiley & CFD-simulation in order to judge the gas flow through Sons, 2008. the system [6] D. R. Jones, M. Schonlau, and W. J. Welch. Efficient global optimization of expensive black-box functions. The ESP optimization was approached with a combination Journal of Global optimization, 13(4):455–492, 1998. of a parallelized model-based evolutionary algorithm that [7] C. Sievert, C. Parmer, T. Hocking, S. Chamberlain, was equipped with newly created task-specific mutation and K. Ram, M. Corvellec, and P. Despouy. plotly: Create recombination operators. Tuning these operators was re- Interactive Web Graphics via ’plotly.js’, 2017. R quired in order to be able to reduce the overall runtime of package version 4.7.1. each optimization to fit into standard project run times. In this industry project, the spotGUI was successfully applied to set up parameter tuning for the evolutionary algorithm and its operators. 216 Expensive Optimisation Exemplified by ECG Simulator Parameter Tuning Beate Breiderhoff, Boris Naujoks, Bogdan Filipič Thomas Bartz-Beielstein Jožef Stefan Institute Institute of Data Science, Jamova cesta 39 Engineering and Analytics Ljubljana, Slovenia Steinmüllerallee 6 bogdan.filipic@ijs.si Gummersbach, Germany {firstname.lastname}@th-koeln.de ABSTRACT model is constructed using a three-dimensional grid [5, 6]. This article describes the tuning of an Electrocardiogram For the simulation, the APs are described mathematically (ECG) simulator as a benchmark problem to show the ap- and represent voltage as a function of time for an individual plication of surrogate modelling in complex global optimi- cell. The function AP(t) is parameterised with nine param- sation. After presenting the background on ECG, its sim- eters and is approximated by a combination of exponential ulation and the optimisation task, the main concepts and functions [20]. Out of nine AP function parameters, two methods of surrogate modelling and Efficient Global Opti- have predefined values, while the remaining seven are sub- misation (EGO) are presented. Here, next to the standard ject to optimisation. As three layers of heart muscle cells techniques regularly involved in the algorithm, alternative are considered in the model, the total number of optimisa- approaches are discussed briefly. Finally, first results apply- tion variables is 21. The optimisation goal is to find the ing the depicted algorithm on the ECG simulator optimisa- best set of parameters to produce properly shaped APs and tion problem are presented. approximate simulated ECG waveforms to a measured ECG waveform by perfecting the shape of the APs. Keywords Pearson’s correlation coefficient (PC) is the covariance of ECG, evolutionary algorithms, surrogate assisted optimisa- the two variables divided by the product of their standard tion deviations. The coefficient PC1 between the measured ECG waveforms and the simulator output builds the objective 1. INTRODUCTION function, which is required to be maximized to obtain a good The heart muscle pumps blood in a specific rhythm through- match between the two waveforms. out the entire body. In order to do this, the heart muscle requires an electrical impulse to contract. This electrical 3. THE EGO APPROACH impulse acts as a natural pacemaker. The electric current As each run of the ECG simulator takes around 15 minutes, is then transmitted via specific pathways throughout the finding the best solution is a time consuming process that heart, enabling regular contraction and relaxation. ECG is can take days or weeks. One way of relieving the burden of the result of recording this electrical activity of the heart expensive simulation runs is by constructing approximation over a period of time using electrodes placed on the skin. models that mimic the behavior of the simulator as closely It provides information about the heart’s rhythm and rate. as possible while being computationally cheaper to evaluate. The normal ECG shape and some typical defects are well The basic idea of using surrogate models in optimisation can known, but the transfer function that maps the ECG mea- be quite simple. First, the surrogate models for the objective sured on the skin to individual cells of the middle layer of the function with sufficient accuracy are built; second, the opti- heart wall is unknown. Gathering additional knowledge on mum is found by an optimizer, with the objective function the transfer function would help to improve ECG-based di- evaluated by surrogate models, rather than by the expensive agnostics and enable better prediction of health condition, simulation runs. Since prediction with the surrogate models based on the ECG reading. Simulation models may help, is much more efficient than that by the expensive simulation but the simulation of a human ECG signal is a complex runs, the optimisation efficiency can be largely improved. optimisation problem. Although the framework of the surrogate-based optimisation 2. THE ECG SIMULATOR is very intuitive and simple, questions may arise, e.g.: Is the The ECG simulator1 considered within the SYNERGY2 pro- surrogate model accurate enough and has the true optimum ject is a tool which tries to mimic the activity of the left been reached? The solution gained by the surrogate model ventricle of the heart, by producing ECG waveforms for a is only an approximation to the true optimum. One has to given set of Action Potential (AP) parameters. The heart refine the surrogate models by adding new sample points, which are observed by running the ECG simulator. The 1https://github.com/synergytwinning/ekgsim flowchart of the surrogate-based optimisation is sketched in 2http://synergy-twinning.eu/ Figure 1. 217 the Expected Improvement (EI, cf. [11]) infill criterion. EI Start not only considers the objective function provided by the no model but also the model quality to suggest new points for time-consuming evaluations. DoE EGO starts by building an initial Kriging model using some initial design points which are often produced by an exper- iment design method. Then, in each iteration, the point with the highest EI value is selected by using a traditional Build Surrogate Model optimisation algorithm. The selected point is evaluated us- (e.g. Kriging) ing the real expensive objective function and used to update the Kriging model. In such a way, the EI criterion guides the search toward the optimum of the real problem. Main - Optimise on Surrogate Sub - 3.2 Design of Experiments optimisation (e.g. GA) optimisation The first mandatory step in surrogate modelling is the col- lection of data to set up an initial model. This is normally done by a DoE approach [2], which results in an initial sam- Converge pling plan. By choosing an initial sampling plan the chal- No Add New lenge is to limit the number of samples but nevertheless get a Yes Data Point good and suitable design. There are various sampling tech- niques available such as Uniform Random Sampling, Latin Hypercube Sampling, and Orthogonal Array Sampling. A Run ECG Simulator common choice is Latin Hypercube Sampling (LHS, [16]) a statistical method for generating a near-random sample of Yes parameter values from a multidimensional distribution. No Converge 3.3 Modelling Approaches An important issue is the huge number of surrogate models available in the literature. Here we limit our discussion to Stop three popular techniques that are shortly described below. 3.3.1 Kriging Figure 1: Flowchart of the surrogate-based optimi- Kriging is a popular choice of surrogate models. It under- sation stands observations as realisations of a Gaussian process. The popularity of this technique is due to the fact that it not only produces accurate predictions, but also provides an The steps from the figure like Design of Experiments (DoE), estimate of the prediction uncertainty [14, 18]. building the surrogate model, optimising on the surrogate etc. selection are explained in a bit more detail in the follow- 3.3.2 Random Forests ing. Here, we mention common techniques next to promi- Random Forests [3] are ensembles of prediction trees such nent alternatives. A focus is put on the techniques that are that each tree depends on the values of a random vector used for addressing the ECG simulator optimisation prob- sampled independently and with the same distribution for lem. all trees in the forest. The generalisation error of a forest of tree classifiers depends on the strength of individual trees 3.1 Single-objective Surrogate Modelling in the forest and the correlation between them. Internal In single-objective surrogate-assisted optimisation, there ex- estimates monitor the error, strength, and correlation, and ists only one objective function, which is the fitted surrogate these are used to show the response to increasing the num- model using the acquired data points. The most straightfor- ber of features. Internal estimates are also used to measure ward approach is to find the global optimum of this model. variable importance [3]. The major problem is that the search may stall at a local op- timum. Solving this problem implies that the search needs 3.3.3 Support Vector Regression to combine exploration and exploitation; i.e., the search ex- SVR is a modelling technique based on the theory of sup- plores the total experimental area and zooms in on the local port vector machines [7, 19]. SVR models produce a pretty area with the apparent global optimum. accurate estimate of the objective function, provided that a suitable kernel is selected and parameters are appropri- Efficient Global Optimisation [11, 10] is a popular search ately tuned. This tuning process is expensive, especially for heuristic that tries to realize this exploration and exploita- models with higher dimensions and a high amount of sample tion. Many alternatives exist, one is the pre-selection ap- points [12]. proach described in [9]. EGO is a widely used surrogate- based optimisation algorithm for expensive single-objective optimisation specialised on utilising Kriging modelling and 218 3.4 Optimisers Choosing a suitable search strategy which can perform effec- tive global optimisation is the most difficult part in surrogate- assisted optimisation. In our work we use two well-known optimisers, namely an Evolutionary Algorithm and Simu- lated Annealing. 3.4.1 Evolutionary Algorithms (EAs) EAs are metaheuristics inspired by the process of natural selection that belong to the larger class of evolutionary al- gorithms. They are commonly used to generate high-quality solutions to optimisation and search problems by relying on bioinspired operators such as mutation, crossover and selec- tion [4, 8]. A subclass of EAs are Genetic algorithms (GAs). 3.4.2 Simulated Annealing (SA) SA is a method to solve complex optimisation problems [13]. This method models the physical process of heating a ma- terial and then slowly lowering the temperature to decrease Figure 2: Box-plot of RMSE obtained for cross- defects, thus minimizing the system energy. At each iter- validation of PC1 ation of the simulated annealing algorithm, a new point is randomly generated. The distance of the new point from the current point, or the extent of the search, is based on worst (as its weight was optimized to a value of zero). a probability distribution with a scale proportional to the temperature. The algorithm accepts all new points that improve the objective value, but also, with a certain proba- 5. CONCLUSIONS AND OUTLOOK bility, points that worsen the objective value. By accepting The efficient global optimisation approach is presented. Next points that raise the objective, the algorithm avoids being to standard settings from the methods involved in the algo- trapped in local optima in early iterations and is able to rithm, some alternatives were discussed briefly. The result- explore better solutions globally. ing algorithm is applied to the ECG simulator optimisation problem and first results are presented. 3.5 Model Selection and Validation However, the parameters used for optimisation (i.e., models K-fold cross-validation is an improved scheme which allows invoked, evolutionary algorithm, simulated annealing etc.) us to use most of the data for constructing the surrogates. were not tuned for the best performance. Parameter tuning In general, the final quality of the surrogate model is judged of optimisers might further enhance the surrogate-assisted using the mean and the standard deviation of the root-mean- optimisation process. Here, SPOT [1] might be invoked, square error (RMSE) for each cross-validation set [17]. which provides a set of tools for model-based optimisation and tuning of algorithms. It also includes surrogate models, 4. RESULTS optimisers and design of experiment approaches. The purpose of this article is to summarize approaches to surrogate modelling which are applicable to the ECG simu- The simulator provides two simulated ECG signals at differ- lator. The various surrogate models selected were Kriging, ent positions on the body surface. A second coefficient PC2 SVR, RF and a convex combination ensemble of the former could be used for multi-objective optimisation, also known three models. as multi-criteria optimisation or Pareto optimisation. It is a special case of solving optimisation problems with more The ensemble model performed best in K-fold cross-validation than one objective function to be optimised simultaneously. tests, while SVR performed worst. This provided an insight The final result is a set of solutions known as Pareto opti- of how the models would actually perform during the op- mal solutions. The Pareto front is a set of non-dominated timisation process as shown in Figure 2. Single-objective solutions, being chosen as optimal, if no objective can be optimisation was carried out to investigate the performance improved without sacrificing at least one other objective. of the surrogates in a practical scenario. The single-objective surrogate-assisted optimisation yielded some pretty interest- 6. ACKNOWLEDGMENTS ing facts about the behavior of the ECG simulator. Firstly, This work is funded by the European Commission’s H2020 pro- the maximum value that is achieved for the objective func- gramme, through the Twinning project SYNERGY under Grant tion is 0.31. The EGO algorithm based on Kriging (Ex- Agreement No. 692286 as well as through the UTOPIAE Marie pected Improvement) using simulated annealing performed Curie Innovative Training Network, H2020-MSCA-ITN-2016, Grant superiorly relative to other strategies when comparing the Agreement No. 722734. mean and standard deviation of the best obtained values as shown in Figure 3 [15]. 7. REFERENCES The optimisation of the weight vector for building the en- [1] T. Bartz-Beielstein, C. W. G. Lasarczyk, and semble model revealed that the SVR model did perform M. Preuss. Sequential parameter optimization. In 2005 219 Figure 3: B-plot of best observed values for optimisation of PC1 IEEE Congress on Evolutionary Computation, [11] D. R. Jones, M. Schonlau, and W. J. Welch. Efficient volume 1, pages 773–780 Vol.1, 2005. global optimization of expensive black-box functions. [2] G. Box, J. Hunter, and W. Hunter. Statistics for Journal of Global Optimization, 13(4):455–492, 1998. Experimenters: Design, Innovation, and Discovery. [12] S. Karamizadeh, S. M. Abdullah, M. Halimi, Wiley, New York, 2nd edition, 2005. J. Shayan, and M. j. Rajabi. Advantage and drawback [3] L. Breiman. Random forests. Machine Learning, of support vector machine functionality. In Computer, 45(1):5–32, 2001. Communications, and Control Technology (I4CT), [4] K. A. DeJong. Evolutionary Computation: A Unified pages 63–65, 2014. Approach. MIT Press, Cambridge, MA, 2006. [13] S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi. [5] M. Depolli, V. Avbelj, and R. Trobec. Optimization by simulated annealing. Science, Computer-simulated alternative modes of U-wave 220(4598):671–680, 1983. genesis. Journal of Cardiovascular Electrophysiology, [14] G. Matheron. Principles of geostatistics. Economic 19(1):84–89, 2008. Geology, 58(8):1246–1266, 1963. [6] M. Depolli, R. Trobec, and B. Filipič. Asynchronous [15] A. Mazumdar. Performance analysis of different master-slave parallelization of differential evolution for models in surrogate assisted optimization. Master’s multiobjective optimization. Evolutionary thesis, TH Köln, 2018. Computation, 21(2):261–291, 2013. [16] M. D. McKay, R. J. Beckman, and W. J. Conover. A [7] H. Drucker, C. J. C. Burges, L. Kaufman, A. Smola, comparison of three methods for selecting values of and V. Vapnik. Support vector regression machines. In input variables in the analysis of output from a International Conference on Neural Information computer code. Technometrics, 21(2):239–245, 1979. Processing Systems (NIPS96), pages 155–161. MIT [17] N. Queipo, R. Haftka, W. Shyy, T. Goel, Press Cambridge, MA, 1996. R. Vaidyanathan, and P. Kevin Tucker. [8] A. E. Eiben and J. E. Smith. Introduction to Surrogate-based analysis and optimization. Progress in Evolutionary Computing. Natural Computing Series. Aerospace Sciences, 41:1–28, 2005. Springer, Berlin, 2 edition, 2015. [18] J. Sacks, W. J. Welch, T. J. Mitchell, and H. P. [9] M. T. Emmerich, K. C. Giannakoglou, and Wynn. Design and analysis of computer experiments. B. Naujoks. Single- and multiobjective evolutionary Statistical Science, 4(4):409–423, 1989. optimization assisted by Gaussian random field [19] A. J. Smola and B. Schölkopf. A tutorial on support metamodels. IEEE Transactions on Evolutionary vector regression. Statistics and Computing, Computation, 10(4):421–439, 2006. 14(3):199–222, 2004. [10] A. Forrester, A. Sobester, and A. Keane. Engineering [20] B. Wohlfart. A simple model for demonstration of design via surrogate modelling: A practical guide. John STT-changes in ECG. European Heart Journal, Wiley & Sons, 2008. 8(4):409–416, 1987. 220 A Hybrid Optimization Strategy with Low Resource Usage for Large Scale Multi-objective Problems Wellington Rodrigo Monteiro Gilberto Reynoso-Meza Pontifical Catholic University of Paraná Pontifical Catholic University of Paraná R. Imaculada Conceição, 1155 R. Imaculada Conceição, 1155 Curitiba, Paraná, Brazil Curitiba, Paraná, Brazil wellington.monteiro@pucpr.edu.br g.reynosomeza@pucpr.edu.br ABSTRACT might be more interested to purchase the thighs or other The use of multi-objective approaches to solve problems in cuts, all the other parts must also be processed and sold industry grew in the last years. Nevertheless, these strate- somehow. gies are still unused in many fields where their performance is suboptimal or when they are too complex to be implemented As a result, the production plan is by itself a problem com- or even are simply unknown. One example is in the poultry posed of a large number of variables (at least 2000) [7]. This industry with its particularly complex chain. In this paper, plan is usually executed as a single-objective problem (the we will discuss a hybrid multi-objective approach with low objective being the overall profits) since it is currently able computational resource usage intended for this scenario as to provide results within the same day. However, it is well well as other similar ones. known that these profits differ from the real values since the reliability of the plants vary. By reliability impacts we Keywords mean both internal (e.g. different production costs between multi-objective, optimization, many variables, low resource the plants, worker strikes, unscheduled maintenance, stock usage. issues) or external (suppliers, weather) causes that are re- sponsible to reduce the projected profits. Therefore, the 1. INTRODUCTION company could benefit if the reliability of the production plans was known beforehand—if it was converted to a multi- In corporate environments, the use of simpler tools in some objective problem (MOP) with the objectives being the ex- situations might be favored against other tools that would pected profits and the reliability of said plans, depending generate better results. This business decision might be on the case the analyst could choose a production plan that caused due to the lack of technical understanding of these has less expected profits, but higher reliability rates. On tools or due to cost and performance reasons. One practical the other hand, he or she could also be more aggressive and example shown in a previous work of the author [7] is the attempt higher profits, but also with higher chances of not Production Plan algorithm used by one of the largest meat achieving the expected value. companies in the world, specifically its poultry business line. The work presented in [7] proved it was possible to convert By definition, the supply chain in industry is a complex set the production plan of the company into a multi-objective of operations and resources that must be extremely opti- problem. However, two problems were found: 1) there was mized in order to achieve its maximum potential which does an issue with the input data (the reliability rates of each include the management of upstream and downstream rela- plant), which resulted in very similar grades for all the plants tionships in order to achieve an outcome which is more prof- and 2) the multi-objective optimization algorithm took too itable to all the parties in the chain [2]. One of its parts is long (more than 24 hours in an i7 desktop with 16 GB RAM) the Production Plan, which defines what should one or more to generate the production plans. Nevertheless, before its plants build considering a myriad of variables—i.e. market implementation can be greenlit by the company, additional demand, production line capacity, logistics and stock limits, work on both sides is required. As such, while the data issue suppliers constraints, raw material limits, etc. Therefore, an was corrected by the plants themselves so that it can be accurate Production Plan is a key component to maximize usable, the multi-objective optimization algorithm needed the potential profits. to be greatly improved in order to be executed faster and with lower resource usage so that it could be used by an In the meat industry the challenges are greater. Since it is off-the-shelf corporate laptop and able to generate a Pareto livestock, at the same time the supply chain is very large and front approximation in under one hour. very tight [3]. The former because its production involves genetics, feeding, breeding, growth control from the current Considering this background, the proposal is to generate a animal up to its grandparents and the latter because there multi-objective optimization (MOO) algorithm intended for are very strict sanitary (including, but not limited to the problems with a large number of variables (more than 2000 health, safety and environment) controls with the ration, since the original problem is expected to grow in complex- water, effluents, temperature and vaccination, for example. ity). As such, the main objective here is to have an algorithm Also, the whole animal must be pushed to the market—in that balances both the performance (i.e. low computational the case of the chicken, for instance, even though the market 221 resource usage) and the scalability (i.e. capable of processing and the Ackley’s function and, as shown in [1], implements problems with thousands or tens of thousands of variables). features such as a chaos-based pseudo random number gen- On that end, a test set with similar characteristics of the erator (to address the nonuniform grouping), a correlation real-world problem will be used to evaluate the algorithm. matrix to keep track of the correlations between variable groups and the objective groups, different basic fitness land- The remainder of this document is structured as follows. scape functions (to achieve the mixed separability) and also The second section explains the background of the test prob- making use of a linkage function to define the variable link- lems that will be used as well as their rationale. The third ages. section shows the proposal to modify and test the presented case into a MOP. Then, the following section specifically 3. PROPOSAL shows the technical details of the created MOP as well as As mentioned in the Section 2, this article proposes to imple- its results after optimization. The fifth section presents the ment an algorithm capable of resolving multi-objective prob- conclusions and the future work to be done from this docu- lems with many variables with low resource usage. For this ment. reason, the approach chosen was a hybrid multi-objective genetic algorithm. By hybrid, as shown in [4], it is consid- 2. BACKGROUND ered to be an algorithm (in this case a genetic algorithm) Currently, in the meat industry some optimization solutions enhanced with an additional local search step shortly after are both single-objective and using specialized algorithms the selection of the individuals. The algorithm, shown in built from scratch with the profitability in mind such as the Algorithm 1 and implemented in MATLAB c , heavily OtimixTM. However, said algorithms might not enable the focuses on the parallelization for performance purposes. On industries to use two or more objectives or provide greater the other hand, it attempts to overuse large matrices in or- parameter tuning possibilities. Since such algorithms are der to reduce the memory footprint. targeted towards only one objective, the problem designer usually has only one solution as the result of the minimiza- Algorithm 1: The proposed algorithm tion/maximization, eliminating the possibility to analyze the tradeoffs between different production plans consider- Data: design space, objective vector Result: the Pareto front approximation ing two or more objectives. As a result, the production generate n random solutions; planners are required to empirically consider the differences rank the solutions by a dominance filter; between the plants from a reliability standpoint, leaving no for each generation until it reaches the stopping criteria do possibility to compare the solutions based on this factor. store previous solutions and their objective values; generate offspring by tournament, recombination and The scenario that originated this algorithm had two objec- mutation; join the offspring to the other solutions; tives: profitability and reliability [7]. 2032 variables were partially rank all solutions with a dominance filter; employed from which all were integers—however, this sce- locally improve the best solutions; nario was known to be a test—therefore, more variables were replace the best solutions with locally improved expected. As such, considering these characteristics and the solutions; other business requirements, the new algorithm had to meet rank all the joined solutions with a dominance filter; the following objectives: prune some solutions by their crowding distances. end • Be a multi-objective optimization algorithm; • Be able to resolve problems with many variables (more The local improvement algorithm (shown in Algorithm 2) than 1000, ideally with more than 15000), all integers; has a new, optional parameter named number of random • Low computer resource usage (preferably less than 1 neighbors for the local search (NumberRandomNeighbors). GB RAM per 5000 variables); It is used to improve the performance in scenarios where • Be able to generate a Pareto front approximation (even there are too many variables and this algorithm would take if there is still room for improvements) in less than one too long to process all of the variables. hour. For example, if a solution has 1000 variables and this pa- Since the original data needed additional work from the rameter is not used, this algorithm will create 1000 new teams responsible for it, the alternative was to choose eas- solutions based on the original solution where the first so- ily configurable and reliable mathematical problems. The lution assigned a new random value for the first variable choice was the test problems in [1]—nine large-scale, multi- while keeping all the other variables intact; the second so- objective problems were considered to evaluate the proposed lution assigned a new random value for the second variable algorithm, each configured with 1000, 5000, 15000, 30000 while keeping all the other variables intact and so on. If 100 and 50000 variables. All the variables are integers similar to solutions are involved in the local search, at least 100 thou- the production plan problem—all of the problems were set sand new solutions would be created as a result. On the to have 2 objectives, analogous to the production plan prob- other hand, if a solution had 15000 variables, considering lem. These test problems are henceforth named LSMOPn, the same 100 solutions as a result 1.5 million new solutions where n is the problem number ranging from 1 to 9 and form- would be created. Considering the local search would hap- ing a different problem. All the tests use a combination of pen more than once during the algorithm execution, this six basic single-objective functions. These functions are the method—based on the exhaustive neighborhood exploration sphere function, the Schwefel’s function, the Rosenbrock’s [5] would take too long. This results in greater performance function, the Rastrigin’s function, the Griewank’s function gains between the different generations. 222 Algorithm 2: Local improvement Problem NVar gamultiobj sp-Mode II new algorithm Data: a solution LSMOP1 1000 0.998933 0.916951 0.966355 Result: N eighbors LSMOP2 1000 0.531942 0.472204 0.497616 LSMOP3 1000 0.999881 0.916340 0.960246 initialize the list of neighbors N eighbors; LSMOP4 1000 0.562390 0.494451 0.521077 if N umberRandomN eighbors was given then LSMOP5 1000 0.997482 0.427428 0.628847 select N umberRandomN eighbors random variables; LSMOP6 1000 0.999599 0.626870 0.916751 else LSMOP7 1000 0.999644 0.615992 0.947478 select all the variables; LSMOP8 1000 0.995152 0.508990 0.686416 end LSMOP9 1000 0.991079 0.483426 0.695966 foreach one of the variables selected do LSMOP1 5000 0.990506 0.931265 0.958904 copy the original solution; LSMOP2 5000 0.512829 0.470381 0.488354 randomly modify its value according to its bounds; LSMOP3 5000 0.996004 0.925004 0.954693 evaluate the new solution; LSMOP4 5000 0.545665 0.496839 0.516004 add it to N eighbors; LSMOP5 5000 0.995279 0.561639 0.609145 LSMOP6 5000 0.947114 0.575706 0.677455 end LSMOP7 5000 0.999411 0.752861 0.882027 replace N eighbors with only its anchors. LSMOP8 5000 0.993857 0.599694 0.609580 LSMOP9 5000 0.958439 0.507450 0.593416 LSMOP1 15000 0.979783 - 0.945874 4. TEST LSMOP2 15000 0.513371 - 0.484434 LSMOP3 15000 0.998651 - 0.941039 The new, proposed algorithm and the other algorithms were LSMOP4 15000 0.535847 - 0.510049 tested on an i7 desktop equipped with 16 GB RAM and LSMOP5 15000 0.974157 - 0.688771 LSMOP6 15000 0.771705 - 0.585666 a dedicated video card running Windows 10 Pro running LSMOP7 15000 0.999059 - 0.845907 MATLAB c R2015a. This desktop ran all the tests over the LSMOP8 15000 0.989229 - 0.680188 course of three weeks with one weekly system restart. In LSMOP9 15000 0.990760 - 0.540451 MATLAB c , the start and end time of each algorithm were LSMOP1 30000 - - 0.840306 tracked (therefore storing the time taken for each execution) LSMOP2 30000 - - 0.466373 LSMOP3 30000 - - 0.885842 as well as the Pareto front approximation found for each ex- LSMOP4 30000 - - 0.486888 ecution, each limited to 300 seconds. Outside MATLAB c , LSMOP5 30000 - - 0.192226 the Windows Performance Monitor (perfmon.exe, a known, LSMOP6 30000 - - 0.604263 LSMOP7 30000 - - 0.223234 built-in performance monitor tool in Windows) was used to LSMOP8 30000 - - 0.184088 track performance and memory usage in MATLAB c . As LSMOP9 30000 - - 0.668866 such, it was able to properly track resource usage by each al- LSMOP1 50000 - - 0.831989 gorithm. The new algorithm was tested against MATLAB c ’s LSMOP2 50000 - - 0.476713 own gamultiobj, a built-in, optimized algorithm based on LSMOP3 50000 - - 0.887729 LSMOP4 50000 - - 0.485269 NSGA-II and a Multi-objective Differential Evolution with LSMOP5 50000 - - 0.220266 Spherical Pruning algorithm (sp-MODE II) [8]. LSMOP6 50000 - - 0.573581 LSMOP7 50000 - - 0.081000 LSMOP8 50000 - - 0.208853 4.1 Evaluation Methods LSMOP9 50000 - - 0.669033 With the aforementioned tools both the hypervolume found for each algorithm considering the best Pareto front approxi- Table 1: Median hypervolume values found for each mations found for them and the memory and processor usage algorithm and problem. The values are relative to for each algorithm were measured. the utopia and nadir determined from all solutions in the Pareto front approximations found for all runs For each of the nine LSMOP problems and number of vari- and all algorithms. The best values are in bold. ables (1000, 5000, 15000, 30000 and 50000 variables), 51 runs were executed for each one of the three algorithm. Num- berRandomNeighbors was set to 50 in the new algorithm On the performance side, the memory allocation behavior and the initial number of random solutions set to 20 for all was similar for the problems with the same size for all the algorithms, limited to 200 * number of generations and a three algorithms. The Figure 1 represents the memory usage maximum of 5 generations without improvement (where an for all the runs with 5000 variables for a given problem. improvement is determined when the utopia had improved First, from 11:34:49 PM to around 03:48:00 AM all the 51 in at least 0.0001% for at least one objective) per run. runs for the new algorithm took place, followed by the 51 runs of the gamultiobj from that time to around 7:00:00 AM and later by the 51 runs of sp-MODE II. The new algorithm, 4.2 Findings as specially shown in Figure 1, used a little more than 1 GB The values shown in Table 1 represent the median values in memory in order to achieve the results. The processor for each one of the runs for each algorithm and for each usage registered the same pattern with around 15% in overall problem based on the implementation in [6]. From the hy- usage for the new algorithm. pervolume in itself the new algorithm proved incrementally better performance when the problem has more variables— Furthermore, the memory usage grew considerably when in fact, starting with 15000 variables the other algorithms changing the problem size to 15000 variables, as seen in are unable to run these problems due to out of memory er- an attempt recorded in the Figure 2. While the new al- rors (as shown by their lack of median values depending on gorithm, with its runs recorded between 11:28:23 PM to the case). around 3:30:00 AM kept the memory usage around 1 GB in 223 Figure 3: Hypervolume distribution for all the prob- lems for 15000 variables. The y -axis represents the Figure 1: Results for the memory usage for three hypervolume values, where ga represents the runs algorithms with 5000 variables. The y-axis represent with gamultiobj and new with the new algorithm. the memory usage in hundreds of megabytes. variables since each industrial boiler, production line and memory, gamultiobj (executed from that time up to around work shift could also be accounted. 8:00:00 AM) registered around 10 GB in usage. sp-MODE II, on the other hand, attempted to allocate more than 16 6. ACKNOWLEDGMENTS GB (as registered by the last spike to the right), culminating The authors would like to thank the Pontifical Catholic Uni- in an out-of-memory error. This behavior happened again versity of Paraná for supporting the research. This study with 30000 and 50000 variables, but with gamultiobj as well. was financed in part by the Coordenaç˜ ao de Aperfeiçoa- The processor usage also showed the same behavior from an mento de Pessoal de N´ıvel Superior - Brasil (CAPES) - Fi- usage standpoint. With 15000 variables the new algorithm nance Code 001 and the PQ2|304066/2016-8 grant. used around 25% from the processor, peaking in around 35% with 50000 variables, depending on the problem. 7. REFERENCES [1] R. Cheng, Y. Jin, M. Olhofer, and B. Sendhoff. Test problems for large-scale multiobjective and many-objective optimization. IEEE transactions on cybernetics, 2016. [2] M. Christopher. Logistics & supply chain management. Pearson UK, 2016. [3] F. Flanders and J. R. Gillespie. Modern livestock & poultry production. Cengage Learning, 2015. [4] V. Kelner, F. Capitanescu, O. Léonard, and L. Wehenkel. A hybrid optimization technique coupling an evolutionary and a local search algorithm. Journal of Computational and Applied Mathematics, 215(2):448–456, 2008. [5] A. Liefooghe, J. Humeau, S. Mesmoudi, L. Jourdan, and E.-G. Talbi. On dominance-based multiobjective local search: design, implementation and experimental Figure 2: Results for the memory usage for three al- analysis on scheduling and traveling salesman gorithms with 15000 variables. The y-axis represent problems. Journal of Heuristics, 18(2):317–352, 2012. the memory usage in hundreds of megabytes. [6] MATHWORKS. Hypervolume approximation - matlab central, 2015. 5. CONCLUSIONS [7] W. R. Monteiro and G. Reynoso-Meza. A multi-criteria The proposed algorithm showed that it is better suited for based approach for the production distribution in the problems with a large number of variables. As of the time of poultry industry. In 24th ABCM International writing this paper the company still could not make a new, Congress of Mechanical Engineering. Brazilian Society real-world case available for tests. On the other hand, the of Mechanical Sciences and Engineering, 2017. new algorithm can also be implemented in other complex [8] G. Reynoso-Meza, J. Sanchis, X. Blasco, and scenarios where, for example, the objectives are both the M. Mart´ınez. Design of continuous controllers using a maximum profits for a given distributed production plan and multiobjective differential evolution algorithm with the energy and steam consumption reduction in the plants. spherical pruning. In European Conference on the The energy and steam consumption variables would result Applications of Evolutionary Computation, pages in an estimated multi-objective problem with around 40000 532–541. Springer, 2010. 224 Electric Vehicle Routing Problem: State of the Art Jihane Serrar Rachid ELLAIA El-Ghazali Talbi LERMA Laboratory, E3S center LERMA Laboratory, E3S center BONUS, INRIA Lille-Nord Europe Mohammed V university of Rabat Mohammed V university of Rabat University of Lille, France BP765, Ibn Sina av, Rabat, Morocco BP765, Ibn Sina av, Rabat, Morocco 59655 - Villeneuve d’Ascq cedex serrar.jihan@gmail.com ellaia@emi.ac.ma el-ghazali.talbi@univ-lille1.fr ABSTRACT we assign to each arc the cost considering that the dis- Electric vehicles (EVs) represent a clean alternative but have tance is known and given for each arc. Time windows some limitations especially in terms of autonomy. Therefore, associated with nodes or arcs may also be defined in efficient routing of EVs is crucial to encourage their use. some problems. This article surveys the existing research related to electric vehicle routing problems (EVRP) and their variants. It ex- • Demand: The demands are either given for each node amines EVRP in terms of their definitions, their objectives, and are known in advance in the case of deterministic and algorithms proposed for solving them. problem or given by probabilistic formulas. Keywords • Fleet: The fleet refers to a set or a group of EVs. In optimization, electric vehicles, routing problem fact, it is associated to the electric vehicles available to the routing problem, hence we can either have a 1. INTRODUCTION homogeneous or a heterogeneous fleet. Although, electric vehicles face several challenges such as: the low energy density of batteries compared to the fuel of • Electric vehicles recharging technologies: Un- combustion engined vehicles, the long recharge times com- like the combustible vehicles, the electric vehicles are pared to the relatively fast process of refueling a tank and charged by plugging the car to the electric grid. There the scarcity of public charging stations, they contribute to are four main technologies: a sustainable and environmental friendly freight transporta- tion by reducing the air pollution. – Household charging: EVs can be charged by a conventional household plug using a cable and a Electric vehicle routing problem and variants are considered connector in the vehicle. This technology is slow. as optimization problems and, more specifically, they be- – Fast charging: This technology is a conductive long to the combinatorial optimization problem that can be charging method. It’s faster than the previous solved by two types of solution methods: exact methods and one. approximate ones. – Wireless charging systems: also known as induc- This paper presents an overview of different problems re- tive charging is an emerging technology that al- lated to the electric vehicle routing, different variants and lows EV recharging without the use of a cabled solution methods found in the scientific literature. The rest connection. of the paper is structured as follows. First, Section 2 gives the main characteristics of EVRP. Then Section 3 enumer- – Battery swap: It’s a a high-speed method. ates the electric vehicles drawbacks. Section 4 describes the variants of EVRP presented in the literature. Finally, Sec- • Cost: The cost is a term that depends on different tion 5 reports on different solution methods for EVRP. parameters. It depends on the distance traveled, the energy consumed and the time of the travel. In ad- 2. CHARACTERISTICS OF EVRP dition, in the case of the time window variant, there are some penalties that could be added if the window The electric vehicle routing problem aims at routing a fleet isn’t respected. Moreover, the cost changes from one of EVs on a given network, or a subset of a network, to recharging technology to another. serve a set of customers under specified constraints in order to optimize one or several fixed objective(s). So, an EV routing problem can be defined in terms of the following • Objectives: It could be a single-objective problem or components: a multi-objective problem according to the number of objectives considered. The objectives are very diver- sified because the EVRP has a lot of components in • Network: The network can be represented as a graph its definition, for instance, minimizing the total trav- composed of nodes referring to cities, customers and elling distance, the delay time and the waiting time, depots and arcs standing for connections. Sometimes, the total cost, etc. 225 3. ELECTRIC VEHICLES CHALLENGES 4.3 Electric Vehicle Routing Problem To combat environmental and energy challenges, electric ve- Lin et al. [7] presents a general Electric Vehicle Routing hicles may provide a clean and safe alternative to the inter- Problem (EVRP) that seeks to optimize the routing problem nal combustion engine vehicles. However, electric vehicles while minimizing the total cost related to the distance as well are still facing several weaknesses: as to the energy consumption by the battery. The proposed EVRP finds the optimal routing strategy in which the total 3.1 Autonomy Limitations cost is minimized such that each customer is visited exactly once by one vehicle on its route, the total demand of the The vehicles have a much smaller driving range due to the customers served on a route does not exceed the vehicle limited battery capacity. The range of an electric vehicle capacity. depends on the number and type of batteries used but gen- erally the driving range varies between 80 and 130 km for light duty EVs according to [13]. 4.4 Electric Vehicle Routing Problem with Non- Linear Charging Functions (EVRP-NL) 3.2 Long Charging Times Montoya [12] extended current EVRP models to consider EVs often have long recharge times compared to the rel- partial charging and non-linear charging functions which is atively fast process of refueling a tank which takes just a more realistic for the charging process. In EVRP-NL, the couple of minutes. Its charging time ranges between 0.5 and task consists of minimizing the total traveling distance as 12 hours as mentioned in [12]. Hence, the user must think well as the charging time since it does not depend on the about refueling at night for example. total tour distance. 3.3 Scarce Charging Infrastructure 4.5 Electric Vehicle Routing Problem with Time The number of electric recharging stations is still very small Windows (EVRP-TW) compared with that of conventional fuel stations as the elec- This variant seeks to satisfy the order of customers within tric fuelling points are still in the development stages. So, certain time window. Many researches have been interested the driver must do a research about the plug-in stations in studying this variant. Some of works found in the litera- localisation to know where and when he will have the op- ture are outlined below. portunity the recharge his EV. 4.5.1 EVRP-TW with recharging stations 4. EVRP AND VARIANTS In fact the time window variant of EVRP was first intro- Several versions and extensions of the basic electric vehicle duced by Schneider et al. [14]. They studied the electric routing problem have been presented in the literature. vehicle routing problem with time windows and recharging stations (E-VRPTW) which incorporates the possibility of recharging at any of the available stations considering that 4.1 Green Vehicle Routing Problem (GVRP) the required recharging time depends on the state of the Erdo˘ gan and Miller-Hooks [3] are the first to introduce the charge. Hence, electric vehicles, which have a restricted ca- Green VRP which consists of alternative fuel-powered ve- pacity, must reach cutomers whithin a time window while hicle fleets with limited driving range and limited refueling minimizing the number of vehicles used and the total travel infrastructure. The objective is to minimize the total dis- distance. tance traveled by the vehicles while allowing them to visit stations when necessary. 4.5.2 Electric vehicle routing problems with time win- In [10], Koç et al. proposed the same problem as Erdo˘ gan dows and Miller-Hooks with the motivation of saving the ecosys- Desaulniers et al. [2] tackled the routing problem in which tem and the health of humans while serving and executing route planning has to take into account the limited driving the transportation and good distribution process. range of EVs and the customer time window. The authors studied four variants of this problem. The first one allows More recently, J. Andelmin et al. [8] also studied the green a single recharge per route knowing that batteries must de- vehicle routing problems taking into account the several par- part fully recharged from the station, the second one permits ticularities of autonomy and charging process of this type multiple recharges but only full rechargement are allowed of vehicles. Hence, the refueling stops are allowed. Their unlike the next one where partial battery recharges are al- model aims to find optimal routes while minimizing the to- lowed but just one time and the last one with partial but tal distance and by using a homogeneous fleet of vehicles. multiple recharges permitted. Contrary to Erdo˘ gan and Miller-Hooks, they didn’t put the restriction on the number of vehicles that must be used. 4.5.3 The electric fleet size and mix vehicle routing problem with time windows and recharging sta- 4.2 The Green Vehicle Routing Problem With tions Multiple Technologies And Partial Recharges Hiermann et al. [6] aim to optimize the fleet and the vehicle Felipe et al. [4] presents a variant in which different charg- routes including the choice of recharging times and recharg- ing technologies are considered and partial EV charging is ing stations as the refuelling operation is assumed necessary allowed in recharging stations when needed in order to en- for EVs because of the limited capacity storage of electricity sure the continuity of the route. by batteries. They considered that the fleet is heterogeneous 226 which adds complexity to the problem. Furthermore, they modified Clarke and Wright savings (MCWS) heuristic as incorporate the time windows constraint where customers the original Clarke and Wright algorithm was developed to have to be reached within a specified time interval. tackle the classical vehicle routing problem and its variants, thus it was modified to take into consideration the need to 4.5.4 The recharging vehicle routing problem with visit stations that have to be inserted in the routes while avoiding redundant. Meanwhile, the second heuristic is the time window density-based clustering algorithm (DBCA) that consists of Conrad and Figliozzi [1] introduced the recharging vehicle forming clusters in a clustering step dedicated for that and routing problem wherein vehicles with a limited range must then the MCWS algorithm is applied for each single cluster. service a set of customers, but may recharge at certain cus- tomer locations instead of using only dedicated recharging 5.2 Exact Algorithms stations while operating whithin customer time window. In Desaulniers et al. [2] decided to solve the different variants other words, the battery of a vehicle can be recharged while of EVRP-TW presented in their paper using exact meth- servicing the customer if needed. Also, the authors showed ods. They used the exact branch-price-and-cut algorithms the impact of the customer time windows on the tour dis- adapted to each variant. Hence, for each variant a set of tance taking into account that the driving range is limited routes is generated and for that monodirectional and bidi- and the recharging time is long. rectional labeling algorithms are presented. 4.5.5 Electric vehicle routing problem with time win- Branch-and-price is a metaheuristic that was used by Hier- dows and mixed fleet mann et al. [6] to solve the E-FSMFTW which is formulated Goeke et al. [5] proposed to study a mixed fleet of electric as a mixed-integer linear program (MILP). In fact, the algo- vehicles and internal combustion vehicles. They consider rithm has to insert the charging constraints in its procedure. that the energy consumption function isn’t linear and follows a realistic model depending on multiple parameters like the Exact methods were also used by J. Andelmin et al. [8] speed of the vehicle and the load distribution. Hence, EVs to solve set partitioning (SP) formulation of the green ve- can recharge anytime en route to enhance the driving range. hicle routing problem where each variable corresponds to a simple circuit of a route, thus each SP contains a limited subset of routes. The authors proposed an exact method 4.5.6 Partial recharge strategies for the electric ve- composed from two phases: Phase I computes the lower and hicle routing problem with time windows upper bounds, while Phase II executes the set partitioning In their work, M. Keskin et al. [9] relax the full recharge heuristic and the dynamic programming algorithm. restriction and allow partial recharging in order to minimize time. Therefore, shorter recharging durations are allowed Koç & Karaoglan [10] implemented the B & C (branch and especially when the customer time window is set. The ob- cut) algorithm for the exact solution of the GVRP where jective of the model proposed is to minimize the total dis- the initial solution is generated using the classical simulated tance while respecting the time constraints. Concerning the annealing. In addition, the authors adapted the simulated partial recharge scheme, the charging process is identified annealing to the problems related to the electric vehicle rout- by a continuous decision variable. ing problem by adding the GVRP constraints to improve the results. At each step of the method the new solution is 4.5.7 Heterogenous electric vehicle routing problem compared with the current one so that the best solutions is accepted. with time dependent charging costs and a mixed fleet 5.3 Local Search Heuristics Sassi et al. [13] studied a new real-life routing problem in In [4], some constructive and local search heuristics have which they consider a number of realistic features such as: been proposed by Felipe et al. to find feasible routes while different charging technologies, coupling constraints between considering the recharge constraints as well as the real-world vehicles and charging technologies, charging station avail- size problems. In addition, the authors used the 48A algo- ability time windows, and charging costs depending on the rithm in which they consider 48 combinations of improving time of the day. Also, partial charging is allowed and the algorithms with different neighborhood structures. cost of vehicles as well as the total travel and charging costs. In their study, Sassi et al. [13] formulated the Heteroge- 5. SOLUTION APPROACHES TO EVRP FROM nous Electric Vehicle Routing Problem with Time Depen- LITERATURE dent Charging Costs and a Mixed Fleet (HEVRP-TDMF) using a Mixed Integer Programming Model. And to solve it, In the literature, many studies work on finding sophisticated they worked with a Charging Routing Heuristic (CRH) in and efficient solution methods that can be applied to EVRP. order to find feasible routes. This algorithm is based on two main steps: the first one manages the charging of EVs in 5.1 MCWS and DBCA depot and the second one solves the problem starting from Two heuristics were proposed by Erdo˘ gan and Miller-Hooks the depart of EVs from the depot. Moreover, a Local Search [3] with the goal of finding a set of routes that represents Heuristic based on the Inject-Eject routine with three dif- the feasible solution of the green vehicle routing problem ferent insertion strategies has been introduced. knowing that the authors have formulated it as a mixed- integer linear program (MILP). Actually the first one is the 227 5.4 Hybrid Heuristics [2] G. Desaulniers, F. Errico, S. Irnich, and M. Schneider. Exact algorithms for electric vehicle-routing problems 5.4.1 Hybrid VNS/TS heuristic with time windows. Operations Research, To solve the E-VRPTW, Schneider et al. [14] used a combi- 64(6):1388–1405, 2016. nation of Variable Neighbourhood Search (VNS) and Tabu [3] S. Erdo˘ gan and E. Miller-Hooks. A green vehicle Search (TS) heuristics in order to make use of the diversi- routing problem. Transportation Research Part E: fication of the search provided by the VNS algorithm and Logistics and Transportation Review, 48(1):100–114, the efficiency of TS as many combinatorial problems have 2012. proved that this last heuristic is very strong. This combina- [4] Á. Felipe, M. T. Ortu˜ no, G. Righini, and G. Tirado. A tion has the aim to find feasible solutions while respecting heuristic approach for the green vehicle routing all the constraints. problem with multiple technologies and partial recharges. Transportation Research Part E: Logistics 5.4.2 Multi-space sampling heuristic + Hybrid meta- and Transportation Review, 71:111–128, 2014. heuristic [5] D. Goeke and M. Schneider. Routing a mixed fleet of Montoya [12] adapted the multi-space sampling heuristic electric and conventional vehicles. European Journal of (MSH) used before to tackle the VRP with stochastic de- Operational Research, 245:81 – 99, 2015. mand [11] to the green vehicle routing problem by designing [6] G. Hiermann, J. Puchinger, S. Ropke, and R. F. Hartl. a tailored route extraction procedure. MSH is a heuristic The electric fleet size and mix vehicle routing problem that consists of two main phases: the sampling phase and with time windows and recharging stations. European the assembling phase. Furthermore, a hybrid metaheuristc Journal of Operational Research, 252(3):995–1018, is proposed to tackle the EVRP with non linear charging 2016. function. The metaheuristic combines two heuristics: the [7] O. W. Jane Lin, Wei Zhou. Electric vehicle routing iterated local search and heuristic concentration. problem. The organization, Transportation Research Procedia, 2016. The 9th International Conference on 5.5 Adaptive Large Neighborhood Search Al- City Logistics, Tenerife, Canary Islands (Spain). [8] E. B. Juho Andelmin. An exact algorithm for the gorithm green vehicle routing problem. Transportation Science, The ALNS algorithm was also used by Hiermann et al. [6]. 71:1288 – 1303, 2017. In order to optimize the location of the refueling stations [9] M. Keskin and B. Catay. Partial recharge strategies during the routing process, a hybrid heuristic has been pro- for the electric vehicle routing problem with time posed. This heuristic is a combination of the Adaptive windows. Transportation Research Part C: Emerging Large Neighbourhood Search (ALNS) and an embedded lo- Technologies, 65:111 – 127, 2016. cal search procedure that uses different neighbourhoods. In- [10] C. Koc and I. Karaoglan. The green vehicle routing deed, the local search was used to itensify and strengthen problem: A heuristic based exact solution approach. the search operation guided by the ALNS. Applied Soft Computing, 39:154 – 164, 2016. [11] J. E. Mendoza and J. G. Villegas. A multi-space Moreover, like Hiermann et al., Goeke et al. [5] developed sampling heuristic for the vehicle routing problem the Adaptive Large Neighborhood Search algorithm to ad- with stochastic demands. pages 1503–1516, 2013. dress the Electric Vehicle Routing Problem with Time Win- [12] J.-A. Montoya. Electric Vehicle Routing Problems : dows and Mixed Fleet. They also enhanced the algorithm models and solution approaches. Computation and by a local search for intensification. Language. PhD thesis, Université de Nantes Angers Le Mans, 2016. Also, the ALNS algorithm was proposed by M. Keskin et al. [9] to solve the EVRP with time window. The authors [13] O. Sassi and A. Oulamara. Electric vehicle scheduling formulated the problem as a mixed integer linear program. and optimal charging problem: complexity, exact and heuristic approaches. International Journal of Production Research, 55(2):519–535, 2017. 6. CONCLUSIONS [14] M. Schneider, A. Stenger, and D. Goeke. The electric Over the last several years, the green vehicle routing problem vehicle-routing problem with time windows and has been widely studied. This survey lists the main works recharging stations. Transportation Science, that exist in the scientific literature since its appearance in 48(4):500–520, 2014. 2011 by Erdo˘ gan and Miller-Hooks. Based on this paper, the models that have been proposed are single-objective. Yet, most of real problems in industry are multi-objective by nature, so a multi-objective variant of EVRP must be proposed. 7. REFERENCES [1] R. G. Conrad and M. A. Figliozzi. The recharging vehicle routing problem. In IIE Annual Conference. Proceedings, pages 1–8. Institute of Industrial Engineers-Publisher, 2011. 228 Optimization of End-to-End Deep Learning for Obtaining Human-Like Driving Models Erik Dovgan Jaka Sodnik Bogdan Filipič Department of Intelligent Faculty of Electrical Department of Intelligent Systems Engineering Systems Jožef Stefan Institute University of Ljubljana Jožef Stefan Institute Jamova cesta 39 Tržaška cesta 25 Jamova cesta 39 SI-1000 Ljubljana, Slovenia SI-1000 Ljubljana, Slovenia SI-1000 Ljubljana, Slovenia erik.dovgan@ijs.si NERVteh, raziskave in razvoj, bogdan.filipic@ijs.si d.o.o. Kidričeva ulica 118 SI-1236 Trzin, Slovenia jaka.sodnik@fe.uni-lj.si ABSTRACT data, i.e., driving data of only a small subset of driving sit- Modeling human driving with human-like driving models uations. Consequently, the time to collect the driving data can help companies in the evaluation of human drivers. Whi- is reduced, while the driver or more precisely his/her clone le a human-like driving model can be tested in various sce- is still evaluated in a large number of situations. narios, this is not feasible for driver evaluation due to time constraints. During the evaluation, only a small set of driv- Existing work has demonstrated that end-to-end approach ing data can be typically collected for each driver, which for learning to drive is appropriate when large sets of learn- represents an issue for advanced modeling approaches such ing data are available [5, 6, 14]. On the other hand, the as deep learning. To overcome this issue, an optimization problems with small sets of learning data have not been ad- approach is proposed, which tunes deep learning when a dressed appropriately. This paper aims at tackling this issue small learning dataset is available. by enhancing end-to-end deep learning approach with opti- mization in order to obtain human-like driving models from Keywords small sets of learning data. optimization, deep learning, human-like driving models The paper is further organized as follows. Section 2 presents 1. INTRODUCTION the optimization approach for end-to-end deep learning. Ex- periments and results are described in Section 3. Finally, Human-like driving models have been learned with several Section 4 concludes the paper with ideas for future work. methods, such as ARX models [8], Gaussian processes [11], Gaussian mixture models [1], artificial neural networks [15], 2. OPTIMIZATION OF END-TO-END DEEP support vector regression [13], etc. Recently, Deep Neural Networks (DNN) are being effectively used in learning tasks LEARNING from various application fields. For example, when driving End-to-end deep learning approach applies deep neural net- a vehicle, DNN can be used to recognize the road, other works to learn the transformation between the input and the vehicles, pedestrians, etc. from video data [7]. Moreover, output data. The main property of this approach is that a DNN has been also applied to directly learn the control single model is used to obtain this transformation. There ex- actions from video data without firstly reconstructing the ist also other approaches that decompose the problem and scene. This approach is called end-to-end learning and its apply specific models for each subproblem. For example, examples aim to learn steering, throttle and braking control one model can be used to recognize the objects, while an- actions, etc. [5, 6, 14]. other model can be used for higher-level reasoning [7]. The end-to-end approach aims at solving all the subproblems at Unfortunately, deep learning has a significant drawback: it once with a single model [5]. requires a lot of learning data. Existing driving datasets used for training DNN models vary from about 10 hours Existing work in the field of end-to-end deep learning for to up to 10,000 hours [14]. However, in some cases such a obtaining human-like driving models has shown that the se- large set of driving data is not available. For example, the lection of deep learning model and its parameter values is deep learning approach can be used to assess a driver, e.g., not straightforward [9]. In addition, the data need to be if he/she drives safely, is able to avoid critical situations, augmented to learn how to recover from poor positions or etc. [12]. This can be done by building a human-like driver orientations [3]. We propose to automate the selection of model, i.e., a clone of the driver, and test it in a large number appropriate parameter values and data augmentation func- of driving situations. A similar approach has been applied in tions with an evolutionary algorithm. Evolutionary algo- related domains where the goal was to learn human behav- rithms are search and optimization algorithms inspired by ior [10]. This procedure requires only a small set of driving the principles of biological evolution. They work with a set 229 • Batch size: Parameter of the deep learning algorithm. Defines the number of training examples utilized in one learning iteration. • Number of epochs: Parameter of the deep learning al- gorithm. Defines the number of passes through the training dataset during learning. • Image multiplier: Data augmentation parameter. De- fines how many times an image is multiplied. If it is multiplied, it is divided into overlapping subimages. For example, if the image is multiplied by 3, three im- ages are created containing: 1) left 80 % of the original image; 2) central 80 % of the original image; 3) right 80 % of the original image. The control actions are also appropriately adapted. For the left images steer- ing is added to simulate turning right, while for the right images steering is subtracted to simulate turning left. • Noise added to output: Data augmentation parame- ter. Defines the amount of noise an to be added to the control actions. The amount of noise is randomly selected at each time step with a uniform distribution Figure 1: Overview of the algorithm for obtaining U (−an, an). human-like driving models. • Flip image: Data augmentation parameter. Defines whether randomly selected images should be vertically of solutions that are improved through several generations flipped. If the image is flipped, the control action is by applying genetic operators, i.e., selection, crossover and also appropriately adapted. mutation [4]. • Activation function: Parameter of the neural network We propose to discover human-like driving models in two model. Defines the activation function of the neural steps. In the initial step, driving models that are able to network layers. drive the vehicle along a route are built, while in the final • Kernel regularizer: Parameter of the neural network step, these models are enhanced to imitate human driving. model. Defines the regularization of the neural net- The approach presented in this paper focuses on the initial work layers, which applies penalties on layer weights. step by applying an evolutionary algorithm to maximize the The penalties try to keep the weights small, which re- length of the route that has been traveled by the driving duces the possibility of overweighting a small subset of model during the simulation. Each solution (consisting of layer’s input data and prevents overfitting. parameters of model construction) is evaluated by applying the following steps: 3. EXPERIMENTS AND RESULTS The developed approach was tested on two scenarios. Both 1. The learning data are augmented to enable recovery scenarios did not contain traffic vehicles or pedestrians. For from poor situations or orientations. both scenarios, the same architecture of the neural network 2. The deep learning algorithm is used to learn a human was used. This architecture is shown in Figure 2 and is based driving model. on the architecture presented in [2]. It contains five convo- lutional layers and three fully connected layers. The con- 3. The driving model is evaluated on a route to measure volutional layers extract features, from simple features such the route length of feasible driving. as lines to complex features such as road contour. The fully connected layers implement the vehicle controller, which cal- The driving simulation stops if the driving becomes infeasi- culates the control action based on the extracted features. ble (e.g., the vehicle goes offroad) or when the entire route is traveled. The evolutionary algorithm applies tournament 3.1 First scenario selection (tournament size = 2), two-point crossover (prob- The first scenario consisted of a circular route of around 2 ability = 0.9) and uniform mutation (probability = 0.1) to km, which is shown in Figure 3a. An example of a route improve the solutions over generations. An overview of the image as input to the neural network is shown in Figure 4a. developed algorithm and its steps, i.e., evolutionary algo- The learning data were obtained from one driving along the rithm steps (selection, crossover and mutation) and solution route. evaluation steps (data augmentation, model building and model evaluation), is shown in Figure 1. The proposed approach was evaluated by tuning only a sub- set of the parameters listed in Section 2, which already en- The evolutionary algorithm optimizes the following deep abled us to obtain models that drove along the entire route learning and data augmentation parameters: for this scenario and consequently no additional parameters 230 (a) (b) Figure 4: Examples of the input images: (a) first scenario, (b) second scenario. Table 1: Parameter values for the first scenario Parameter Values image multiplier {1, 3, 5} noise added to output {0, 0.1} flip image {true, f alse} batch size 40 (not tuned) number of epochs 100 (not tuned) activation function elu (not tuned) kernel regulizer none (not tuned) driven. Figure 2: Architecture of the neural network. The proposed approach was evaluated with the parameter values shown in Table 3. The results show that the built models were able to drive only short routes (see Figure 5). However, it should be noted that due to high time complex- ity of deep learning, only a small number of generations were executed. More precisely, it took more than 17 days to ex- ecute 30 generations on a 3.6 GHz desktop computer with 16 GB RAM. The analysis of the results also shows that the activation function had the most significant effect on the re- sults. It turned out that the majority of models that were able to drive more than 450 m, contained the relu activa- tion function. In addition, the models were able to drive on (a) (b) straight segments, but had issues with crossroads. This is Figure 3: Maps of the testing routes: (a) first sce- probably due to the relatively simple architecture of neural nario, (b) second scenario. network. For example, images of the first route (see Figure 4a) are significantly less complex in comparison to the im- ages of the second route (see Figure 4b), since they do not needed to be tuned. The values of tuned and not tuned contain any buildings, sidewalks, crossroads, etc. Conse- parameters are shown in Table 1. quently, more complex architectures of the neural network are needed for the city roads. These can be obtained by The feasible solutions, i.e., those solutions that drove the optimizing also the topology of the neural network. entire route, are shown in Table 2. These results show that feasible solutions multiply the images by 3 or 5 and flip 4. CONCLUSIONS images, while the noise added to output does not influence This paper presented an optimization approach for tuning the results. In addition, the results also show that a lower end-to-end deep learning that builds human-like driving mod- number of epochs is needed if the images are multiplied more times. Table 2: Tuned parameter values of feasible solu- 3.2 Second scenario tions for the first scenario The second scenario was related to a city whose map is Noise added Image Flip Epochs to feasible shown in Figure 3b. Figure 4b shows an example of the to output multiplier image solution city image, which was given as input to the neural network. 0 3 true 30 The learning data were obtained from one driving through 0.1 3 true 30 several crossroads. In contrast to the first scenario, the sec- 0 5 true 18 ond scenario does not predefine the route. Nevertheless, the simulation stops if a distance of more than 2 km has been 0.1 5 true 16 231 Table 3: Parameter values for the second scenario of the IEEE Intelligent Vehicles Symposium IV, pages 814–819, 2011. Parameter Values [2] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, batch size {20, 40, ..., 200} B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, number of epochs {10, 20, ..., 50} U. Muller, J. Zhang, X. Zhang, J. Zhao, and K. Zieba. image multiplier {1, 3, 5, 7} End to end learning for self-driving cars. arXiv, noise added to output {0, 0.05, ..., 0.20} 1604.07316, 2016. flip image {true, f alse} [3] Z. Chen and X. Huang. End-to-end learning for lane activation function {linear, elu, relu} keeping of self-driving cars. In Proceedings of the kernel regulizer {none, l2(0.001)} IEEE Intelligent Vehicles Symposium IV, pages 1856–1860, 2017. [4] A. E. Eiben and J. E. Smith. Introduction to Evolutionary Computing (2nd ed.). Springer, Berlin, 2015. [5] H. M. Eraqi, M. N. Moustafa, and J. Honer. End-to-end deep learning for steering autonomous vehicles considering temporal dependencies. CoRR, abs/1710.03804, 2017. [6] T. Fernando, S. Denman, S. Sridharan, and C. Fookes. Going deeper: Autonomous steering with neural memory networks. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 214–221, 2017. [7] A. Garcia-Garcia, S. Orts-Escolano, S. Oprea, Figure 5: Length of the feasible route through gen- V. Villena-Martinez, P. Martinez-Gonzalez, and erations for the second scenario. J. Garcia-Rodriguez. A survey on deep learning techniques for image and video semantic segmentation. Applied Soft Computing, 70:41–65, 2018. els. This approach aims at learning good driving models [8] K. Mikami, H. Okuda, S. Taguchi, Y. Tazaki, and when a low quantity of learning data is available. It was eval- T. Suzuki. Model predictive assisting control of vehicle uated with one neural network architecture on two routes: following task based on driver model. In Proceedings of a circular route and a city route. The results show that the IEEE Conference on Control Technology and this approach was able to find driving models for the cir- Applications, pages 890–895, 2010. cular route, but did not manage to find driving models for [9] J. Patterson and A. Gibson. Deep Learning: A handling crossroads inside the city. Practitioner’s Approach. O’Reilly, Sebastopol, 2017. [10] A. Tavčar, B. Kaluža, M. Kvassay, B. Schneider, and Future work will focus on determining the most appropri- M. Gams. Surrogate-agent modeling for improved ate neural network architecture for urban environments. In training. In Proceedings of the Twenty-first European addition, the efficiency of the evolutionary process needs to Conference on Artificial Intelligence (ECAI), pages be increased by, for example, introducing parallelism in the 1103–1104, 2014. model learning. Furthermore, the behavior of the obtained driving models will be compared to human driving behavior [11] Q. Tran and J. Firl. Modelling of traffic situations at to determine how well the models reproduce human driving. urban intersections with probabilistic non-parametric In case of unacceptable reproduction, these models with be regression. In Proceedings of the IEEE Intelligent enhanced to obtain driving models that are able to imitate Vehicles Symposium IV, pages 334–339, 2013. human driving. [12] K. Trontelj, T. Čegovnik, E. Dovgan, and J. Sodnik. Evaluating safe driving behavior in a driving simulator. In Proceedings of the 7th International 5. ACKNOWLEDGMENTS Conference on Information Society and Technology, The authors acknowledge the financial support from the pages 299–302, 2017. Slovenian Research Agency (research core funding No. P2- [13] D. Wei and H. Liu. Analysis of asymmetric driving 0209, and research project “Multiobjective discovery of driv- behavior using a self-learning approach. ing strategies for autonomous vehicles”, funding No. Z2- Transportation Research Part B: Methodological, 7581). This work was also partially funded by NERVteh, 47:1–14, 2013. raziskave in razvoj, d.o.o., and is part of a project that has [14] H. Xu, Y. Gao, F. Yu, and T. Darrell. End-to-end received funding from the European Union’s Horizon 2020 learning of driving models from large-scale video research and innovation programme under grant agreement datasets. In Proceedings of the IEEE Conference on no. 692286. Computer Vision and Pattern Recognition, pages 3530–3538, 2017. 6. REFERENCES [15] L. Xu, J. Hu, H. Jiang, and W. Meng. Establishing [1] P. Angkititrakul, C. Miyajima, and K. Takeda. style-oriented driver models by imitating human Modeling and adaptation of stochastic driver-behavior driving behaviors. IEEE Transactions on Intelligent model with application to car following. In Proceedings Transportation Systems, 16(5):2522–2530, 2015. 232 A Bi-Objective Maintenance-Routing Problem: Service Level Consideration Mohammad Rahimi El-Ghazali Talbi INRIA Laboratory, CRISTAL-CNRS, INRIA Laboratory, CRISTAL-CNRS, 59650 Lille, France 59650 Lille, France Mohammad.Rahimi@inria.fr El-ghazali.Talbi@inria.fr ABSTRACT To the best of our knowledge, few studies attempted to investigate We study a joint maintenance and routing problem and investigate the simultaneous maintenance scheduling and vehicle routing the impact of service level on the optimization of the total expected problem and consider two described assumptions. López-Santana cost. We propose a new bi-objective mathematical model to et al. [6] combine maintenance and routing problems to schedule determine an optimized maintenance-routing policy, maintenance operations for a set of geographically distributed simultaneously. In this model, the first objective function machines and plan to assign a set of technicians to perform minimizes the total costs due to traveling and a delay in start time preventive maintenance at the customer sites. The authors use a of a Preventive Maintenance (PM)/Corrective Maintenance (CM) distribution function for taking into account failures of machines as operation. The second objective function considers the service level an uncertain parameter. In this study, they use two-step iterative which is measured based on waiting times before beginning of the approach to solve the model which causes minimizing the total CM operations. In the proposed model, we consider time windows maintenance and routing cost, waiting time at each customer and in repairing the machines and skill-based technician assignment in failure probability. performing PM/CM operations. The proposed framework is In this study, we propose a new framework to model and to modelled as a mixed-integer linear program and is solved by using establish the trade-off between the service level (measured based the software GAMS. on waiting times before beginning of the CM operations) and Keywords different maintenance costs by taking into account the presented issues. preventive and unforeseen maintenance, vehicle routing problem, scheduling, service level, multi-objective mathematical model 2. PROBLEM DESCRIPTION 1. INTRODUCTION In this section, a bi-objective mathematical model is proposed to determine optimized routing-maintenance policy. In this model, Regularly planned and scheduled maintenance is a critical first objective function minimizes the total costs due to traveling, requirement to reduce the occurrence of an unforeseen failure and delay in start time of a Preventive Maintenance (PM)/Corrective keeping the equipment running at peak efficiency. Maintenance Maintenance (CM) operation at customer while second objective scheduling becomes complex when the machines are function attempts to minimize the waiting times before beginning geographically distributed. In this case, in addition to assigning the of the CM operations. maintenance operations to technicians, it is needed to find the best set of routes for technicians’ visits. In fact, it is necessary to study In this study, we consider a system with geographically distributed the maintenance and the routing scheduling decisions customers, where each customer has one machine that should be simultaneously. Such a joint decision problem is known as the visited and repaired by technician in different cycles. The PM maintenance-routing problems. operations are scheduled with a certain frequency to reduce the occurrence of unforeseen failure in the long term. Regarding the In the literature there are various studies which investigate previous experiences, the time of unforeseen failure occurrence is combination of maintenance and routing problem [1]–[5]. In the known for each machine at each customer, but its repairing can be most of these studies, authors have two initial assumptions: postponed until defined period. The time interval between  The replacement would be done immediately, if an occurrence of unforeseen failure and its repairing named waiting unforeseen failure occurs for the machines. In fact the time. The set of technicians, who need to visit the set of machines authors do not consider waiting time for performing a CM to perform the PM/CM operations to prevent the system failure. operations. While considering the waiting time is important The technician are different in duration time of doing a PM/CM especially where the machines are geographically operation which causes different in salary. A central depot is distributed and the number of technicians and machines are concerned as the point of departure and final destination. Since each limited. technician should travel to perform PM/CM operation at the customer location, the distance between each two customer is  defined. The main aim of this study consist of determining a joint The scheduling is predefined and authors try to assign the routing-maintenance policy for all machines taking into account technicians to machines considering skill of technician, making a balance between the waiting time and total cost of system. time windows and etc. An unforeseen failure causes The optimized maintenance policy determines in which periods the changes of the maintenance scheduling. In this case, PM and CM operations should be performed at each customer. The maintenance scheduling and routing should be done optimized routing policy determines that which technician is simultaneously. assigned to which customers and in which sequence should visit and perform PM and CM operations at each period. 233 The detailed conditions of system are summarized as follows: 2.1 Mathematical Formulation  The time required to perform a maintenance operation The following notations are used in the proposed model. depends on the skill of the assigned technician. Sets M set of customers, index for customers (1,2,…, m)  More skilled technicians receive more salary. M’ set of customers and central depot, (0,1,2,…, m+1) K index for technicians (1,2,…, k)  All technicians are able to perform any PM/CM operation. t, t’, t’’ index for period (1,2,…, T) Parameters  The technicians start in the central depot in the beginning of ck one unit time cost of a PM/CM operation by technician k each period and should return to the central depot by the end pm of the period. k time required to perform a PM operation by technician k time required to perform a CM operation by technician cmk k  Each machine should be repaired by only one technician at duration of the interval between two consecutive PM each period. It means if the machine should be repaired in λ operations the specific period, only one technician should be assigned L allowed duration to repair occurred unforeseen failure to the machine. zit a binary parameter which determines occurrence of unforeseen failure in customer i at period t  The PM operation should be performed on all the machines tjj traveling time between customer i and j at the first period. r transportation cost per unit time earliest and latest possible start time of a PM/CM [ai, bi]  If no unforeseen failure occurs on the machine at planning operation at customer i horizon, the PM operations will be performed regarding the pi penalty cost of one unit time delay due to start time of a defined frequency. The frequency is defined regarding PM/CM operation at customer i after latest possible planning horizon and the duration of the interval between arrival time two consecutive PM operations. G a large value number Variables  In the case of unforeseen failure occurrence on the machine, 1 if customer j is visited exactly after customer i by x no predictive maintenance can be scheduled and performed ijkt technician k at period t, otherwise 0 before performing CM operation. In this case, CM 1 if PM operation is planned in customer i at period t, y operation should be scheduled to assign a technician on the it otherwise 0 machine until maximum L period. Moreover, next PM 1 if a CM operation is planned in customer i at period t operation will be scheduled and performed after λ period. uit’t for the an occurred unforeseen failure at period t’, otherwise 0  After performing a CM operation, the machine returns to 1 if delay occurred in visiting customer i at period t, βitt’ the good condition and no unforeseen failure occurs until otherwise 0 the next repairing that will be a PM operation in λ period. It 1 if customer i is visited by technician k at period t to µikt means two unforeseen failure cannot occur consequently. perform a PM operation, otherwise 0 1 if customer i is visited by technician k at period t to πikt  The time required to perform a CM operation is longer than perform a CM operation, otherwise 0 the time required to perform a PM operation on each start time of an operation by technician k in customer I, Tikt machine. period t delay in start time of a PM/CM operation in customer i at dit period t  The CM cost is larger than the PM cost.  The mathematical model associated with the presented framework The machines impose time windows on the system which is provided in this section. Each equation in this model is detailed means the technician should start maintenance operation below. before the latest possible start time. In cases where this time windows is not respected, a delay penalty applies if the Min f   x .t .r d .p  .c .pm technician starts after the latest allowed time. 1 ijkt ij it i ikt k k i, j ,k ,t i,t i,k,t  The travel time between two customers depends on the    (1) .c .cm ikt k k speed of the vehicle in the rout at each period. i,k,t Min f    2 (2) , , itt i t t   S. t.  T 1  y   1 i   M (3) t it    234 y  1 z i  M,t performing a CM operation return the machine to as good as new (4) it it condition again and no PM operation is needed until next λ periods. Constraint (7) ensures that when a PM operation is performed at the y  1 i  M i1 (5) period t and no unforeseen failure occurs on the machine until the next λ periods, then a PM operation should be scheduled and t  1   y    i  M,t,t performed at the period of t+λ. Constraint (8) checks that when a   1 (y u  ) (6) t t 1  it it it t CM operation is performed, then a PM operation can be scheduled at the interval of λ periods or an unforeseen failure can be occurred t y  y    z i  M,t (7) until next λ periods. Equation (9) determines in which period a CM it i (t  ) tt it operation should be scheduled and performed to repair the occurred unforeseen maintenance. Moreover, this equation checks that CM t u    i  M,t  y  z (8) operation should be scheduled in a way to assign a technician on it t i(t  ) tt 1  it the machine until maximum L period after the failure. Equation t L  (10) calculates the waiting time until performing a CM operation in u  i  M ,t (9)   z  it t it t t  the case where an unforeseen failure occurs. Equation (11) ensures in case of unforeseen failure occurrence, the CM operation should     be performed once. Constraint (12) guarantees that CM operation  (t t).u i  M,t,t itt itt (10) and PM operation cannot be scheduled and performed for the same  u  period, simultaneously. Equations (13) and (14) determine that  1 i  M ,t (11) t itt visiting the customer is related to a PM operation or a CM operation. y  u   1 i  M ,t,t (12) it it t Equation (15) makes a connection between routing and  maintenance variables. This equation checks when a PM/CM   y i  M,t (13) should be performed on the machine, a technician should be k ikt it assigned to the machine.     u i  M,t (14) Constraint (16) guarantees that only one technician should be k ikt t it t assigned on the each machine at each period. Constraint (17) M  1   x     j  M, j  ,ik,t ensures that technician leave the current customer to the next one, (15) i M  , i0 ijkt jkt jkt after finish the PM/CM operation. Equation (18) determines the start time on the machine, which is calculated as the start time of M 1   x  1 j   M , k,t the immediate previous customer, increased by the PM/CM (16) j M  , j 1  0 jkt operation time and the traveling time between the two customers. Equation (19) checks the time windows constraint and calculates M  1 M    x   x  0 j  M,k,t (17) the delay. Finally, (20) and (21) impose bounds on the variables. i M  , i0 ijkt i M  , i2 jikt 3. RESOLUTION METHOD T   .pm  .cm  t In this section we firstly introduce the instance generation method ikt ikt    k ikt k ij i, j M (18) and solution procedure, briefly. Then, a numerical analysis is  T  . G (1 x ) k , t presented which derives managerial results. jkt ijkt Problem instances have been generated by a random generator. In M  1 M   1  i  M  this way, parameters of the problem are generated using random a .  x  T  b .  x  d (19) numbers by a discrete uniform distribution. Then, to solve the i jikt ikt i jikt it  problem, we use the weighted sum method [7]. Under this method, j M  , j 0 j M     , j 0 k , t the problem is solved by considering each objective function x , y , u     separately in both the maximization and the minimization for  ,   ,  , 0,  1 i, j M , k , t (20) ijkt it it t it t ikt ikt finding extreme points of each objective function. Then, a new single objective is considered that aims to minimize the weighted T , d  0 i  M , k , t (21) sum over the normalized and non-dimensional objective function. ikt it In order to show the feasibility and applicability of proposed model, The first objective function (1) minimizes the total cost which a small size problem is generated and it is solved based on consist of traveling cost between customers, penalty cost due to generated instance problem. It is assumed that there are 6 customers start time out of time windows and the wages of technicians for (m=6) where 3 periods are defined as duration of the interval PM/CM operations. The second objective function (2) optimizes between two consecutive PM operations (λ=3) and 2 periods are the customer satisfaction level by minimizing the waiting times considered as allowed duration to repair occurred unforeseen until performing a CM operation in the case where an unforeseen failure (L=2) by using 3 types of technicians (k=3) during 10 failure occurs. periods. To solve this problem, the “GAMS v22.2” optimization Constraint (3) checks number of PM operations on the machine of software using solver CPLEX v10.1 is used. each customer should not be exceeded. Constraint (4) guarantees At the beginning, the problem is solved without considering the that if the unforeseen failure occurred, then the PM cannot be second objective. In this case the total cost is optimised. The results scheduled and performed for the same period. Equation (5) show that minimum value of total cost is 402 while waiting time in determines that at the first period, PM operation should be this situation is 14. In the next step, the first objective function is performed on the all the machines. Equation (6) guarantees that relaxed and model is solved by minimizing the second objective 235 function. The obtained results shows when the minimum value of 4. CONCLUSION waiting time (second objective function) is 6, the value of total cost In this paper the integration of maintenance and routing problem is is 1,053. Table 1, shows the minimum and maximum value of investigated by taking into account waiting time for performing a objective functions. CM operation when unforeseen failure occurs. For this Purpose, a Table 1: Min. and Max. value of objective functions bi-objective mathematical model is proposed to find the optimized Minimum value Maximum Value policy of maintenance and routing problems and make a trade-off First objective 402 1,053 between maintenance costs and service level which is measured by Second objective 6 14 waiting time for performing a CM operation. In the proposed model The bi-objective model can be converted to a MILP model with one the time windows is considered for starting maintenance operation objective function using the equation (22). on the machine by technicians. Moreover, the technician’s skill regarding required time to perform a maintenance operation is min min f  f f  f 1 1 f    1  2 2 considered. Our results for a small size instance show that to  max min max min f  f f  f decrease by 57% of the waiting time, we have to increase the costs 1 1 2 2 by 62%. In this equation, α presents the importance degree of each objective function and varies between 0 and 1. Our future research in this area includes the consideration of stochastic parameters and proposing an efficient solution approach. Furthermore, the Objective Functions Value (OFV) by changing α value is introduced in Table 2. 5. ACKNOWLEDGMENTS Table 2: OFV against α This project is supported by The ELSAT2020 project which is co- α financed by the European Union with the European Regional 0 0.3 0.5 0.7 1 Development Fund, the French state and the Hauts de France First OFV 1,053 886 689 602 402 Region Council. Second OFV 6 8 9 12 14 6. REFERENCES Run time (second) 157 166 147 133 158 [1] R. Macedo, R. Benmansour, A. Artiba, N. Mladenović, and D. Urošević, “Scheduling preventive railway maintenance According Table 2, the total cost from 1,053 to 402 causes activities with resource constraints,” Electron. Notes increasing 57% in waiting time (from 6 to 14). It means the best Discret. Math., vol. 58, pp. 215–222, Apr. 2017. value of waiting time can be reached by increasing 62% in total [2] F. Blakeley, B. Argüello, B. Cao, W. Hall, and J. cost. Knolmajer, “Optimizing Periodic Maintenance Operations for Schindler Elevator Corporation,” Interfaces (Providence)., vol. 33, no. 1, pp. 67–79, Feb. 2003. [3] A. A. Kovacs, S. N. Parragh, K. F. Doerner, and R. F. Hartl, “Adaptive large neighborhood search for service technician routing and scheduling problems,” J. Sched., vol. 15, no. 5, pp. 579–600, Oct. 2012. [4] A. Goel and F. Meisel, “Workforce routing and scheduling for electricity network maintenance with downtime minimization,” Eur. J. Oper. Res., vol. 231, no. 1, pp. 210– 228, Nov. 2013. [5] M. D. Berrade, P. A. Scarf, C. A. V. Cavalcante, and R. A. Dwight, “Imperfect inspection and replacement of a system  with a defective state: A cost and reliability analysis,” Figure 1: Variation of total cost against waiting time by Reliab. Eng. Syst. Saf., vol. 120, pp. 80–87, Dec. 2013. changing value of α [6] E. López-Santana, R. Akhavan-Tabatabaei, L. Dieulle, N. Labadie, and A. L. Medaglia, “On the combined The variation of objective functions value by changing of α value maintenance and routing optimization problem,” Reliab. is presented in Figure 1. In this figure, X-axe shows value of total Eng. Syst. Saf., vol. 145, pp. 199–214, Jan. 2016. cost and waiting time while Y-axe presents different value of α. By [7] M. Rahimi, A. Baboli, and Y. Rekik, “A bi-objective this figure changes of total costs and waiting time is visualized inventory routing problem by considering customer against variation of α. satisfaction level in context of perishable product,” in 2014 IEEE Symposium on Computational Intelligence in Production and Logistics Systems (CIPLS), 2014, pp. 91– 97. 236 Study on Reducing Turn-Around Time of Multi-Objective Evolutionary Algorithm on an Industrial Problem Hiroaki Fukumoto Akira Oyama Institute of Space and Astronautical Science, Institute of Space and Astronautical Science, JAXA JAXA 3-1-1, Yoshinodai, Chuo, Sagamihara 3-1-1, Yoshinodai, Chuo, Sagamihara Kanagawa, Japan Kanagawa, Japan fukumoto@flab.isas.jaxa.jp oyama@flab.isas.jaxa.jp ABSTRACT scale many-core architectures [2] and the computational al- Multi-objective evolutionary algorithms (MOEAs) are pop- gorithms, say MOEAs, should utilize the large-scale compu- ulation based global optimization algorithms and it is said tational resources efficiently. One of the simple yet effective that the performance of the MOEAs depends on the pop- ways of MOEAs for utilizing the large-scale computational ulation size. Considering that the recent trends of com- resources would be to increase the number of concurrent so- puter development is in large-scale many-core architectures, lution evaluations, i.e., the population size. Note that the and massive parallel computation is getting feasible in more increased number of objectives of multi-objective optimiza- companies and laboratories, the available population size tion problems also gives a reason to increase the population is increasing and the efficiency of MOEA with large pop- size: the necessary number of solutions to cover the entire ulation size should be enhanced. This study examines the Pareto front exponentially increases as the number of objec- effect of the population size on MOEAs’ performance on tives increases [9, 18]. Therefore, the increase in the popu- a real-world-derived benchmarking optimization problem, lation size would be the right direction for recent MOEAs. with large population size. In this paper, three mate se- lection schemes with different degree of elitist strategy are This study aims to reduce the turn-around time of MOEAs adapted to NSGA-II-M2M. The experimental results show when large population size is used. This paper demonstrates that the elitist strategy can efficiently make use of the ef- the population size effect on the performance of an MOEA fect of the large population size, therefore can reduce the on a real-world-derived benchmarking problem and the re- turn-around time. duction of the turn-around time by making use of the pop- ulation size effect is attempted. This paper is organized Keywords as follows. In Section 2, the experimental settings are ex- multi-objective optimization, large population size, mate se- plained first, and the results demonstrating the impact of lection, real-world problem the population size on the performance of the MOEAs is presented. Then the method to reduce the turn-around time 1. INTRODUCTION is described and the experimental results are provided. Sec- tion 3 concludes this paper. Many of industrial design problems involve multiple objec- tives and constraints and they are so-called constrained multi- objective optimization problems. Considering that creating 2. REDUCTION OF TURN-AROUND TIME high value-added products in industries is getting more and more important along with the increase of the sophistica- 2.1 Experimental setting tion and diversity of social needs, it is very important to • Problem: The Mazda CdMOBP problem [11]. This catch up to the changes in customer demands and so short problem has two objectives, 54 constraints, and 222 development time of each product is highly appreciated. variables. The problem originates from an actual de- sign optimization of car models and the constraints For multi-objective optimization problems, multi-objective comprise the requirements for crashworthiness, body evolutionary algorithms (MOEAs) have been regarded as torsional stiffness, and low frequency vibration modes. a promising approach. With respect to the application of These constraints are evaluated by finite element sim- MOEAs to industrial design problems, the development time ulations on a supercomputer in actual design process, of the products corresponds to the turn-around time for however, in the benchmark problem these simulation MOEAs. The turn-around time of MOEAs corresponds to results are modeled with radial basis functions so as to the number of generations in MOEA, supposing that the shorten the evaluation time while retaining the nonlin- runtime for MOEA itself is negligible compared with the earity as much as possible. The details are presented runtime for solution evaluations. Here the turn-around time in [11] and the problem is available from the website is the time from the beginning of the optimization to the [12]. end of the optimization when a desired quality of solution set is obtained. • MOEA: NSGA-II-M2M [15] with the subproblem size of 10. The probability that the parents are chosen from One of the recent trends of computer development is in large- the corresponding subproblem δ is set to 0 . 9. 237 • Constraint handling technic: Multiple constraint rank- 0.40 ing (MCR) [5], which generally performs well on con- 0.35 strained optimization benchmarking problems [7]. The constraint handling technic is incorporated into NSGA- 0.30 II-M2M with the MOEA-CHT incorporation frame- 0.25 work [7]. N = 100 0.20 • Mate selection schemes: Random selection, binary tour- N = 316 nament (BT) [1], or Elitist BT (EBT, explained in next 0.15 N = 1000 subsection) [8]. The random selection scheme is the 0.10 N = 3162 default mate selection scheme for NSGA-II-M2M [15] Mean hypervolume values and its modified version of Jain et al. [10] is employed 0.05 so as to handle constraints. 0 0.00 60 120 180 240 300 • Reproduction operators: The crossover and mutation Number of generations operators with the same control parameter values as (a) Random selection in [14, 15]. • Direction vector generation method: Das and Dennis’s Figure 1: Convergence history of the HV values with systematic approach [4]. various population sizes for the case with random se- • Stopping criterion: The number of generations of 300. lection. The mean and the standard deviation values The number of fitness evaluations differs according to are plotted. the population size at a given generation, but the focus is in this study is on the reduction of the required generations, and so the differences in the number of 2.4 Reduction of turn-around time by enhanc- fitness evaluations is not considered in this study. ing the population size effect • Independent runs: Each case run for 31 times indepen- The population size affects the diversity of the solutions and dently. the convergence speed, and now it is commonly accepted • The population size N: N is set to be the numbers in that the population size should be large enough to guarantee a geometric progression with a scale factor of 100, and √ the diversity of the solutions while the large population size a common ratio of 10 is used to see the population makes the convergence slow [16, 3, 17, 19]. size effect. Specifically, the population sizes of 100, 316, 1000, and 3162 are used, especially for drawing Considering that the phenomenon of the population size ef- Figure 3. fect can be explained by a term “selection pressure” [19], we attempt to mitigate the slow convergence with large popu- 2.2 Performance Metric lation by somehow strengthening the selection pressure. In this study, a standard and popular mate selection of BT and The hypervolume (HV) indicator [20] is used as the perfor- a recently proposed mate selection scheme with a strong eli- mance indicator. In this study, the solution set used for the tist strategy named EBT [8], both of which have stronger calculation of the HV value is the solutions not only in the selection pressure than the random selection, are employed. final population but also in the external unbounded archive In EBT, i) the usual BT selection is conducted at first for [13], considering that the designers in actual industries use all the solutions in each subproblem then the indices of the MOEAs as design support tools for decision making and so selected solutions are sorted according the number of times use of the unbounded external archive is more practical than the each index is selected. Apart from that, ii) the indices of use of the solutions obtained only at the final generation. For the solutions are also sorted according the solutions’ fitness. calculating the HV value for a generation, non-dominated so- Finally, every sorted indices i) is replaced by the index in ii) lutions are extracted from all the feasible solutions obtained with the same rank order with i), so that the solution with by the generation and are used to calculate the HV value. higher rank is selected more. For further details of EBT, For the details of the formulation for the HV calculation, please refer to [8]. please refer to [11]. The larger the HV value, the better the approximation to the Pareto front. The most elitist is EBT, followed in order by BT and random selection. 2.3 Impact of the population size on the per- formance It must be noted that the strong elitist strategy tends to Figure 1 presents the convergence history of the mean HV deteriorate the diversity of the solutions, and the negative values with various population sizes for NSGA-II-M2M with effect of the strong elitist strategy should be compensated random selection. It is observed that the cases with higher by using some diversity-enhancing method. In this study, we population size show generally higher mean and smaller stan- enhance the MOEAs’ capability of keeping diversity by em- dard deviation values. This result supports the motivation ploying M2M, and this is the reason why the base algorithm for increasing the population size, however, the effect of the in this study is not NSGA-II [6] but NSGA-II-M2M. increased population size is not clearly observed until around the number of generations of 200, between the cases with the Figure 2 shows the convergence history of the mean HV val- population size of 1000 and 3162. ues with various population sizes for NSGA-II-M2M with 238 0.40 0.40 0.35 0.35 0.30 0.30 0.25 0.25 N = 100 0.20 0.20 N = 316 0.15 0.15 N = 1000 0.10 0.10 N = 3162 Mean hypervolume values 0.05 Mean hypervolume values 0.05 0 0.00 60 120 180 240 300 0.00 Number of generations 0 60 120 180 240 300 Number of generations (a) BT (b) EBT Figure 2: Convergence history of the HV values with various population sizes for the cases with BT and EBT selection. The mean and the standard deviation values are plotted. BT and EBT. Comparing Figure 1 and 2, it can be observed will include further improvement of the population size ef- that the strong elitist mate selection enhances the large pop- fect, even with much smaller population size. ulation size effect and the differences in the mean HV values can be observed more clearly and from earlier generations. Acknowledgment This research is supported by the HPCI System Research With regard to the reduction of the turn-around time, Fig- Project “Research and development of multi-objective de- ure 3 shows the generation that is required to attain a HV sign exploration and high-performance computing technolo- value against the population size. For example, in Figure gies for design innovation”. 3, the HV value of 0 . 2 can be attained with the number of (Project ID: hp160203 and hp170238) generations of approximately 300 with the population size of approximately 300, and with the number of generations 4. REFERENCES of approximately 160 with the population size of approx- [1] T. Blickle and L. Thiele. A mathematical analysis of imately 1000. The subfigures in Figure 3 show that the tournament selection. In Proceedings of the Sixth required generation to attain a certain HV value is reduced International Conference on Genetic Algorithms, with stronger mate selection scheme. pages 9–16. Morgan Kaufmann, 1995. Compared with the case with BT, the results for EBT shows [2] S. Borkar. Thousand core chips: A technology relatively poor performance with small population sizes, and perspective. In Proceedings of the 44th Annual Design so further development of more robust and more efficient al- Automation Conference, DAC ’07, pages 746–749, gorithm for reducing the turn-around time will be required. New York, NY, USA, 2007. ACM. [3] T. Chen, K. Tang, G. Chen, and X. Yao. A large population size can be unhelpful in evolutionary 3. CONCLUSIONS algorithms. Theoretical and Computational Science, 436:54–70, 2012. This paper demonstrates the population size effect on the performance of an MOEA on a real-world-derived bench- [4] I. Das and J. E. Dennis. Normal-boundary marking problem (Mazda CdMOBP) and the reduction of intersection: A new method for generating the pareto the turn-around time by making use of the population size surface in nonlinear multicriteria optimization effect is attempted. problems. SIAM Journal on Optimization, 8(3):631–657, 1998. By the demonstration of the population size effect, it is [5] R. de Paula Garcia, B. S. L. P. de Lima, A. C. shown that the large population size can improve the perfor- de Castro Lemonge, and B. P. Jacob. A rank-based mance of an MOEA, and it is also shown that the population constraint handling technique for engineering design size effect is not clearly shown until late stage of the evolu- optimization problems solved by genetic algorithms. tion with random mate selection scheme. Computers & Structures, 187(Supplement C):77 – 87, 2017. The late-appearing population size effect is then improved [6] K. Deb, S. Agrawal, A. Pratap, and T. Meyarivan. A by employing two techniques: a strong mate selection scheme fast and elitist multiobjective genetic algorithm: and its complementary scheme of M2M. The results show NSGA-II. IEEE Transactions on Evolutionary that the case with stronger elitist strategy exhibits relatively Computation, 6(2):182–197, 2002. faster large population size effect and the aim of reducing [7] H. Fukumoto and A. Oyama. A generic framework for turn-around time is achieved in some degree. Future work incorporating constraint handling techniques into 239 (a) Random selection (b) BT (c) EBT Figure 3: Plot for the generation that is required to attain a HV value against population size. multi-objective evolutionary algorithms. In K. Sim sub-regional search. In 2009 IEEE Congress on and P. Kaufmann, editors, Applications of Evolutionary Computation, pages 1928–1934, 2009. Evolutionary Computation, pages 634–649, Cham, [15] H. L. Liu, F. Gu, and Q. Zhang. Decomposition of a 2018. Springer International Publishing. multiobjective optimization problem into a number of [8] H. Fukumoto and A. Oyama. Study on improving simple multiobjective subproblems. IEEE efficiency of multi-objective evolutionary algorithm Transactions on Evolutionary Computation, with large population by m2m decomposition and 18(3):450–455, 2014. elitist mate selection scheme. In 2018 IEEE [16] R. Mallipeddi and P. N. Suganthan. Empirical study Symposium Series on Computational Intelligence on the effect of population size on differential (SSCI), 2018, to appear. evolution algorithm. In 2008 IEEE Congress on [9] H. Ishibuchi, Y. Sakane, N. Tsukamoto, and Evolutionary Computation (IEEE World Congress on Y. Nojima. Evolutionary Many-Objective Computational Intelligence), pages 3663–3670, 2008. Optimization by NSGA-II and MOEA/D with Large [17] D. Mora-Meliá, F. J. Martinez-Solano, P. L. Populations. In IEEE International Conference on Iglesias-Rey, and J. H. Gutiérrez-Bahamondes. Systems, Man and Cybernetics, pages 1758–1763, Population size influence on the efficiency of 2009. evolutionary algorithms to design water networks. [10] H. Jain and K. Deb. An Evolutionary Many-Objective Procedia Engineering, 186:341 – 348, 2017. XVIII Optimization Algorithm Using Reference-Point Based International Conference on Water Distribution Nondominated Sorting Approach, Part II: Handling Systems, WDSA2016. Constraints and Extending to an Adaptive Approach. [18] T. Tatsukawa, T. Watanabe, and A. Oyama. IEEE Transactions on Evolutionary Computation, Evolutionary computation for many-objective 18(4):602–622, 2014. optimization problems using massive population sizes [11] T. Kohira, H. Kemmotsu, O. Akira, and on the k supercomputer. In 2016 IEEE Congress on T. Tatsukawa. Proposal of benchmark problem based Evolutionary Computation (CEC), pages 1139–1148, on real-world car structure design optimization. In 2016. Proceedings of the Genetic and Evolutionary [19] T. Weise, Y. Wu, R. Chiong, K. Tang, and J. Lässig. Computation Conference Companion, GECCO ’18, Global versus local search: The impact of population pages 183–184. ACM, 2018. sizes on evolutionary algorithm performance. Journal [12] T. Kohira, H. Kemmotsu, A. Oyama, and of Global Optimization, 66(3):511–534, 2016. T. Tatsukawa. Mazda/JAXA/TUS simultaneous car [20] E. Zitzler, L. Thiele, M. Laumanns, C. M. Fonseca, structure design problem, and V. G. da Fonseca. Performance assessment of http://ladse.eng.isas.jaxa.jp/benchmark/index.html, multiobjective optimizers: an analysis and review. 2018. IEEE Transactions on Evolutionary Computation, [13] O. Krause, T. Glasmachers, N. Hansen, and C. Igel. 7(2):117–132, 2003. Unbounded population mo-cma-es for the bi-objective bbob test suite. In Proceedings of the 2016 on Genetic and Evolutionary Computation Conference Companion, GECCO ’16 Companion, pages 1177–1184. ACM, 2016. [14] H. l. Liu and X. Li. The multiobjective evolutionary algorithm based on determined weight and 240 Evolution of Electric Motor Design Approaches: The Domel Case Gregor Papa Gašper Petelin Peter Korošec Computer Systems Faculty of Computer and Computer Systems Department Information Science Department Jožef Stefan Institute University of Ljubljana Jožef Stefan Institute Ljubljana, Slovenia Ljubljana, Slovenia Ljubljana, Slovenia gregor.papa@ijs.si peter.korosec@ijs.si ABSTRACT Slovenian smart specialisation strategy [2], it is planned to The paper presents the evolution of geometry design ap- transfer this solution into Slovenian industry. proaches in the optimisation of an electric motor, more specif- ically its rotor and stator. It starts with the initial manual The rest of the paper is organized as follows: Section 2 approach, which was replaced with the automatic approach briefly describes the geometry elements of an electric mo- that introduced evolutionary algorithms to allow the intel- tor and the optimisation goal; Section 3 presents the con- ligent search in collaboration with evaluation tools. Next, ventional manual approach to the motor design; in Section 4 the new platform for remote optimisation was recently in- the use of evalutionary algorithms in electric motor design is troduced that allows remote optimisation with various algo- outlined; Section 5 introduces the new developed platform rithmic approaches, including multi-objective optimisation. for remote optimisation; and Section 6 draws conclusions At the end we propose further solutions that will improve and proposes possible future work. high performance of the design process. Keywords electric motor, design, evolution, high-performance 1. INTRODUCTION Many widely-used home appliances (e.g., mixers, vacuum cleaners, drills, etc.) use electric motors. These small motors are required to have high power and provide high starting and running torques, despite their small sizes. While having sufficient output power they should be energy efficient and inexpensive to manufacture [12]. There is a number of past works addressing the geometry optimisation design of rotor and stator parts [6], [10], [12], electric motor casing [7] and impeller [4]. These works, per- formed on various products of Domel company [1], intro- duced various artificial intelligence methods to implement automatic search of an optimal design. The reported optimi- sation approaches were mostly single objective. Still, there were some initial steps identified towards multi-objective handling of the design process. This paper focuses on the approaches for automatic optimi- sation of the electric motor geometry. The main parts of the electric motor, i.e., stator and rotor, are presented in Figure 1. Figure 1: Rotor and stator of an electric motor [10]. While improving the applicability of the multi-objective op- timisation, supported by parallelisation and surrogate mod- 2. PROBLEM DESCRIPTION elling through the support of the Horizon 2020 Twinning The rotor and the stator of an electric motor are constructed project SYNERGY - Synergy for smart multi-objective op- by stacking the iron laminations. The shape of these (rotor timisation [3], we implemented a platform for an efficient and stator) laminations is described by several geometry pa- optimisation with different methods and approaches. The rameters that define the rotor and stator in two dimensions platform is briefly presented in this paper. In line with (2D). 241 The whole set of geometry parameters consists of invariable and variable ones. Invariable parameters are fixed, as they cannot be altered, either for technical reasons (e.g., the air rer gap) or because of the physical constraints on the motor (e.g., the radius of the rotor’s shaft). Variable parameters, on the other side, do not have predefined optimal values. ryt Among these parameters, some are dependent (upon oth- ers variables), while some variable parameters are mutually independent and without any constraints. The mutually in- rpw dependent set of variable parameters of the rotor and stator a) geometry (see details in Figure 2) can be subject to optimi- sw sation: syh ssr • rotor yoke thickness (ryt), • rotor external radius (rer), • rotor pole width (rpw). • stator width (sw), sml • stator yoke horizontal thickness (syh), str • stator yoke vertical thickness (syv), • stator middle part length (sml), • stator internal edge radius (sie), • stator teeth radius (str), sie • stator slot radius (ssr). syv b) One of the optimisation tasks is to find the values of geom- etry parameters that would generate the rotor and stator Figure 2: Geometry parameters of a) rotor and b) geometry with minimum power losses. stator [12]. 2.1 Mathematical formulation of the problem The efficiency of an electric motor is defined as the ratio of two components: the eddy-current losses and the hysteresis the output power to the input power. It depends on various losses: power losses (see details in [9]), which include: • Copper losses: the joule losses in the windings of the PF e = keB2f 2 rotmrot+keB2f 2 statmstat+khB2f 2 statmstat (2) stator and the rotor. • Iron losses: including the hysteresis losses and the where ke is an eddy-current material constant of 50 Hz, kh is eddy-current losses, which are primarily in the arma- a hysteresis material constant of 50 Hz, B is the maximum ture core and in the saturated parts of the stator core. magnetic flux density, f is the frequency, and m is the mass. • Other losses: brush losses, ventilation losses and fric- Three additional types of losses also occur, i.e., brush losses tion losses. PBrush, ventilation losses PV ent, and friction losses PF rict. The output power P The overall copper losses (in all stator and rotor slots) are 2 of the motor is a product of the elec- tromagnetic torque T , and the angular velocity ω, as follows: X P P 2 = T ω (3) Cu = (J 2Aρlturn)i (1) i where ω is set by the motor’s speed, and T is a vector prod- where i stands for each slot, J is the current density, A is uct of the distance from the origin r, and the electromagnetic the slot area, ρ is the copper’s specific resistance and l force F . turn is the length of the winding turn. The overall efficiency of an electric motor is defined as: Due of the non-linear magnetic characteristic, the calcula- P2 tion of the iron losses is less exact; they are separated into η = (4) P2 + PCu + PF e + PBrush + PV ent + PF rict 242 2.2 Fitness evaluation 3. For evaluation of each solution (i.e., their fitness) each Each solution candidate of the population was decoded into geometrical configuration is analyzed using some FEM a set of the rotor and stator parameters. The fitness was program (e.g., ANSYS). This step requires a decoding estimated by performing a finite-element numerical simula- of the encoded parameters into a set of geometrical tion to calculate the iron and the copper power losses (using parameters that define the rotor and the stator. the above mentioned equations). The sum of power loses 4. After the fitness calculation, the reproduction of the corresponds to the solution’s fitness. individual solutions is performed and the application of various recombination operators to a new population For multi-objective version we can also introduce additional are done. objective like material costs, making it a typical price/ per- formance optimisation. The cost is calculated by taking into 5. The evolutionary algorithm repeats the above proce- account the amount of materials (i.e., iron and copper), that dure until some predefined number of iterations have are used to produce the electric motor, and their correspond- been accomplished or some other stopping criteria is ing prices. met. 3. MANUAL OPTIMISATION A manual design procedure of an electric motor consists of Some evident advantages of this approach are: the geometry estimation of the rotor and the stator of an electric motor by an experienced engineer. The suitability • There is no need for an experienced engineer to be of the proposed geometry is usually analyzed by means of present during the whole process. He is required only numerical simulation (e.g., FEM with an automatic finite- at the beginning to decide on the initial design. element-mesh generation) of the electromagnetic field of each proposed solution separately. • There is no need to know the mechanical and physical details of the problem. The problem can be solved, by The manual procedure can be repeated until the satisfied the use of optimisation algorithm, irrespective of any evaluation results is obtained. Similarly, the conventional knowledge about the problem. approach in most new designs starts with manual design, as there exist no prior design. Some possible drawbacks of this approach can appear: The advantage of the manual approach is that the engineers can significantly influence the progress of the design pro- • The improper use of recombination operators leads to cess with their experiences and react intelligently to any slow search progress. noticeable electromagnetic response with proper geometry redesign. • An initial solution set that is not divergent enough, can lead to a longer convergence time. The drawback of this approach is that an experienced en- gineer and a large amount of time (that is mostly spent on 5. REMOTE OPTIMISATION PLATFORM computation) are needed. The multi-objective optimisation is a natural approach to solve difficult real-world problems. As the presented elec- 4. AUTOMATIC OPTIMISATION tric motor geometry design can have several contradictory The above-described manual design approach can be up- constraints, it is useful to introduce the multi-objective al- graded with one of the stochastic optimisation techniques gorithms (e.g., NSGA-II, IBEA) into this process [11]. which, in connection with reliable numerical simulators, al- low for highly automated design process where the need for Within the project SYNERGY, we developed and imple- an experienced engineer to navigate the process is signifi- mented a platform for an efficient optimisation with different cantly reduced. methods and approaches. Its main role is to allow compari- son and testing of an effective optimisation methods for the So far, several evolutionary approaches have already been optimisation of electric motor geometry. The platform al- proven to be efficient in the process of the electric motor ge- lows comparison of single objective as well as multi-objective ometry optimisation; e.g., electromagnetism-like algorithm algorthms. [5], multi-level ant-stigmergy algorithm [6], adaptive evolu- tionary search algorithm [8], genetic algorithm [9], particle The platform is based on web-based services to allow remote swarm optimization, and differential evolution [12]. work of different experts, while keeping some important, se- cret features and characteristics hidden. The remote tool The automatic approach with the use of an evolutionary also allows for parallel processing, which allows for fast cal- algorithm can be summarized into the following steps: culations, without any intervention from the expert. Remote access enables experts to use the evaluation of the 1. The initial set of solutions is defined according to an proposed solution regardless of his location. The platform initial electric motor. allows remote access towards any simulation tools (e.g., FEM 2. It provides a set of problem solutions (i.e., different analysis). Furthermore, all evaluations are being stored in configurations of the mutually independent geometri- database and in case the same solution is being put to eval- cal parameters of the rotor and the stator). uation, the result is immediately returned without the need 243 to wait for it to be actually evaluated again, which furthers [2] Slovenia’s Smart Specialisation Strategy S4. speed up the evaluation process. http://www.svrk.gov.si/fileadmin/svrk.gov.si/ pageuploads/Dokumenti za objavo na vstopni strani/ Since actual parameter values are not relevant for optimisa- S4 strategija V Dec17.pdf. Accessed: 2018-08-26. tion process and to ensure that no secrets about the problem [3] SYNERGY – Synergy for Smart Multi-Objective are being shared, the platform hides the important prop- Optimisation. http://synergy-twinning.eu/. Accessed: erties of the solutions. Meaning all parameter values and 2018-08-26. evaluation results are being normalised within the interval [4] P. Korošec, J. Šilc, K. Oblak, and F. Kosel. [0.0, 1.0]. This way, the problem can be tackled by any op- Optimizing the shape of an impeller using the timisation expert without acquiring any relevant knowledge differential ant-stigmergy algorithm. In (e.g., actual dimensions, problem specifications) about the R. Wyrzykowski, J. Dongarra, K. Karczewski, and problem. J. Wasniewski, editors, Parallel Processing and Applied Mathematics, pages 520–529, Berlin, Parallelisation within the platform is considered on the level Heidelberg, 2008. Springer. of solution evaluation. Any other parallelisation on the level [5] P. Korošec, G. Papa, and J. Šilc. Optimization of optimisation algorithm is left to the optimisation expert. algorithms inspired by electromagnetism and stigmergy in electro-technical engineering. WSEAS 6. CONCLUSION Transactions on Information Science and This paper presented the evolution of approaches to the Applications, 5(2):587–591, May 2005. optimisation of the geometry design of the electric motor. [6] P. Korošec and J. Šilc. The distributed multilevel From the initial manual approach, through the automatic ant-stigmergy algorithm used at the electric-motor approach that uses some evolutionary algorithm combined design. Engineering Applications of Artificial with evaluation tools, towards the platform that allows re- Intelligence, 21(6):941–951, 2008. mote optimisation with various algorithms. The latter al- [7] K. Oblak, P. Korošec, J. Šilc, and F. Kosel. Stigmergic lows simple comparison and study of different methodologies optimization of plane constructions. Electrotehniški and algorithms. vestnik, 74(5):279–284, 2007. In Slovenian. [8] G. Papa. Parameter-less algorithm for In the future version of the optimisation platform we plan to evolutionary-based optimization. Computational introduce some surrogate models as well as some multi-level Optimization and Applications, 56(1):209–229, 2013. approaches, which would allow for additional speed up of [9] G. Papa and B. Koroušić-Seljak. An artificial the evaluation process, since most of the real-world problems intelligence approach to the efficiency improvement of have time-complex evaluations. a universal motor. Engineering Applications of Artificial Intelligence, 18(1):47–55, February 2005. 7. ACKNOWLEDGMENTS [10] G. Papa, B. Koroušić-Seljak, B. Benedičič, and The authors acknowledge the financial support from the T. Kmecl. Universal motor efficiency improvement Slovenian Research Agency (research core funding No. P2- using evolutionary optimization. IEEE Transactions 0098). This work is also part of a project SYNERGY that on Industrial Electronics, 50(3):602–611, June 2003. has received funding from the European Union’s Horizon [11] G. Petelin and G. Papa. Application for multi-criteria 2020 research and innovation programme under grant agree- optimization and visualization of solutions. Technical ment No 692286. Report 12380, Jožef Stefan Institute, 2017. [12] T. Tušar, P. Korošec, G. Papa, B. Filipič, and J. Šilc. 8. REFERENCES A comparative study of stochastic optimization [1] Domel company. http://www.domel.com/. Accessed: methods in electric motor design. Applied Intelligence, 2018-09-17. 27(2):101–111, 2007. 244 Model-Based Multiobjective Optimization of Elevator Group Control Aljoša Vodopija1,2, Jörg Stork3, Thomas Bartz-Beielstein3, Bogdan Filipič1,2 1Department of Intelligent Systems, Jožef Stefan Institute, Jamova cesta 39, SI-1000 Ljubljana, Slovenia 2Jožef Stefan International Postgraduate School, Jamova cesta 39, SI-1000 Ljubljana, Slovenia 3TH Köln, Institute of Data Science, Engineering, and Analytics, Steinmüllerallee 1, D-51643 Gummersbach, Germany {aljosa.vodopija, bogdan.filipic}@ijs.si, {joerg.stork, thomas.bartz-beielstein}@th-koeln.de ABSTRACT Sahin et al. [6], a real-time monitoring system is installed to Finding a suitable control strategy for the elevator group reduce the number of redundant stops, and improve passen- controller (EGC) is a complex optimization problem with ger comfort and energy consumption. In [1], an approxima- several objectives. We utilize the sequential-ring (S-Ring) tion model for EGC systems, the so-called sequential ring model of EGC systems and propose a biobjective formula- (S-Ring) [4], is used to benchmark single-objective heuris- tion of the EGC optimization problem. Unlike the previous tics. Using the S-Ring model, it is possible to retain a high work, we use true multiobjective optimizers in solving this level of complexity and optimize an EGC control strategy problem. Their results on three real-world elevator systems using modern heuristics with a high number of strategy eval- reveal the possible trade-offs between the objectives and of- uations, while keeping a feasible computational load. fer a valuable insight into the problem. In this paper, we utilize the S-Ring model of EGC systems Keywords and propose a biobjective formulation of the EGC optimiza- tion problem. In this formulation, the objectives are normal- elevator group control, S-Ring, perceptron, multiobjective ized to allow for comparison of results for elevator systems of optimization, NSGA-II, DEMO various configurations. As opposed to previous work, we ap- ply true multiobjective optimizers capable of finding approx- 1. INTRODUCTION imations for Pareto-optimal solutions that represent trade- With larger number of people living in urban areas and offs between the objectives. Specifically, we use two multiob- modern barrier-free building design, elevator systems are jective evolutionary algorithms (MOEAs) and demonstrate becoming more and more important. Modern multi-car el- their performance in optimizing EGC for three real-world evator systems are controlled by elevator group controllers elevator systems. (EGC) that assign elevator cars to their destinations based on the customer service calls. The control strategy strongly The paper is further organized as follows. Section 2 intro- affects the desired service quality, customer satisfaction, en- duces the S-Ring model, explains its elements and illustrates ergy consumption, and material attrition. Thus, finding an it with an example. Section 3 provides the optimization adequate control strategy depicts a complex optimization problem formulation. In Section 4, numerical experiments problem with several objectives, which is further dependent on the three test elevator systems and the results are pre- on the building structure and the passenger traffic situa- sented. Section 5 concludes the paper by summarizing the tion. Optimization of EGC imposes challenges, such as be- study and planning future work. ing nonlinear and multimodal, as well as highly dynamic and stochastic due to the stochasticity of customer arrivals. 2. S-RING MODEL OF ELEVATOR GROUP This renders classic gradient-based optimizers as not appli- CONTROL cable to these problems [1]. Moreover, EGC simulators are The S-Ring is a discrete, nontrivial event system to optimize computationally expensive and limit the number of control and benchmark control strategies without the need to use strategy evaluations. expensive EGC simulators [4]. It focuses on modeling the operation of an elevator system, i.e., handling the passenger While EGC optimization problems are widely discussed and traffic and serving passengers in the fastest and most com- known for involving conflicting objectives, they are seldom fortable way. We adapted the S-Ring model to feature two solved with true multiobjective optimization. Hakonen et service quality related objectives as described in Section 3. al. [3] utilize a set of objectives, such as the customer waiting time, the ride time, and the total number of elevator stops, In general, the S-Ring consists of three key elements: but combine them linearly into a single objective. Tyni and Ylinen [7] use a weighted aggregation method to optimize the landing call waiting time and energy consumption with • The deterministic state-space representation of the el- an evolutionary algorithm in a real-time environment. In evator control inputs for the customers ci and elevator 245 cars si, i = {1, ..., Ns}, where Ns = 2n − 2 is the the average number of states with waiting customers, and number of states, while n is the number of floors. Fig- ii) the total number of elevator stops [3, 6, 7]. In contrast to ure 1 shows an example of this state-space representa- previous publications, we do not combine the objectives into tion. The size of the S-Ring depends on the number a single function, but adopt the multiobjective perspective. of floors n, and the number of active elevator states is Moreover, to make it possible to compare the performance equal to the number of elevator cars m. The number of elevator systems of various configurations (determined by of currently active customer states is influenced by the the number of floors n and the number of elevator cars m), probability of a new arriving customer, p. we consider normalized objective function values. • The state transition table, which is explained in detail The first objective (h1) is the proportion of states with wait- by Markon [4], defines fixed and dynamic rules for a ing customers. It is expressed as the average number of transition in the current position of the S-Ring. If no states with waiting customers, Mw, divided by the number fixed rule is triggered, the dynamic rules decide how of all states, Ns: the state transition is performed. They are established M by a control policy. w h1 = . (1) Ns • The control policy π can be realized by a lookup table, The second objective (h2) is the proportion of elevator stops. but as its size grows exponentially with n, it is main- It is equal to the total number of elevator stops, Mt, divided tained by a perceptron with a weight vector of length by the maximum possible number of elevator stops. The |w| = 2Ns. The perceptron represents the most ele- latter can be calculated as the number of elevator cars m mentary implementation of neural network (NN). For multiplied by the number of EGC simulation cycles, which a given setup of n, m and p the objectives are only is in turn equal to the number of state transition steps, Nt, influenced by the weight vector w of the NN controller divided by the number of states, Ns, therefore and the number of state transition steps, Nt. At each M state, it is first checked whether a new customer ar- t h2 = . (2) rived. Next, if the current state is an active elevator mNt/Ns state, the controller determines whether the elevator Intuitively, the customers’ discomfort with long waiting times car stops or continues to the next state. Finally, the and long riding times due to many elevator stops does not indication of the customer active state is updated de- increase linearly with time, but more drastically. To model pending on whether or not the customers were served. this effect, we have additionally modified the original objec- tives as f1 = hα 1 and f2 = hβ, (3) 2 3rd floor where α, β ∈ [1, 2] are the objective function coefficients. The choice of their values is subjective, but the idea is to reflect the elevator system characteristics and the custumer 2nd floor preferences. customer ci The control policy π is represented by a perceptron as π(x) = θ(wTx), where x is a binary input vector, i.e., a concate- elevator si nation of the waiting customer and the elevator car state 1st floor vectors of total length equal to 2(2n − 2) = 4(n − 1), θ is the Heaviside function, and w a vector of perceptron weights from W = [−1, 1]4(n−1). In this framework, the policy π is ground floor defined by the weight vector w only. Therefore, the decision space of the EGC optimization problem as defined here is Figure 1: S-Ring: No waiting customer at the equal to W . ground and floor (“0”), two customers who want to go up on the first and second floor (“1,1”), and no 4. EXPERIMENTS AND RESULTS customers who want to go down on the third, sec- The multiobjective optimization of EGC was experimentally ond and first floor (“0,0,0”). By combining these evaluated on three test problems reflecting the characteris- information we obtain the following state vector for tics of real-world elevator systems operating in various build- waiting customers: (0,1,1,0,0,0). The state vector ings in Ljubljana, Slovenia. They are as follows. for the elevator is obtained in a similar manner. • S1: This system operates in a parking building (“Park- Due to its low computational costs, the S-Ring can quickly ing garage Šentpeter”) situated in the city center. In- evaluate a broad variety of EGC instances as benchmarks tensive passenger traffic can be observed in the build- for the proposed multiobjective optimization approach. ing on workdays. • S2: This is an elevator system installed in a typical 3. OPTIMIZATION PROBLEM FORMULA- residential building in the densely populated neighbor- TION hood (“Nove Fužine”) in the eastern part of Ljubljana. In this work, we deal with two EGC objectives that are often Here the traffic intensity alternates between high (e.g., studied in the literature and both need to be minimized: i) early in the morning) and low (e.g., at midday). 246 • S3: This is the elevator system in the “Crystal Palace”, robust and repeatable algorithm behavior on all three ele- a skyscraper situated in the north-western area of the vator systems. Similarly, small deviations are present for city. With its 89 meters it is currently the tallest build- execution times no matter which MOEA is used to produce ing in Slovenia. As an office building it has low pas- approximations for Pareto fronts. senger traffic. Figures 2 and 3 show Pareto front approximations for the test elevator systems resulting from typical runs of NSGA- The characteristics of these elevator systems are summarized II and DEMO, respectively (there were negligible differences in Table 1. between the results of different runs). As we can see, both MOEAs obtain well-distributed and very similar sets of so- Table 1: Characteristics of the test elevator systems: lutions. The best solutions with respect to both objectives number of floors n, number of elevator cars m, prob- were found for system S3. This was expected since S3 has ability of new arriving customer p, objective func- more elevator cars and a lower probability of new customer tion coefficients α and β, number of states in the arrivals than S1 and S2. S-Ring representation Ns. System n m p α β Ns 0.6 System S1 7 2 0.6 1.0 1.5 12 S1 S2 13 2 0.3 1.4 1.8 24 S2 S3 21 4 0.2 1.5 1.5 40 ator stops S3 v 0.4 Based on the multiobjective formulation of the EGC op- timization problem, the experimental evaluation aimed at tion of ele finding sets of trade-off solutions representing approxima- 0.2 tions for Pareto fronts. For this purpose we used two well- known MOEAs: Nondominated Sorting Genetic Algorithm f2: Propor II (NSGA-II) [2] and Differential Evolution for Multiobjec- tive Optimization (DEMO) [5]. The algorithms were as- 0.0 sessed from the point of view of both effectiveness (quality 0.7 0.8 0.9 1.0 of results) and efficiency (spent computational resources). f1: Proportion of states with waiting customers The experimental setup was defined in the following way. Figure 2: Pareto front approximations for the test Both algorithms were run with populations of 100 solutions elevator systems produced by NSGA-II. for 100 generations. Specifically, in NSGA-II, the crossover probability was set to 0.7 and the mutation probability to 0.2, while DEMO was run using the SPEA selection proce- 0.6 System dure, the crossover probability of 0.3 and the scaling factor S1 of 0.5. On each test problem every MOEA was run 30 times, S2 each time with a new randomly initialized population. ator stops S3 0.4 v Population members were the perceptron weight vectors of length 2Ns = 4(n − 1). Each solution was evaluated through a computer simulation of the perceptron EGC during which tion of ele the values of objectives f 0.2 1 and f2 were calculated. The sim- ulation was performed for a predefined number of simulation cycles which was 100.000 for all test problems. As a conse- f2: Propor quence, the number of state transition steps was equal to 0.0 Nt = 100.000Ns. 0.7 0.8 0.9 1.0 f1: Proportion of states with waiting customers The quality of results of an algorithm run was measured with the hypervolume of the Pareto front approximation found in Figure 3: Pareto front approximations for the test that run. Given f1, f2 ∈ [0, 1], the reference point for hyper- elevator systems produced by DEMO. volume calculations was set to (1.1, 1.1)T. As the compu- tational efficiency measure the execution time of algorithm runs was recorded. The experiments were run on a 3.50 GHz An additional experiment was devoted to the analysis of Intel(R) Xeon(R) E5-2637V4 CPU with 64 GB RAM. hypothetical variants of system S3 with various numbers of elevator cars. While S3 has its fixed configuration, such The hypervolume and execution time results are shown in a study is relevant for designing elevator systems for new Table 2, both averaged over 30 runs of every MOEA on each buildings and assessing potential configurations. test problem. From these results it is evident that regard- less of the elevator system, the hypervolumes obtained with Pareto front approximations obtained with NSGA-II for vari- NSGA-II and DEMO are very similar. Standard deviations ants of S3 with 2, 3, 4, 5 and 6 elevator cars are presented for both optimizers are small (less than 10−3), indicating in Figure 4. The figure clearly shows how the number of 247 Table 2: Average hypervolume and average execution time for both optimizers on the test elevator systems. NSGA-II DEMO Elevator system Hypervolume Time [min] Hypervolume Time [min] S1 0.28066 ± 0.00005 38 ± 1 0.28069 ± 0.00005 28 ± 1 S2 0.32455 ± 0.00016 147 ± 1 0.32450 ± 0.00014 128 ± 1 S3 0.46506 ± 0.00081 398 ± 2 0.46543 ± 0.00037 401 ± 2 In the future we plan to further assess the resulting elevator 0.4 No. of control policies through a comparisson with the results of elevators single-objective optimization and investigate the scalability 2 of the applied optimization methodology. We will also an- 0.3 3 ator stops alyze the produced trade-off solutions in the design space, v 4 and deal with alternative, potentially more transparent pol- 5 icy implementations. 0.2 6 tion of ele 6. ACKNOWLEDGMENTS 0.1 This work is part of a project that has received funding from the European Union’s Horizon 2020 research and innovation f2: Propor programme under grant agreement no. 692286. We acknowl- 0.0 edge financial support from the Slovenian Research Agency 0.5 0.6 0.7 0.8 0.9 1.0 (young researcher programme and research core funding no. f1: Proportion of states with waiting customers P2-0209). Figure 4: Pareto front approximations for variants 7. REFERENCES of S3 with different numbers of elevator cars (2, 3, [1] T. Bartz-Beielstein, M. Preuss, and S. Markon. 4, 5, 6) found by NSGA-II. Validation and optimization of an elevator simulation model with modern search heuristics. In Metaheuristics: Progress as Real Problem Solvers, cars affects the trade-off EGC policies. Higher number of pages 109–128. Springer, 2005. cars implies policies that can reduce the proportion of states [2] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan. A with waiting customers and the proportion of elevator stops fast and elitist multiobjective genetic algorithm: simultaneously. For example, in the case of only 2 elevator NSGA-II. IEEE Transactions on Evolutionary cars the lowest value of objective f1 is about 0.8, while with Computation, 6(2):182–197, 2002. 6 elevator cars it can be reduced to 0.5. However, one should [3] H. M. Hakonen, A. Rong, and R. Lahdelma. be careful in comparing the results with respect to f2, since Multiobjective optimization in elevator group control. the maximum possible number of elevator stops increases In P. Neittaanm with the number of elevator cars. Nevertheless, these results äki, T. Rossi, S. Korotov, E. O˜ nate, J. Périaux, and D. Kn allow for better problem understanding and are insightful to örzer, editors, European Congress on Computational Methods in Applied Sciences and various stakeholders involved in deciding on elevator system Engineering, ECCOMAS 2004, Jyv configurations, ranging from EGC designers to investors. äskylä, 2004. 17 pages. [4] S. Markon. A solvable simplified model for elevator 5. CONCLUSIONS group control studies. In 2015 IEEE 4th Global We studied the optimization of EGC needed in the design Conference on Consumer Electronics (GCCE), pages and operation of multi-car elevator systems. Utilizing the S- 56–60, 2015. Ring model of EGC systems, we proposed a biobjective for- [5] T. Robič and B. Filipič. DEMO: Differential evolution mulation of the EGC optimization problem that takes into for multiobjective optimization. In C. A. Coello Coello, account the proportion of states with waiting customers and A. Hernández Aguirre, and E. Zitzler, editors, the proportion of elevator stops, both subject to minimiza- Proceedings of the Third International Conference on tion. In this formulation, the objectives are normalized to Evolutionary Multi-Criterion Optimization, EMO 2005, support comparative empirical studies on elevator systems Guanajuato, Mexico, volume 3410 of Lecture Notes in with various numbers of floors and elevator cars. Computer Science, pages 520–533. Springer, 2005. [6] Y. G. Sahin, S. Uzunbayir, B. Akcay, and E. Yildiz. As opposed to previous work, we applied true multiobjec- Real-time monitoring of elevators to reduce redundant tive optimizers capable of finding approximations of Pareto- stops and energy wastage. In Proceedings of the 2013 optimal solutions. The results of two MOEAs for three real- International Conference on Systems, Control and world elevator systems were comparable regarding both the Informatics, pages 264–269, 2013. quality and execution time. They revealed the nondom- [7] T. Tyni and J. Ylinen. Evolutionary bi-objective inated sets of trade-off control policies for the considered optimisation in the elevator car routing problem. elevator systems. Moreover, we demonstrated how the ap- European Journal of Operational Research, proach can be used to support the elevator system configur- 169(3):960–977, 2006. ing at the design stage. 248 From a Production Scheduling Simulation to a Digital Twin Gregor Papa Peter Korošec Jožef Stefan Institute Jožef Stefan Institute Jamova cesta 39 Jamova cesta 39 Ljubljana Slovenia Ljubljana Slovenia gregor.papa@ijs.si peter.korosec@ijs.si ABSTRACT If the approach would be left as is, it would be considered Digital twins are becoming ever more important in smart only as multi-objective approach using a simulation tool to specialisation of factories of the future. Transition from find an approximation set of non-dominated solutions. But using current state in industry to using digital twins is a since it can be introduced into the actual production, mean- big step. We propose an initial step to upgrade simulations ing that the current information of the state of production, to digital twins to enhance the productivity even further. with regard to standing orders and orders which have al- The multi-objective optimisation approach is important in ready been processed so far, we can consider such an en- achieving high efficiency of production scheduling. The goal hanced simulation model to be a digital twin of the produc- of the optimisation is to find a production schedule that tion. With it, we could not only simulate theoretical future satisfies different, contradictory production constraints. We capacities, but also include actual production and its daily take a simulation tool that was used by a memetic version specifics to predict future events with higher accuracy. of the Indicator-Based Evolutionary Algorithm with cus- tomized reproduction operators and local search procedures The rest of the paper is organized as follows: in Section 2, to find a set of feasible, non-dominated solutions and anal- we briefly describe the production scheduling problem; in yse the required steps to achieve a digital twin. We show Section 3, we introduce required changes to create a digi- that with a multi-objective approach that is able to find tal twin; in Section 4, we present the main idea of Memetic high-quality solutions and flexibility of many “equal” solu- IBEA; in Section 5, we present the experimental environ- tions, the digital twin becomes a powerful tool for a decision ment and the results of the evaluation with the real-world maker. data; in Section 6, we present the usability study; and in Section 7, we draw conclusions and propose possible future Keywords work. multi objective, scheduling, optimisation, real world, digital twin 2. PRODUCTION SCHEDULING 1. INTRODUCTION PROBLEM Since production scheduling is important for smart special- The scheduling problem was introduced for a company that isation goals in factories of the future, we decided to take produces components for domestic appliances, including hot relevant results from [4], and apply them to see the impact plates, thermostats and heating elements. The fabrication of digital twins. A digital twin is a digital copy of physical process for components used in different types of plates is world (physical twin) in form of processes and systems. It similar, but due to clients’ demands the models differ in size provides both, the elements and dynamics of the real-word, (height, diameter), connector type, and power characteris- so one can simulate and predict the future events with an tics (wattage). For logistic reasons, the clients group dif- up-to-date model, which is relevant for a decision maker. ferent models of plates within the same order, implying the same due-dates for different products. As a consequence, In [4] we applied the multi-objective approach that uses spe- their production must be scheduled very carefully to fulfil cific local search procedures to the problem of production all the demands (quantities and due-dates), to maintain the scheduling. As the basic algorithm we used the Indicator- specified amounts of different models in stock, to optimally Based Evolutionary Algorithm (IBEA) [8]. We decided to occupy their workers, and to make efficient use of all the use the IBEA because it was shown that it can substantially production lines. The assignment of due-dates is usually outperform results generated by other multi-objective algo- performed separately and before the production scheduling, rithms, such as the improved Strength Pareto Evolutionary but since there are strong interactions between the two tasks, Algorithm [9] and NSGA-II [2], in terms of different per- using the proposed digital twin can allow for more accurate formance measures [8]. Due to the addition of local search arrangement of due-dates. For each order, the completion procedures, we called our approach the Memetic Indicator- time should be as close as possible to the due-date in order Based Evolutionary Algorithm (M-IBEA). As such it repre- to reduce the waiting time and costs [7]. Furthermore, not sents a synergy of the multi-objective evolutionary approach all the production lines are equal, since each of them can with separate, individual, learning or local improvement pro- produce only a few different models. A detailed formulation cedures (local searches). of the production scheduling problem is presented in [5]. 249 The required inputs to such a problem are: An indicator function assigns each Pareto-set approxima- tion a real value that reflects its quality. The optimisation goal becomes the identification of a Pareto-set approxima- • Production norms that specify which products are be- tion that minimizes an indicator function. The main advan- ing produced on each line and what is the changeover tage of the indicator concept is that no additional diversity- time from one product to another for each specific line. preservation mechanisms are required [1]. • Amount of stock for each product. The detailed description of the memetic IBEA can be found • Orders that need to be processed and their deadlines. in [4], but the main idea is presented as a pseudo code in Algorithm 1. In our implementation of the basic version, the • Number of planned shifts. IBEA is used to guide the local search procedures. Since we • Number of lines. are dealing with a combinatorial problem, we implemented problem-specific versions of the crossover and mutation op- erators. Additionally, we added different local search proce- Looking from the perspective of a simulation tool that is dures to enhance the efficiency of the algorithm. able to take into account all this inputs and evaluate the expected time of production for every order, it is a simple Algorithm 1 Memetic IBEA simulation tool. But such a tool alone lacks the dynamics of the real world, so it is not able to react “instantly” to the 1: SetInitialPopulation(P ) 2: Evaluate(P ) changes in the production environment. 3: while not EndingCondition() do 4: P 0 = MatingSelection(P ) 3. DIGITAL TWIN 5: Crossover(P 0, pc) For a simulation tool to become a digital twin, some capa- 6: Mutation(P 0, pm) bilities need to be added. Mainly, the interaction between 7: Evaluate(P 0) what is happening in the real world and the description of 8: LocalSearch(P 0) the problem instance. First of all, the relevant information, 9: P = CalculateFitness(P ∪ P 0) which defines the problem instance, can be gathered from 10: P = RemoveWorse(P ) the company’s information system. This allows receiving 11: end while up-to-date information about new orders, the current stock, and amount of products that were produced so far in the Compared to the basic version of the algorithm, the main day. With the way production companies are working, usu- difference is in the procedure LocalSearch(P 0). Here, not ally this needs to be done only once a day, since production only one but many problem specific local search procedures plans do not change for the current day (actually they are are applied [4]. fixed for up to several days in advance), due to the require- ments of having the required materials for producing orders Such a version of the algorithm is suitable for running a sim- at hand. The main reason for this is that an additional re- ulation based approach, but it lacks the required dynamicity quirement is also to have the stock of materials at the factory to actively adapt to changes in the production environment. as small as possible. We must be aware that any unneces- Two things need to change, first, the changes in the pro- sary stock is actually an expense that every company would duction environment should be transferred to the algorithm like to reduce or even remove. solution space, and second, the algorithm should be able to detect and adapt to such changes. Since the production is The simulation tool only takes into account the technical not a living system that changes every second and requires data provided by the company with regard to the above men- immediate changes (as mentioned above, the production is tioned required inputs. Though any changes in production fixed for several days in advance) this is not a crucial aspect, can be “detected” by the simulator through changes in in- since this changes could be applied to the algorithm on a puts (e.g., how many products were actually produced), this daily basis. But from the point of view of acquiring new or- does not provide a good baseline for predicting future pro- ders and providing potential deadlines to the customers, this duction with inclusion of predicting maintenance. For pre- is another matter. By providing a more dynamic system, a diction maintenance to be included in the digital twin a ma- product sales person could easily insert a new potential or- chine learning techniques should be used to estimate/model der and determine what would be the most efficient and safe any informalities that happen, but are not included in pro- deadline to be offered to the customer. And if a customer re- duction norms (e.g., failures on lines). All this is based on quires an earlier deadline, which could force other orders to previous experiences and requires to gather lots of data, so be put in jeopardy of missing the deadline, it allows a prod- the machine learning algorithm is able to be trained to de- uct sales person to better estimate the required higher price tect abnormal, correlated patterns in production, which will for covering the costs ocured from delays of other orders. lead to better predicting future production and provide in- The use of machine learning would also cover the irregular- sight into preventing maintenance, which will lead to further ities that happen in production. reducing of delays on production lines due to failures by ap- plying maintenance before a defect happens. 5. CASE STUDY 4. MEMETIC IBEA 5.1 Test cases The IBEA is a multi-objective version of a genetic algorithm, The algorithm was tested on two real order lists from the where the selection process is based on quality indicators. production company. Task 1 consisted of n = 470 orders 250 Table 1: Comparison of the BF (12 threads) and Table 2: Results of optimisation for Task 1. M-IBEA approach (1 thread). Statistics norders nworkers tchange ndays Evaluations Time Pareto Pareto min 18 631 353 127 n BF M-IBEA BF M-IBEA matching Pareto max 88 823 867 681 7 3.94 · 108 3.5 · 104 22 s 17 s 4/4 Single-objective 18 767 714 156 8 1.58 · 1010 5 · 105 15 min 33 s 5/5 9 7.09 · 1011 5 · 106 11 h 5 min 15/15 Table 3: Results of optimisation for Task 2. Statistics norders nworkers tchange ndays for 189 different products and Task 2 consisted of n = 393 Pareto min 16 538 355 59 orders for 175 different products. The number of orders n Pareto max 50 778 433 330 represents the problem dimension, with m = 5 representing Single-objective 15 702 443 155 the number of available production lines. To mimic the digital twin which is being updated with in- single-objective solution. Though we used the same num- formation once a day (after the end of the daily production) ber of evaluations, this single-objective solution does not we ran a task overnight and looked at the results. In this stand out with respect to any objective – quite the oppo- time, the algorithm made about 300 million evaluations, so site is the case. This can also be observed from Table 2, this was set as our stopping criterion for future tests. A where the single-objective solution returns an average qual- lexicographic evaluation [6] was used for presenting multi- ity solution on all the objectives except n objective solutions. In the simulation evaluation, the num- orders. The results are summarised in Tables 2 and 3, where the width of the ber of delayed orders (norders) was set as the most impor- Pareto approximation front is denoted with “Pareto min” tant objective, followed by the required number of workers and “Pareto max”. (nworkers), the sum of delayed days for all the delayed or- ders (ndays), and the sum of the change-over downtime in From the results we can conclude that using the Pareto-front minutes (tchange). This order was set according to the most approach gives us an expected greater versatility in choosing common objective hierarchy. a good solution, while at the same time we are not sacrificing one, likely the most important, objective. The only impor- 5.2 Evaluating the approach tant drawback is that multi-objective approaches need many To make sure that our proposed M-IBEA was working well, more evaluations than single-objective approaches. So, if we we ran a brute-force (BF) approach where all the possible do not have time to carry out enough evaluations, then the solutions were evaluated for n < 10 orders and the optimal single-objective approach is the only way. Pareto front was constructed for each of them. Table 1 shows a comparison of the number of problem evaluations, the ex- ecution time, and the matching of the Pareto front obtained 6. USABILITY OF MULTI-OBJECTIVE for n = 7, 8, 9. We did not include smaller n values, since SOLUTIONS in all cases a sub-one-second time was needed with perfect The multi-objective approach provides a set of feasible so- Pareto matching. From the obtained results it is clear that lutions, offering the possibility to select the final schedule with more than nine orders, the complexity increases well based on the specific decision maker needs. Since none of beyond an acceptable time (approximately two months) to the given solutions dominates the other solutions, all of them calculate all the solutions. Also, in all cases we were able are acceptable. Based on the current conditions, and accord- to acquire the same Pareto front using the BF and M-IBEA ing to the proposed set of solutions, a decision maker can approaches. When considering times, one must take into give more weight to some of the decision criteria. For this consideration that the BF was ran multithreaded with 12 an intuitive representation of the resulting solutions inside threads fully utilized, while the M-IBEA approach was sin- the GUI of the Planer application was provided, which is gle threaded. The perfect Pareto-front matching is unsur- presented in Figure 1. prising, since the IBEA already proved to be one of the best algorithms for solving multi-objective problems with more After the M-IBEA algorithm found the set of non-dominated than three objectives [3], which was also the main reason solutions, they are presented in the Planer application. In that we selected the IBEA as our base algorithm. the upper-right section there is a list of all the non-dominated solutions. In general, there might be up to several hundred 5.3 Results possible solutions. In [5], we optimized only according to the number of orders. To show that the multi-objective approach presented in [4] However, some of the criteria can be set tighter according is a better alternative, we compared the results with regard to the resulting range of each criterion, and according to the to the best result from the single-objective to the multi- current business conditions. In the specific example shown objective approach. The results showed that the single- in Figure 1, the initial set consisted of 518 solutions. The de- objective solution primarily concentrated on the number cision maker put the first objective into the range from 16 to of orders, while it neglected other objectives. But this is 17 out of 50, which in consequence moved the sliders of the not a surprise, as multi-objective solutions were able to find second objective from 697 to 738, the third objective from equally good solutions with regard to the number of orders 405 to 415, and the fourth objective from 60 to 111. So ir- and significantly better for other objectives, compared to regardless of which slider is moved, the ranges move accord- 251 ten not realistic. 8. ACKNOWLEDGMENTS The authors acknowledge the financial support from the Slovenian Research Agency (research core funding No. P2- 0098). This work has also received funding from the Eu- ropean Union’s Horizon 2020 research and innovation pro- gramme under grant agreements No 692286 (project SYN- ERGY) and No 722734 (under the Marie Sk lodowska-Curie action project UTOPIAE). 9. REFERENCES [1] M. Basseur and E. Burke. Indicator-based multi-objective local search. In Evolutionary Computation, 2007. CEC 2007. IEEE Congress on, pages 3100 –3107, sept. 2007. [2] K. Deb, S. Agrawal, A. Pratap, and T. Meyarivan. A fast elitist non-dominated sorting genetic algorithm for multi-objective optimization: Nsga-ii. In Figure 1: The graphical user interface. M. Schoenauer, K. Deb, G. Rudolph, X. Yao, E. Lutton, J. Merelo, and H.-P. Schwefel, editors, Parallel Problem Solving from Nature PPSN VI, ingly to the possible solutions of other objectives. Simulta- volume 1917 of Lecture Notes in Computer Science, neously, the list of possible solutions is updated to reflect the pages 849–858. Springer Berlin / Heidelberg, 2000. current setting of the objectives’ ranges. In the above ex- [3] H. Ishibuchi, N. Tsukamoto, and Y. Nojima. ample, the list narrowed down to 14 solutions. From them, Evolutionary many-objective optimization: A short the decision maker can select one solution which best fulfils review. In Evolutionary Computation, 2008. CEC 2008. the current demands. The visual representation consists of (IEEE World Congress on Computational Intelligence). all the relevant data, i.e., the production lines’ load, the or- IEEE Congress on, pages 2424–2431, june 2008. der types’ distribution, and change-over downtime lengths, [4] P. Korošec, U. Bole, and G. Papa. A multi-objective which are necessary to make the final decision. If the visual approach to the application of real-world production representation of the solution is accepted, it becomes the scheduling. Expert Systems with Applications, production schedule. By determining (using sliders), which 40(15):5839–5853, 2013. objective is the most important in the current situation and [5] G. Papa, V. Vukašinović, and P. Korošec. Guided to what extent, we automatically determine which part of restarting local search for production planning. Pareto front is important and at the same time disregard all Engineering Applications of Artificial Intelligence, the solutions from the Pareto front, which do not fulfil the 25(2):242–253, 2012. selected conditions. This way we are able to freely move the [6] M. Rentmeesters, W. Tsai, and K.-J. Lin. A theory of useful part of the Pareto front by moving sliders. lexicographic multi-criteria optimization. In Engineering of Complex Computer Systems, 1996. 7. CONCLUSION AND FUTURE WORK Proceedings., Second IEEE International Conference on, pages 76 –79, oct 1996. We presented what steps would be needed to make a memetic, multi-objective approach that used a simulation tool to asses [7] R. Zhang and C. Wu. A hybrid local search algorithm some real-world test cases of a production scheduling prob- for scheduling real-world job shops with batch-wise lem a more dynamic system by upgrading a simulation tool pending due dates. Engineering Applications of to a digital twin. From the perspective of the algorithm not Artificial Intelligence, 25(2):209 – 221, 2012. Special many changes would be required, since with a restart proce- Section: Local Search Algorithms for Real-World dure being already implemented any changes in the problem Scheduling and Planning. description could be “inserted” into the problem solving part. [8] E. Zitzler and S. Künzli. Indicator-based selection in multiobjective search. In in Proc. 8th International On the other hand, more substantial changes are required Conference on Parallel Problem Solving from Nature within the simulation tool. Primarily, how required inputs (PPSN VIII), pages 832–842. Springer, 2004. are being automatised (gathering data directly from the [9] E. Zitzler, M. Laumanns, and L. Thiele. SPEA2: company’s system). Additionally, an inclusion of some ma- Improving the strength pareto evolutionary algorithm chine learning algorithm, that would be able to detect and for multiobjective optimization. In K. C. Giannakoglou, predict failures on production lines, is foreseen for better D. T. Tsahalis, J. Périaux, K. D. Papailiou, and longterm estimation of production. T. Fogarty, editors, Evolutionary Methods for Design Optimization and Control with Applications to For future work, we are planning to implement the pro- Industrial Problems, pages 95–100, Athens, Greece, posed changes, which will enable for more real-life scenarios 2001. International Center for Numerical Methods in (including uncertainties-based worst-case scenarios), while Engineering. currently only “ideal” solutions are provided, which are of- 252 Zbornik 21. mednarodne multikonference INFORMACIJSKA DRUŽBA – IS 2018 Zvezek E Proceedings of the 21st International Multiconference INFORMATION SOCIETY – IS 2018 Volume E Delavnica AS-IT-IC AS-IT-IC Workshop Uredila / Edited by Matjaž Gams, Jernej Zupančič http://is.ijs.si 12. oktober 2018 / 12 October 2018 Ljubljana, Slovenia 253 254 PREDGOVOR Delavnica AS-IT-IC omogoča predstavitev primerov uporabe ter izmenjavo izkušenj med znanstveniki in drugimi deležniki na področju pametnega turizma, ki ga omogočajo inteligentna orodja in storitve, podprte z informacijsko-komunikacijskimi tehnologijami (IKT), predvsem umetno inteligenco (UI). Delavnica omogoča krepitev vezi in sodelovanja med izvajalci praktičnih turističnih storitev in znanstveno-raziskovalno sfero in spodbuja uporabo naprednih rešitev v turizmu. Delavnica je ena izmed aktivnosti projekta Avstrijsko-Slovenski Turistično- Informacijski Center (AS-IT-IC), ki je bil sprejet na Programu sodelovanja Interreg V-A Slovenija- Avstrija 2014-2020. Glavni cilj projekta je operativni center, kjer ponudniki turističnih informacij in virtualni asistenti sodelujejo pri odgovarjanju na turistično orientirana vprašanja turistov in jim pomagajo pri načrtovanju izletov na Slovensko-Avstrijskem čezmejnem območju. Sprejeti prispevki opisujejo stanje projekta AS-IT-IC eno leto pred zaključkom projekta. Prispevek Avstrijsko-Slovenski Inteligentni Turistično-Informacijski Center: Poročilo o napredku projekta povzame napredek glede na projektne in programske kazalnike, med tem ko se ostali prispevki osredotočajo na posamezne komponente končnih projektnih rezultatov. V Pregledu IKT orodij v turizmu so predstavljene različne IKT rešitve za pomoč turistom in ponudnikom turističnih infomacij. V Podatkovnih zbirkah AS-IT-IC so predstavljeni podatki ter podatkovni servisi, ki so na voljo na platformi AS-IT-IC. V prispevku API za podatke, je predstavljen sistem za pridobivanje ter dostop do podatkov. V e-Turist2.0 je predstavljena nadgrajena verzija sistema za načrtovanje in priporočanje izletov. V prispevku Testiranje varnosti pogovornih asistentov z uporabo planiranja avtorji opisujejo napredni sistem za ugotavljanje varnostnih pomanjkljivosti pogovornih asistentov. INTRODUCTION The AS-IT-IC Workshop is a forum for presenting the use cases and exchanging experience among academic and service industry partners on deploying intelligent information communication technology, in particular artificial intelligence, supported tools and services for enabling smarter tourism, as well as stimulating further adoption of such solutions through promotional activities and establishing direct collaboration between academia and industry. The workshop is an activity of the Cooperation Programme Interreg V-A Slovenia-Austria 2014-2020 project Austrian-Slovenian Intelligent Tourist-Information Center (AS-IT-IC project). The main project output will be the operational center with human and virtual assistants enabling automatic answering to the tourism-oriented questions in natural language and performing services to enable trip planning in the Slovenian-Austrian cross- border region. Accepted papers describe the AS-IT-IC project state one year before the project conclusion. Austrian-Slovenian Intelligent Tourist-Information Center: Project Progress Report summarizes the project progress with respect to project and programme indicators, while other contributions focus on specific modules of the final project results. In Tourism Related ICT Tools: a Review different ICT solutions with the aim to help tourist and tourist information providers are presented. In AS-IT-IC Databases the data and data services made available through the AS-IT-IC Platform are described. In the Content API paper the system for retrieving and serving the data is presented. In e-Tourist2.0 authors write about the upgraded trip planning and recommendation solution. Finally, the smart security testing for security leaks for common attack scenarios is presented in Planning-based Security Testing for Chatbots paper. 255 PROGRAMSKI ODBOR / PROGRAMME COMMITTEE Matjaž Gams, IJS (chair) Franz Wotawa, IST (co-chair) Josip Božič, IST Jernej Zupančič, IJS Tomaž Šef, IJS Oliver August Tazl, IST Dieter Hardt-Stremayr, GRAZ Katarina Čoklc, ZOS Marija Lah, SPOTUR 256 Austrian-Slovenian Intelligent Tourist Information Center: Project Progress Report 2018 Jernej Zupančič Matjaž Gams Jožef Stefan Institute and Jožef Stefan Institute and Jožef Stefan International Postgraduate School Jožef Stefan International Postgraduate School Jamova cesta 39 Jamova cesta 39 Ljubljana, Slovenia Ljubljana, Slovenia jernej.zupancic@ijs.com matjaz.gams@ijs.si ABSTRACT missed, stay longer and better satisfy their needs. Local Austrian-Slovenian Intelligent Tourist Information Center communities will easily offer local services and information (AS-IT-IC) is a project that was accepted in the Cooper- to visitors, e.g. a tour might include visiting a specialized ation Programme Interreg V-A Slovenia-Austria 2014-2020 craftsman and boost the selling of local products. Tourist call. The project goal is to create a joint Austrian-Slovenian officers will get better access to tourists. AS-IT-IC project center – an information and communication technology (ICT) (Table 1) provides the integration of virtual and human ser- supported network of service providers and tourist offices, vices from Austria and Slovenia with the uniform functional- municipalities, tourists and citizens to enhance continuous ity – to provide most relevant information, attract tourists, cooperation between them. The main project output will be and prolong their stay. the AS-IT-IC operational center with humans involved, hav- ing support of the ICT tools for communication, automatic question answering in natural language, information provi- Table 1: Project information card sion, trip recommendation and trip planning. This paper Title Austrian-Slovenian Intelligent Tourist overviews the current state of the project progress. Information Center Partners 1. Institut “Jožef Stefan” (lead partner) Keywords 2. Technische Universität Graz, Institut virtual assistants; chatbots; chat platforms; tourism; natural f˝ ur Softwaretechnologie language understanding; AS-IT-IC project 3. Javni zavod za turizem, šport, mladinske in socialne programe 1. INTRODUCTION SPOTUR Slovenj Gradec According to [1] a tourist cannot get the desired informa- 4. Združenje občin Slovenije tion in an integrated way from both humans and Web ser- 5. Graz Tourismus und Stadtmarketing vices, and much less the joint Austrian-Slovenian services. GmbH Typically, Slovenian or Austrian tourist office will provide only predefined national tours and not user-centric cross- Duration From 1. 7. 2016 to 30. 6. 2019 border tours. As a consequence, tourists may miss loca- tions they might be interested in visiting and tourist loca- The rest of the paper goes as follows. In Section 2 we de- tions get less visits. The goal of the project is to create scribe the state of deliverables and project workpackages, a joint Austrian-Slovenian center – an ICT supported net- in Section 3 we describe the project idea, while Section 4 work of service providers and tourist offices, municipalities, overviews the state of the prototypes and provides informa- tourists and citizens to enhance continuous cooperation be- tion on what has been accomplished by now. Sections 5 and tween them. Thus, cross-border tourist exchange, collabo- 6 describe the project dissemination activities and project ration and expertise transfer between providers will largely impact, respectively, so far. Section 7 concludes the paper. increase with respect to the current state. The main project output will be an operational center having support of the 2. PROJECT PROGRESS following tools: Virtual assistant (providing automatic an- The project has entered the last year of implementation swering in natural language to the questions and perform- (Figure 1). While the majority of the technical details (Ta- ing services according to demands from tourists), Commu- ble 2) have been resolved, the project results are still un- nication service (ICT solution that will enable conversation der active implementation and testing. Additionally, the between the tourists, virtual assistants, tourist information dissemination strategy and sustainability plan will be ad- workers and local communities), Information sources (inclu- dressed in more details in the coming months. sion of existing information sources), Recommender system for tour planning, Network of tourist services and services from local communities. 3. PROJECT IDEA The AS-IT-IC project tries to combine several solutions that The system will help tourists better plan their cross-border already provide partial solutions for smarter tourism: at- visits, discover less popular sites that would otherwise be traction discovery, trip planning [2], and communication 257 Figure 1: Project Gantt chart the state-of-the art platform that enables smarter tourism Table 2: Project deliverables status several open source technologies, data sources and internal Management Project reports 3/6 tools and services were examined, upgraded and are in the Dissemination and promo- not process of integration into one tourism platform – the AS- tion report started IT-IC platform. Using the open source software enables us Promotion material to start with a solid working solution and provide neces- Communication Publications 2/4 sary modifications as required by the project. A simplified Scientific publications 7/1 reference architecture is presented in Figure 2. Workshops on AS-IT-IC 4/3 Participation in tourist re- 0/3 3.1 Communication platform lated events The communication platform enables the users to commu- Project website nicate with each other (tourist – tourist, tourist – tourist System requirements and information provider, tourist – virtual assistant) over the Tourist specification chat based interface. Increased popularity of chat applica- information Tourist information plat- in progress tions (Facebook messenger1, WhatsApp2) prove that this is platform form a valid communication option used for exchanging and ob- Content items taining information. The main benefit being the option to Content creation guidelines upgrade the communication by integrating various virtual Communication applica- in progress assistant services. tions Virtual assistant require- Virtual 3.2 Virtual assistants ments assistant Virtual assistant service in progress Virtual assistants (also chatbots or conversational robots [3]) prototype are computer programs that can process input in natural Virtual assistant service not language and provide a reply. The input can either be voice started or text and the answer is usually a combination of a response in natural language and an action that was carried out by Tour planning requirements Tour taking into account the user input. An example would be as Tour planning service pro- in progress planning follows. User asks ”What are some cultural heritage sights I totype could visit near me?”. The virtual assistant would then first Tour planning service not identify the intent (the user would like to execute a search) started and the arguments (location: near the user, type of sight: AS-IT-IC AS-IT-IC Deliverable cultural heritage). Then it would acquire user location and Center AS-IT-IC Center in progress user preferences from the system and issue a request to the system database in order to obtain relevant attractions. The 1https://www.messenger.com/ with human and virtual assistants [3, 4]. In order to provide 2https://www.whatsapp.com/ 258 Figure 2: Simplified architecture corresponding to the project idea results would then be properly formatted and presented to As a base an open source team communication software the user: ”I have found the following attractions that match Rocket.Chat5 was chosen. In order to meet the project your search: Ljubljana castle, Cankar memorial house and requirements, several additions were developed: a custom Manor pavilions. Would you like to learn more about a home dashboard; a message modification for improved user specific attraction?”. experience; a custom information tab with information about the trip; notification modifications for better operation of 3.3 Trip planner mobile communication application; custom application pro- gramming interface for automatic message processing and Trip planners [2] help the user in planning the trip by keep- posting. Screenshot of the conversation user interface is pre- ing track of the places the user intents to visit, recommend- sented in Figure 3. ing attractions and points of interest relevant for the user, and automatically arranging the itinerary in order to op- timize the travel between the items on the itinerary. Trip 4.2 Virtual assistants planners usually also enable the user a visual overview of The virtual assistant used in the AS-IT-IC platform com- the whole trip and sometimes even enable the navigation. prises several modules. Two approaches were used when de- signing the assistant modules: rule-based approach (which is 3.4 Databases an upgrade of virtual assistants deployed at Jožef Stefan In- At the heart of the platform are the databases that provide stitute and the majority of Slovenian municipalities [3]); and all the information required by the AS-IT-IC services. The natural language based approach [4]. The rule based models databases provide structured data that can be used by sev- are more stable and easier to debug and understand, how- eral services for further processing. The databases consist ever, they have the issue of rule design, since every rule has of: the information about the attractions and other points- to be designed by hand, which is why they take a long time to of-interest (castles, caves, restaurants, etc.); the information implement. The natural language based modules, however, about the geographical entities (places, regions, rivers, mu- enables one to produce a virtual assistant that transforms nicipalities, etc.); and information scraped from the useful the natural language input into a structured format that can webpages (municipality information, opening hours, etc.). be further used by computer programs. The main disadvan- tage of such systems is the need of a language model (which Beside the ”Content” databases the system also requires is an active area of research, especially for smaller languages databases for user management and storing of the system such as Slovenian) and the need for a large set of training states and user generated content (conversation, saved trip data. itineraries, etc.). Within the AS-IT-IC Platform the rule-based approach is 4. PROTOTYPES used for the virtual assistant action that results from the The AS-IT-IC platform will consists of services deployed ei- user interaction with the uniquely identifiable objects present ther using Docker3 virtualization technologies or Flynn4 – a in the user interface (for instance buttons) and for com- self-hosted platform-as-a-service. mon text input provided by the user. The natural language based approach is used for intelligent search capabilities and in cases where the rule-based approach fails to work. Two 4.1 Communication platform backends are currently used for parsing the user input and 3https://www.docker.com 4https://flynn.io/ 5https://rocket.chat/ 259 Figure 3: Conversation user interface translating it into structured text – Dialogflow6 and Rasa7. tem; several functions that were previously coded by hand Additionally, in some cases we take the advantage of the were moved to the database, which significantly reduced the full-text search capability of the PostgreSQL8 database – we project size; and all the modules used by the application have made the required adjustments for the full-text search were upgraded to the latest versions, therefore increasing to work in all three project languages: Slovenian, English the application security. and German. 4.4 Databases Adapters were developed that enable the interaction be- Webpages and datasources that contain relevant tourism- tween the communication system and the virtual assistant oriented information and enable the use of the content for services: reading the user input, processing the text, per- non-commercial purpose were reviewed and gathered. While forming required actions and returning a response to the there are many datasources with relevant information (Slove- user. nia.info9, DEDI10, OPSI11, Geoportal ARSO12, e-Geodetski podatki13, eVode14, register kulturne dediščine15) there was 4.3 Trip planner additional work needed to unify the data formats, remove A basis for the trip planner used in the AS-IT-IC plat- the data that was not of sufficient quality and integrate all form was the e-Tourist application [2]. In order to meet the 9https://www.slovenia.info/en project requirements heavy modifications were made to the 10http://www.dedi.si/ e-Tourist codebase: the databases were extended to support 11https://podatki.gov.si/ additional attraction and geographical data; the routing was 12https://gis.arso.gov.si adjusted so that the trip planner can be used by third party 13https://egp.gu.gov.si/egp/ applications; user management was upgraded to enable AS- 14http://www.evode.gov.si/sl/vodni-kataster/ IT-IC platform users to automatically log-in into the sys- zbirka-vode/zbirka-podatkov-o-povrsinskih- vodah/hidrografija/ 6https://dialogflow.com/ 15http://www.mk.gov.si/si/storitve/razvidi_ 7https://rasa.com/ evidence_in_registri/register_nepremicne_kulturne_ 8https://www.postgresql.org/ dediscine/ 260 the data into one database. 2. Tourist workers networking: to change behaviour and increase the cooperation of tourist workers and providers 5. DISSEMINATION of ICT tools for tourism. Nine meetings and workshops were organized to promote AS-IT-IC and enable such Project partners have been active in disseminating the project networking. results, producing scientific papers at conferences and inter- national journals, producing publications for general public, 3. eHeritage: to raise the awareness of the need to make maintaining the project website, hanging project posters at the description of heritage sights and attractions avail- partner sites, and organizing workshops. able on the Internet in a way that enables easy search and the inclusion of such attractions into the tourists’ Scientific papers and publications for general public intro- itineraries. So far cca. 1000 heritage attractions were duced the project to the wider audience by presenting the inserted into the AS-IT-IC content database. project idea, describing the need for such a project, and presenting tools, services and prototypes, developed within 7. CONCLUSION the project. Project partners have so far contributed to 7 In the paper we presented the AS-IT-IC project, its goals scientific papers and 2 publications for general audience. and the issues it addresses. Project partners from the Slovenian- Austrian cross-border area came together in order to enable Project website16 was deployed in the first half year of the smarter tourism by integrating several tools and services into project. It presents all relevant information about the project one platform. The project has entered its last year of imple- and project partners, together with the project news. Addi- mentation and so far the development has gone according tionally it enables the visitors of the website to contact the to the plan. The prototypes for communication platform, project partners. virtual assistant, and trip planners are under development and will soon be ready for integration into the AS-IT-IC Project posters were designed according to the Coopera- platform and ready for testing by end users. tion Programme Slovenia-Austria rules and posted at part- ner sites (Jožef Stefan Institute, GUT Institute for Software The majority of the future work will be on disseminating technologies, and Association of municipalities of Slovenia). the project results and attracting a larger user base. To this end the tourist partners are in contact with the major tourist Workshops are one of the main dissemination channels, where organizations (such as Slovenian tourist board), which will partners invite general audience to attend or the partners enable us to reach a wider target audience and receive useful present the project and project results to the general au- user feedback. dience. Several workshops were organized, where AS-IT- IC project was presented: ”AS-IT-IC Workshop” within the Additionally, the partners will look into sustainable options Information society 2017 Multiconference, ”Presentation of for transferring the AS-IT-IC Platform management to in- tourism applications” workshop on behalf of the invitation in terested organizations. This will enable additional growth Nazarje, ”Artificial Intelligent into every municipality” work- of the AS-IT-IC Center, while the partners will maintain the shop organized at Jožef Stefan Institute, and a ”Site visit” functionality of the AS-IT-IC platform as developed for the workshop organized at Jožef Stefan Institute, where Jožef purpose of the project. Stefan Institute employees were invited. 8. ACKNOWLEDGMENTS 6. PROJECT IMPACT The work was co-funded by Cooperation Programme Inter- The AS-IT-IC project enables the project partners to greatly reg V-A Slovenia-Austria 2014-2020, project AS-IT-IC. increase the cooperation in the cross border area. The part- ners have organized five cross-border meetings, which re- 9. REFERENCES sulted in the exchange of information, data, examples of [1] AS-IT-IC Project partners. About the project. good practice in the field of tourism and also in additional project application. The partners also cooperated in the or- https://as-it-ic.ijs.si/about/, 2016. [Online; accessed 16-September-2018]. ganization of workshops, which enabled the project partners to reach several third party stakeholders: 7/50 (7 reached [2] B. Cvetković, H. Gjoreski, V. Janko, B. Kaluža, out of 50 promised) representatives from local public author- A. Gradišek, M. Luštrek, I. Jurinčič, A. Gosar, ities – entities were reached so far (ministries and munici- S. Kerma, and G. Balažič. e-turist: An intelligent palities); 10/30 representatives from interest groups includ- personalised trip guide. Informatica, 40(4):447, 2016. ing non-governmental organizations (development centers, [3] D. Kužnar, A. Tavčar, J. Zupančič, and M. Duguleana. tourist organizations); 7/5 small and medium sized compa- Virtual assistant platform. Informatica, 40(3):285, 2016. nies; and 89/3000 interested individuals. [4] J. Zupančič, G. Grasselli, A. Tavčar, and M. Gams. Virtual assistants for the austrian-slovenian intelligent The main communication goals of the project are: tourist-information center. In Proceedings of the 12th International Multiconference Information Society - IS 2017, volume E, pages 27–30, Ljubljana, Slovenia, 2017. 1. Integrated tourist communication service: to raise aware- Jožef Stefan Institute. ness of the AS-IT-IC platform. AS-IT-IC was men- tioned five times in press and web page news. 16https://as-it-ic.ijs.si 261 Tourism Related ICT Tools: a Review Gregor Grasselli Jernej Zupančič Jožef Stefan Institute and Jožef Stefan Institute and Jožef Stefan International Postgraduate School Jožef Stefan International Postgraduate School Jamova cesta 39 Jamova cesta 39 Ljubljana, Slovenia Ljubljana, Slovenia gregor.grasselli@ijs.si jernej.zupancic@ijs.com ABSTRACT and applications, when searching for the trip-related infor- In the paper we review the existing information and commu- mation; tourists to call or go to local tourist information nication technology (ICT) based tools and services that em- offices for additional info; tourist information providers to power tourists and tourist information providers in obtain- keep their information available in several systems all over ing and providing information needed for trip planning. We the internet and structured in different – often incompatible define four tourism-related service categories: search with ways; tourist information providers to pay for the presence booking, trip planners, chatbots, and forums. We summa- on tourism-tools and services. There is clearly room for im- rize the good practices identified in the reviewed tools and provement [3] and the first step would be the establishment expose the issues stemming from the fragmentation of the of an online platform that would in one place integrate sev- tools and data. In order to overcome the identified prob- eral services and tools and enable the users to gather and lems we propose the AS-IT-IC Platform – a project result of provide all necessary information related to trip planning. the Austrian-Slovenian Intelligent Tourist Information Cen- One such platform is being developed within the Austrian- ter (AS-IT-IC) project, accepted in the Cooperation Pro- Slovenian Intelligent Tourist Information Center (AS-IT-IC) gramme Interreg V-A Slovenia-Austria 2014-2020 call. The project. AS-IT-IC project aims to integrate several functionalities usually offered by distinct services, unifying the databases, The rest of the papers continues with the description of four and providing free access to tools and services for tourists categories of tools and services in Sections 2, 3, 4 and 5. and tourist information providers. Good practices are summarized in Section 6 and the need for an integrated solution is addressed in Section 7. Section Keywords 8 concludes the paper. virtual assistants; chatbots; tourism; trip planning; search; review; AS-IT-IC project 2. SEARCH WITH BOOKING The group offering the least functionality is the one of search engines for hotels, plane flights, restaurants and such. The 1. INTRODUCTION user is mostly required to input a predetermined set of pa- Due to the increase in the Internet usage it has become rameters, and is then referred to a list of matching options paramount that organizations and their tools are accessible they can choose from. This list can sometimes be further through the Internet. In the last few years a steep increase filtered and/or sorted based on additional criteria. After in the mobile usage has additionally fueled the development deciding on what suits them best, the user is relegated to of mobile applications and mobile friendly web applications, booking, which is how the site makes money. several of those in the domain of tourism. Of course most if not all hotels, airlines, tourist agencies and In this paper we present the tools provided by organizations other tourist providers have their own websites, which allow that operate on a global scale. We categorize the tools into a prospective user to directly book their services. The exam- four categories: ples below are of search engines that accumulate data from many such sites, or get it directly from the end providers and try to make it more convenient for a prospective tourist 1. Search with booking to find what they are looking for. 2. Trip planners 3. Chatbots – conversational robots • Sabre – https://en.eu.sabretravelnetwork.com/home/ page/book_flights_hotels 4. Forums • Expedia – https://www.expedia.com • Each category is unique: in the ways it helps the user in the TripAdvisor – https://www.tripadvisor.com process of trip planning; in the type of data it uses; and in • Priceline – https://www.priceline.com the target users. This, however, leads to the fragmentation of data and the need for: tourists to check several web sites • Yelp – https://www.yelp.com 262 • OpenTable – https://www.opentable.com/start/home 4.1 Customer service chatbots One of the trends in using chatbots is to automate customer • Booking.com – https://www.booking.com service on a company’s website. Things like providing an- swers to frequently asked questions, or finding relevant in- • Skyscanner – https://www.skyscanner.com formation on the website without having to manually search for it can easily be automated by chatbots. The aim of these 3. TRIP PLANNERS bots is to reduce the load on human customer service staff A slightly more interesting group is the group of trip plan- and to provide a better customer experience by making it ners. They also require the user to input a set of predeter- easier and faster to get answers from data on the website. In mined parameters, but unlike in the previous group where this context a chatbot is an addition to an existing website, the parameters are mostly the same on all websites, here often appearing as a chat window in one of the corners. they vary quite a lot from service to service. What is dif- ferent is also that these services mostly output a single trip Examples of such chatbots are: plan which can later be modified. After a user decides on their final trip plan, they are offered accommodation and • Ana – https://connectmiles.copaair.com/en/web/ transportation booking necessary for the trip’s realization. guest/ask-ana The booking stage on some of the websites utilizes services from the first category. Some interesting examples of such • Julie – https://www.amtrak.com/about-julie-amtrak- services are provided below. virtual-travel-assistant 4.2 Instant messaging chatbots • Roadtrippers – https://roadtrippers.com With rising popularity of instant messaging platforms like Facebook’s messenger6 and Telegram7 that expose APIs for • Mapquest – https://www.mapquest.com/routeplanner bots to converse with users, many bots now exploit this channel for access to users. By responding to messages in • Triphobo Tripplanner – https://www.triphobo.com/ group chats (possible on Facebook’s messenger for instance), tripplanner chatbots are a new way for making a product more discov- erable and for making the user’s trip from wish to purchase • Inspirock – https://www.inspirock.com shorter. • Sygic Travel – https://travel.sygic.com In many cases chatbots present on an instant messaging plat- • routeperfect – https://www.routeperfect.com/trip- form are just a different interface to an existing service. Ex- planner amples of such bots are: • wanderapp – https://www.wanderapp.me • Expedia – https://viewfinder.expedia.com/features/ • Go Real Europe – introducing-expedia-bot-facebook-messenger https://www.gorealeurope.com • Skyscanner – https://www.skyscanner.net/news/tools/ 4. CHATBOTS skyscanner-facebook-messenger-bot With the emergence of natural language processing tools • Cheapflights – https://www.cheapflights.co.uk/news/ like dialogflow1, wit2 and rasa3 that make it quite easy for cheapflights-chat-awards developers to implement chat based interfaces in a growing number of languages, there has been an explosion in the • Hello Hipmunk – https://www.hipmunk.com/hello number of applications and websites that offer a chat based interface. There are even companies like Botflux4 that offer There are a number of chatbots that act as aggregators over their customers custom made chatbots. different services. They provide a conversational interface for searching offers from many sources and providing the Facing the consumer, chatbots offer a more dynamic (re- user with the result that most closely matches their require- quest values for different parameters based on the user’s in- ments. An example of such a bot is Assist8, a chatbot that put so far) and interactive approach to defining the user’s aggregates several services for making hotel reservations, requirements than classical forms and menus. They also of- ride hailing, making table reservations and online shopping. fer a completely new experience when using a voice interface It is the only product of a start up with the same name and through a smartphone or a specialized device such as Ama- can be used through messenger, Telegram, SMS, Twitter, zon’s Echo or Google Home. A good resource for finding in- Google Assistant and Slack. teresting chatbots based on the messaging platform, where one can converse with them, and their area of expertise is According to statista9, business travel in 2016 amounted to botlist.co5. 1.3 trillion USD and represented about 10% of all travel 1https://console.dialogflow.com 6https://www.messenger.com/ 2https://wit.ai 7https://telegram.org/ 3https://rasa.com 8http://www.assi.st/ 4https://www.botflux.com/tourism 9https://www.statista.com/topics/2439/global- 5https://botlist.co/ business-travel-industry 263 spending in 2015. Thus it makes sense that a number of In most cases travel related forums appear as part of a big- travel chatbots are specifically targeting business travellers: ger travel related website. The popular TripAdvisor website also includes a typical travel forum13 of this kind. Questions and responses are checked for destinations and attractions • Carla – https://www.cwtcarla.com/CarlaWeb TripAdvisor knows and if any are found, they are displayed in a card, below the user’s post, showing their name, a pic- • Pana – https://pana.com ture and their ranking. Clicking on the card will show the site for that attraction. • MEZI – https://mezi.com Another example of a website that also includes a forum is Lonely Planet14. According to Wikipedia15 they are the A different kind of application is a so called virtual concierge. largest travel guide publisher in the world. They also pro- Its main goal is to assist in communication between hotel vide a website that would fall into the search with booking staff and their guests by providing an interface for checking category, coupled with their newsletter and of course selling into a hotel, ordering room service, and requesting informa- travel guides. This rather expansive website also includes a tion about the hotel. They have automatic translation inte- forum for exchanging ”travel advice, hints and tips” as they grated into the service, so customers can interact with the put it. The forum is not limited to country based discus- staff in their native language. These applications are in this sions but also has (among others) sections about equipment, category because they mainly function through a conversa- travel health and vaccinations, searching for travelling com- tional interface and use instant messaging technology. Also panions, house sitting and swapping as well as people selling some of their functions are fully automated so they are bots and buying stuff through the forum. and not just messaging apps with translations. An example of such a virtual concierge is The Besty10, a phone app to According to the quick analysis of posts, Fodor’s Travel help people communicate with their hotel’s staff as well as Talks Forum16 seems to be the most popular. They also find and book tours, restaurants and activities at the “low- sell guidebooks and have a very extensive website that also est” prices. They also offer tour guides and live chat with lo- offers hotel bookings. cal tour experts. The MEZI chatbot, mentioned earlier also offers a virtual concierge service of this kind as part of its As we are living in the age of social media, it would be remiss capabilities. Another example of this kind of chatbot is Hi not to mention the #travel tag on Twitter17, used to post Jiffy11. It is available on messenger, and allows searching for about travel experiences, as well as the existence of quite hotels and making reservations in addition to its customer a number of twitter accounts that are dedicated solely to care functionality. It employs a model where queries that travel news18. cannot be answered automatically are forwarded to hotel staff. The query and the provided answer are then included 6. GOOD PRACTICES in the bot’s training set so that it can answer automatically By reviewing the existing ICT solutions available on the in- when a similar query is input by a user. According to its ternet, the following commonalities and good practices were website 77% of its answers are provided automatically at the observed. time of this writing. 5. FORUMS 1. Since almost 53% of all internet traffic in 2017 was When people want opinions, recommendations or advice, produced through mobile devices19, having a mobile they turn to the forums, where they can ask questions re- application or a different way of making the application lated to their planned excursions and get answers from trav- work on a mobile phone (like through messenger, or a ellers who have been there before. On some of the forums, mobile-first web application) is a must. travel agents seem to be quite involved as well, answering 2. When possible it is a good idea to integrate with appli- questions by prospective tourists while advertising their ser- cations users are already using in their everyday life, vices. All of the forums we came across cover travelling to like calendars. This allows to get user data without the whole world, but usually have a different section for each needing the user to type everything, as well as enables continent which is then further divided by country. the user to use the results of an application more con- veniently. Most forums also have a section dedicated to posting longer accounts of travellers who believe they have experienced 3. Integrating multiple data sources into a single view something worth sharing. An interesting website that col- is very helpful for users as they get more complete lects longer posts by travellers as well as photos is Travel- blog12. In addition to their forum for discussing travel plans 13https://www.tripadvisor.com/ForumHome and asking for advice, they also have a blog section, where 14https://www.lonelyplanet.com/thorntree/welcome anyone can write about their experiences or post a photo 15https://en.wikipedia.org/wiki/Lonely_Planet they think is particularly eye catching, and the rest of the 16https://www.fodors.com/community users will vote on the best blog and photo of the week. 17https://twitter.com/ 18http://mashable.com/2012/08/04/travel-twitter/ 10https://thebesty.com/ #KtwUdPjm_Gqw 11http://hijiffy.com/ 19http://www.trendreports.com/article/technology- 12https://www.travelblog.org in-tourism 264 information about a destination and do not have to 3. Cooperation of several local tourist information cen- check multiple sources on their own. ters with vast knowledge on the touristic offers in their area. 4. Many popular applications scrape provider websites for information and special offers. Others rely on providers 4. Free access for tourists and tourist information providers. to manually enter all information. 5. Open access to data and data services. 5. Sites that cover a wider geographical area are more useful, since they provide a one-stop shop for the whole trip as opposed to having to visit several websites to The AS-IT-IC Platform, however, does have one big disad- get informed on each destination individually. vantage compared to the rest of the services – it is a publicly founded project with reserved funds for the development and 6. Availability through multiple channels. Having a web- initial activities to raise the project awareness and dissemi- site is fine, but also being available through other chan- nate results. After the end of the project, no resources have nels, especially instant messaging platforms really helps been granted yet to further promote project results. While with discoverability. the partners have committed to maintain the project results 7. Until the invention of general AI, machines will be lim- for another 5 years after the end of the project, the issue of ited in what they can do, so to minimize customer frus- getting sustainable funds to enable further promotion and tration, keeping humans in the loop on the provider dissemination is yet to be solved. side can be very helpful. 8. CONCLUSION 8. Making customization of automatically generated trip In this paper the ICT tools provided in order to empower plans and other suggestions as easy and complete as smarter tourism are presented. Providers from around the possible, or the users will only use the tool to get the world were taken into account. The tools were classified into suggestions then they will use more low level tools to categories in order to provide a sense of what is available for actualize the parts they liked. This lowers the conver- tourists and tourist service providers. Additionally, the tools sion rate of the tool and makes customers less happy. were critically assessed and good practices were identified. 9. Allowing users to filter and sort displayed information Further, the AS-IT-IC Platform was compared to existing based on their interests. A good example is how Road- tools and main similarities and differences were pointed out. trippers allows users to set what kinds of points of in- terest they want to see on the map. The review provides a basis for anyone interested in the deployment of tourism-oriented services. One has to take 7. RELATION TO AS-IT-IC into account, however, that not every problem in tourism has a technological solution. One of the main components of The AS-IT-IC project tries to combine several partial so- the AS-IT-IC project is the networking one, where the goal lutions already implemented by the ICT tools mentioned is to connect several stakeholders that provide technology in this paper: attraction discovery, trip planning [1], and solutions to the users in need of such solutions. communication with human and virtual assistants [2, 4]. In order to provide the state-of-the art platform that enables smarter tourism several open source technologies and data 9. ACKNOWLEDGMENTS sources were utilized and integrated into one tourism plat- The work was co-funded by Cooperation Programme Inter- form – AS-IT-IC platform. reg V-A Slovenia-Austria 2014-2020, project AS-IT-IC. AS-IT-IC project empowers tourists by: helping in obtain- 10. REFERENCES ing all the required information related to trip planning in [1] B. Cvetković, H. Gjoreski, V. Janko, B. Kaluža, one place; and enabling discovery of local and less known A. Gradišek, M. Luštrek, I. Jurinčič, A. Gosar, but still relevant attractions. Further, it empowers tourist S. Kerma, and G. Balažič. e-turist: An intelligent information providers by: providing an integrated way of ex- personalised trip guide. Informatica, 40(4):447, 2016. posing the tourism content in his or her area to the Internet; [2] D. Kužnar, A. Tavčar, J. Zupančič, and M. Duguleana. and enabling the access to the tourists in an asynchronous, Virtual assistant platform. Informatica, 40(3):285, 2016. modern chat-based style. [3] B. Peischl, O. A. Tazl, and F. Wotawa. Open questions of technology usage in the field of tourism. In The reviewed tools, together with traditional communica- Proceedings of the 12th International Multiconference tion methods, indeed already offer the same or at least very Information Society - IS 2017, volume E, pages 29–22, similar functionality as is planned for the AS-IT-IC plat- Ljubljana, Slovenia, 2017. Jožef Stefan Institute. form. However, even disregarding the obvious benefit of the [4] J. Zupančič, G. Grasselli, A. Tavčar, and M. Gams. functionality integration into one platform, there are still ad- Virtual assistants for the austrian-slovenian intelligent vantages of the AS-IT-IC project results, for the time being tourist-information center. In Proceedings of the 12th mainly for the Slovenian-Austrian cross-border area: International Multiconference Information Society - IS 2017, volume E, pages 27–30, Ljubljana, Slovenia, 2017. 1. Larger database of attractions. Jožef Stefan Institute. 2. Inclusion of path-based attractions – for instance wine roads, or walking trips and geographical information. 265 AS-IT-IC Databases Jernej Zupančič Oliver A. Tazl Blaž Mahnič “Jožef Stefan” Institute and Institute for Software “Jožef Stefan” Institute Jožef Stefan International Technology Jamova cesta 39 Postgraduate School Graz University of Technology Ljubljana, Slovenia Jamova cesta 39 Inffeldgasse 16b blaz.mahnic@ijs.si Ljubljana, Slovenia Graz, Austria jernej.zupancic@ijs.com oliver.tazl@ist.tugraz.at Gregor Grasselli “Jožef Stefan” Institute and Jožef Stefan International Postgraduate School Jamova cesta 39 Ljubljana, Slovenia gregor.grasselli@ijs.si ABSTRACT Roadtrippers3 and Triphobo Tripplanner4. Austrian-Slovenian Intelligent Tourist Information Center (AS-IT-IC) is a project that was accepted in the Cooper- Virtual assistant enables the user to obtain tourism-related ation Programme Interreg V-A Slovenia-Austria 2014-2020 information using a rich-text based interface, similar to the call and has two main goals: one is to build information ones provided by Facebook Messenger5 . Examples include and communication technnology (ICT) tools to support the Hello Hipmunk6 and MEZI7. tourist when he or she creates personalized itinerary for the visit of Slovenian-Austrian cross-border area; and the sec- Forums provide a place where usually users but sometimes ond is to create a sustainable community that will support also professionals provide descriptions of their trips, express the use of the tools. In this paper we describe the provision, their opinions about attractions and places to visit, and pro- cleaning, integration and deployment of data and data ser- vide helpful advice to fellow travellers. Examples include vices needed by the ICT tools in tourism. Data and data Fodor’s Travel Talks Forum8 and Lonely Planet9. services form one of the main pillars that enables the AS- IT-IC platform to provide tools and services, which could In order to provide such services developers need data. The serve tourism-related information to end users – tourists and data can be obtained in several ways: tourist information providers. 1. The information about attractions and other points of Keywords interest can be obtained in advance (as is the case with web spider; data; tourism; databases; web services; AS-IT- some trip planners and virtual assistants). IC project 2. By making an application programming interface (API) 1. INTRODUCTION call to an external service (search with booking). There is an increasing number of services and applications 3. By relying on users to provide the content when the available for tourists and tourist information providers across application is already live (forums). the Internet. The services could be roughly categorized into the following categories: Search with booking ; Trip planners; Virtual assistants; and Forums. The ICT tools that will be used directly by the users and are developed within the AS-IT-IC project integrate several ser- Search with booking enables the user to search for a type vices that belong to the before-mentioned categories: com- of accommodation, transport or adventure, specifying the munication platform that enables communication of tourists time interval of using the services and in some cases even with tourist information providers and virtual assistants; buying the service or reserving it. Examples are Expedia1 virtual assistant, which provides useful information 24/7; and OpenTable2. 3https://roadtrippers.com 4 Trip planners enable the user to view attractions in certain https://www.triphobo.com/tripplanner 5 area, obtain additional attractions by clicking on them and https://www.messenger.com/ 6 forming a customized trip. Examples of such services are https://www.hipmunk.com/hello 7https://mezi.com 1https://www.expedia.com 8https://www.fodors.com/community 2https://www.opentable.com/start/home 9https://www.lonelyplanet.com/thorntree/welcome 266 and tour planning for the automatic creation of a trip. In fourteen data categories on the web site: Population order to provide useful tourist information services, we had and Society, Justice, the legal system and public safety, to combine several data procurement options, also used by Public Sector, Education, Culture and Sport, Social other systems that usually cover only a part of the AS-IT-IC and employment, Health, Environment and Spatial Plan- functionality. ning, Transport and infrastructure, Agriculture, fish- eries, Forestry and nutrition, Finance and Taxes, Econ- Additional difficulty in obtaining the data was the fact that omy, Energy, Science and Technology and International AS-IT-IC covers the Slovenian-Austrian cross-border area, Affairs. Some of the datasets available on the portal for which little structured information is available. This is are of interest also for the tourism domain, for instance especially true for natural and cultural heritage attractions, a computer readable map of bodies of water, where which are the focus of the Interreg Slovenia-Austria Pro- bathing is possible. gramme. We reviewed several data sources that are avail- able for non-commercial use and tried to include the most 5. Slovenian Cultural heritage register14. This is an of- relevant and quality ones. ficial database of cultural heritage on the territory of the Republic of Slovenia, provided by the Ministry of According to the review, the following types of data was Culture of Republic of Slovenia. The registry contains identified as useful: attraction data and tourism-related points- 30.095 entries of several types. The big disadvantage of of-interest (natural and cultural heritage, sights, activities, this database, however, is that the use of the database accommodation, places to eat etc.); geographical data (struc- is prohibited for online applications, which is a big tured representation of geographical entities such as rivers, drawback in the information age – especially since the lakes, municipalities, cities etc.); and services related to get- data is of public interest. ting from place A to place B, so called routing services. 6. Graz Tourism database. The data of this website com- prises the tourism sights, attractions and offers of the In the rest of the paper we review available data sources in city of Graz and its neighbouring regions. The data is Section 2. Further, we describe the data related to attrac- available via a back-end using a REST-JSON API. The tions in Section 3, the geographical data in Section 4, and data is maintained by Graz Tourism and the tourism the chosen routing system in Section 5. Section 6 provides partners to provide detailed and high-quality data. a brief description of how the data will be used within the AS-IT-IC Platform and Section 7 concludes the paper. 2. DATABASES AND DATA SOURCES The following data sources were identified as the most rele- vant: 1. Slovenia.info website10 (Figure 1). The data source comprises 8798 tourist attractions and is still growing, due to the fact that the tourist information providers are constantly uploading and updating new attraction descriptions. 2. Dedi.si website11. The data provided by this website comprises only natural and cultural heritage, there- fore being very suitable for the purpose of the project. Figure 1: Tourist attractions from Slovenia.info However, due to the incompatible data formats, the inclusion of the Dedi.si data is postponed for the time 3. ATTRACTIONS DATA being. This sections describes all the attributes used to represent 3. Europeana website12. This data source stores the data an attraction datum. Data was structured in an appropri- about artworks, artifacts, books, films and music from ate way that will enable the AS-IT-IC services to provide European museums, galleries, libraries and archives different kind of functionality. from around the world. The number of all entries is around 58 millions. However, due to the automatic The data was imported into the PostgreSQL15 database, data collection the information is very often wrong. where each datum insert is represented as follows (the data We have decided to not include the Europeana data. structure is based on the already developed e-Tourist sys- tems [1]): 4. Open data portal of Slovenia13. The portal provides • title (sl, en, de, it): the title of attraction stored in the information, tools, and useful resources, which can four languages – Slovenian, English, German, Italian. be used in web and mobile applications. There are For example “Gostišče Kimovec” 10https://www.slovenia.info/en/map 14http://www.mk.gov.si/si/storitve/razvidi_ 11http://www.dedi.si/ evidence_in_registri/register_nepremicne_kulturne_ 12https://www.europeana.eu/portal/en dediscine/ 13https://podatki.gov.si/ 15https://www.postgresql.org/ 267 • description (sl, en, de, it): description of tourist at- regarding other natural bodies was retrieved from the ARSO traction in four languages Geoportal18. • category (sl, en, de, it): for example “Adrenaline sports” • subcategory (sl, en, de, it): for example “Paragliding” Water bodies data contains the information in the GeoJSON • location: GPS coordinates of tourist attraction, for format19: example “(45.94,13.71” 1. 48 bathing areas. • figure: image that represents the tourist attraction 2. The Slovenian coast. (web path) 3. 20 lakes and larger bodies of water. • trip advisor: attraction rating retrieved from TripAd- 4. 165 rivers. visor16 web site • address: “Zgornji Hotič 15, 1270 Litija“ Additional data on natural heritage also contained the data • recommended viewing duration time: “1:30:00“ in the GeoJSON format: • price range: how much does it cost to visit the attrac- 1. 17 protected areas not included into Natura 2000. tion “1-5” 2. 307 areas of ecologic importance. • expert evaluation: what is the expert opinion in the 3. 10.730 caves with descriptions added. quality of the attraction “1-5” 4. 357 Natura 2000 areas. • parking: parking options 5. 2.657 items from the registry of natural heritage. • campers: availability of camper parking 6. 517 additional protected areas. • web page: “www.gostisce-kimovec.com” • phone: “05 458 654” The geographic information data on statistical regions and • working hours: “mo-fr: 8:00-18:00” settlements has been acquired from GURS (Slovenian Geode- • working hours comment: “Always opened” tic administration) portal e-Surveying data20. All the data • accessibility: how can one visit the attraction “(car, was obtained in the GeoJSON format and it included: Bound- walk, bike, boat, bus)” aries of 12 statistical regions; Boundaries of 6037 settle- • keywords: few keywords that relate most to the at- ments. traction 5. ROUTING SERVICE According to the data analysis, the data categories are pre- There are several routing services available on the Internet, sented in Table 1. the most popular being the Google maps21. While being practical the subscription services are not cost effective and Table 1: Category counts for the attraction data not in line with the project goals. We have therefore looked Category Count into open source solutions available. The most popular open source solution identified was the Open Street Routing Ma- Accommodation 2040 chine [2] (OSRM). Adrenaline sports 63 Casinos 59 We have downloaded the map data22 and combined the Cities 649 Austrian and Slovenian maps into one file using the rec- Culture 2337 ommended osmconvert23 tool. Then we processed the data Cycling and biking 326 according to the official OSRM instructions24. The authors Food and wine 1857 of the OSRM tool also provide a Docker25 image that can Hiking 142 be used for the processing of the maps data and for serving Nature 703 the routing back-end. We have utilized the OSRM docker Spas and health resorts 287 image in order to process and deploy three distinct routing Sports 72 services: for walking, cycling and car riding. This enables Water activities 208 the service to recommend routes to the user based on his or Winter sports 55 her preferred way of traveling. Services are currently available through the API calls, for There are additional 103 subcategories that further classify instance, when the service requires a route from point A to each attraction datum or point-of-interest, however, they are point B, it issues an API call to: not listed here due to the space reasons. http://docker-e9.ijs.si:5007/route/v1/driving/LON-A, 4. GEOGRAPHICAL DATA LAT-A;LON-B,LAT-B?steps=false The database of Geographical data is composed of two parts; 18https://gis.arso.gov.si/geoportal/catalog/main/ geographical data and statistical regions. Geographical data home.page 19 contains geographical data for Slovenia such as lakes, rivers, https://tools.ietf.org/html/rfc7946 20 caves etc. The data related to the water bodies was obtained https://egp.gu.gov.si/egp/ 21 from the “eVode portal”17 (eng. eWaters), while the data https://cloud.google.com/maps-platform/ 22http://download.geofabrik.de/europe.html 16 23 https://www.tripadvisor.com/ https://wiki.openstreetmap.org/wiki/Osmconvert 17 24 http://www.evode.gov.si/sl/vodni-kataster/ https://github.com/Project-OSRM/osrm-backend/ zbirka-vode/zbirka-podatkov-o-povrsinskih- wiki/Docker-Recipes vodah/hidrografija/ 25https://www.docker.com/ 268 The service returns a JSON response with the most impor- the use in public interest. This is the case with the registry tant objects: route specifications in the form of an encoded of cultural heritage. polyline; route distance in meters; and route travel time in seconds. Additional information about the API can be ob- In the future we plan to integrate additional data sources tained on the official OSRM website26. into the AS-IT-IC databases – by performing data-fusion procedures we will merge the data into a single, richer database. Additionally, we will try to provide the data services to third 6. DATA SERVICES IN AS-IT-IC party developers that now have to go through the same pro- The described data and data services will enable the AS-IT- cedure as we did, in order to obtain similar data. Open IC services to provide the functionality as required by the data and services was one of the main project goals from the project. start, since we want to improve the tourist experience not only directly but also indirectly by providing services that Th attractions database enables: the virtual assistant to will enable third party developers to come up with their own search for points-of-interest that best match the users query, innovative solutions. recognize points-of-interest entities, and fetch information about the attraction; the tour planning service to take into 8. ACKNOWLEDGMENTS account the attractions locations and provide recommenda- The work was co-funded by Cooperation Programme Inter- tions to the user based on the attraction category, subcat- reg V-A Slovenia-Austria 2014-2020, project AS-IT-IC. egory, location and similarity to other attractions based on the attraction description; the communication platform to present the data about attractions through the familiar user 9. REFERENCES interface. [1] B. Cvetković, H. Gjoreski, V. Janko, B. Kaluža, A. Gradišek, M. Luštrek, I. Jurinčič, A. Gosar, The geographical data enables: the virtual assistant to rec- S. Kerma, and G. Balažič. e-turist: An intelligent ognized geographical entities, search using the geographical personalised trip guide. Informatica, 40(4):447, 2016. position qualifiers (e.g. “cultural heritage in the Ljubljana [2] D. Luxen and C. Vetter. Real-time routing with city”); the tour planning service to enable recommendation openstreetmap data. In Proceedings of the 19th ACM based on the exact location and geographical area bound- SIGSPATIAL International Conference on Advances in aries. Geographic Information Systems, GIS ’11, pages 513–516, New York, NY, USA, 2011. ACM. The routing service enables: the virtual assistant to take into account the tourist travel options (e.g. “show me natural heritage sites that I can reach in one hour by a bicycle”); the tour planning service to calculate optimal travel plan (since it takes into account the geographical position of the attractions on the itinerary and the transport option chosen by the tourist) and to provide a preview of the trip on a map. 7. CONCLUSION In the paper we described the data sources, procurement, structure and types of data made available for the AS-IT-IC platform. Additionally we provided a short description of services, which are possible due to the data availability. The main problem with the tourism-related data procure- ment is the unavailability of data in a structured, easily ac- cessible format. The portal “Odprti Podatki Slovenije”27 (eng. Slovenian Open Data) for instance is a good start, however, there are still problems with the discoverability, data formats and data availability. Several good data sources were identified only after weeks of searching for relevant data over the Internet. In order to obtain relevant information users with non-commercial intent can still use web-scrapping in order to obtain the desired data, add the authorship no- tice and link to the original data source, however, this leads to fragmentation of data structures, additional stress on In- ternet bandwidth and non-optimal solutions to keep the data updated. Additional problem is the unavailability of data for 26https://github.com/Project-OSRM/osrm-backend/ blob/master/docs/http.md 27https://podatki.gov.si/ 269 Content API - A Cloud-based Data Source for the AS-IT-IC Platform Oliver A. Tazl Franz Wotawa Institute of Software Technology Institute of Software Technology Graz University of Technology Graz University of Technology Inffeldgasse 16b/II, 8010 Graz, Austria Inffeldgasse 16b/II, 8010 Graz, Austria oliver.tazl@ist.tugraz.at wotawa@ist.tugraz.at ABSTRACT 2. Tourism companies (e.g. hotels, restaurants,...) This paper introduces the design and implementation of an element of a microservice application for supporting a mod- 3. Natural Heritage (e.g. rivers, mountains,...) ern application for tourist information. Therefore, we intro- duce the Content Application Pogramming Interface (API) 4. Events system, a microservice, which collects tourism relevant data from multiple sources and provides it to serveral services 5. Cultural heritage within the Austrian-Slovenian Intelligent Turist Information Center (AS-IT-IC) plattform in turn. Content API is built using modern technologies and frameworks, like Docker1, The goal of the content subsystem is to integrate various in- Spring 2 or Vaadin 3. formation sources about tourism offers into a single database. This database is meant to be updateable from these sources, Categories and Subject Descriptors like Google or other tourim websites. The integrated user H.2 [Database Management]: Systems; I.2 [Artificial interface also allows human-computer interaction in order to Intelligence]: Distributed Artificial Intelligence—Intelli- update and review the integrated data. This allows a collab- gent agents; K.6 [Management of Computing and In- orative approach that enables users to add new information formation Systems]: Software Management—Software de- pieces to our database and helps to keep the database up- velopment to-date as well. General Terms The remainer of the paper is organised as follows: In the next section, we present the chosen design of the system. In Database Section 3, we get into details of the architecture. Afterwards, Section 4 provides an overview about the implementation Keywords and the deployment of the system. Finally, we show some User interface, data aquisition, REST API related research and conclude the paper. 1. INTRODUCTION The AS-IT-IC platform4 was introduced in order to en- 2. DESIGN & ARCHITECTURE able tourists and tourism information provider to interact To fit the domain specific data requirements, we figured out and collaborate with each other. Several data sources ex- a data structure that allows to represent the data needed in ist from which the AS-IT-IC partners can retrieve informa- our platform, as shown in Figure 2. tion of sights, natural heritage as well as other events and tourism offers in the program area. The information from The Core Service Layer contains the main functionalities of these databases is not stored in an unified format, so in or- the web service. The layer hosts the data storage logic, the der to integrate this information into the AS-IT-IC platform data aquisition along with the merger functionalities. They it is nesseary to integrate those data into a fitting format. are encapsulated to ensure easy extensibility for new data The content subsystem, called Content API, integrates this sources to be included. We call these modules within the data and provides it to the AS-IT-IC ecosystem. The sys- initial startup sequence as well as on demand. tem collects information from other web sources in order to combine and complete the information for a specific offering. The possiblity to merge data from different sources together is crucial in order to present a good and up-to-date data The core items of these databases are: quality. This functionality is contained within the Core Ser- vice Layer. 1. Sights It is also possible to integrate new data sources in order to 1see docker.com add new information aspects into the database. 2see spring.io 3see vaadin.com The system is integrated into the architecture of the AS-IT- 4see as-it-ic.ijs.si IC platform as discussed in [1]. 270 In Figure 2, we show the Vaadin based UI containing sev- eral subforms to interact with the different entries from the database. 2.3 REST Interface The Representational State Transfer (REST) interface can be used to implement operations like create, read, update and delete (CRUD) the information of the content subsys- tem. There exist endpoints that represent the information in the database and allow to access them via JSON objects, i.e.: Figure 1: Layer of the Content API • Location 2.1 Update Process The update process collects several web data sources and • Points of interest (POI) integrates it into the database in an automated manner. • Equipment Every data source needs a specific update routine that is implemented to integrate the collectable data. These rou- • Transportation tines compare the recently collected data with the stored data stored and update the information in case that there is any change provided by the external source. The process We use these endpoints in order to access and modify the can be scheduled automatically or triggered manually. data in the system via HTTP-JSON calls. 2.2 User Interface 3. IMPLEMENTATION & DEPLOYMENT The web-based user interface (UI) allows the community to We use Docker within the implementation and the deploy- contribute information to the platform. It is possible to add ment step. Docker Compose is a tool for defining and run- new information items, such as sights or tourism offers, to ning multi-container Docker applications. In this project, the database. Reviewing existing items and updating them Docker Compose is used on the developer machine to set up represents also a very important opportunities for the user a testing environment, which configurations corresponds to community in order to allow a wiki-like contribution model. the productive pendant. 4. CONCLUSION In this paper, we presented the Content API, the content microservice and database of the AS-IT-IC platform. Here we focussed on the architecture and design of the system, as well as the implementation and the deployment. We also highlighted the use of the web-based user interface and the REST API as interaction possiblies. Finally we described the automated deployment using an application container technology as well as continous integration and deployment tools. 5. ACKNOWLEDGMENTS Research presented in this paper was carried out as part of the AS-IT-IC project that is co-financed by the Coopera- tion Programme Interreg V-A Slovenia-Austria 2014-2020, European Union, European Regional Development Fund. 6. REFERENCES [1] J. Zupancic, G. Grasselli, A. Tavcar, and M. Gams. Virtual Assistants for the Austrian-Slovenian Intelligent Tourist-Information Center. In Proceedings of the 20th International Multiconference INFORMATION SOCIETY - IS 2017, Volume E, 2017. Figure 2: Screenshot of the web-based UI 271 e-Tourist 2.0: an Adaptation of the e-Tourist for the AS-IT-IC Project Gregor Grasselli Jožef Stefan Institute Jožef Stefan International Postgraduate School Jamova 39 1000 Ljubljana, Slovenia gregor.grasselli@ijs.si ABSTRACT to provide the pieces of the user interface that are missing This article presents a new version of the e-Tourist sys- from Rocket.Chat. tem called e-Tourist2.0. The e-Tourist2.0 system is de- veloped within the Austrian-Slovenian Intelligent Tourist- Initially we believed that this role could be filled by the ex- Information Center (AS-IT-IC) project, accepted in the Co- isting e-Tourist after some light modifications. However, a operation Programme Interreg V-A Slovenia-Austria 2014- more thorough inspection of its capabilities and architecture 2020 call. The new e-Tourist2.0 system brings a number convinced us that extensive adjustments to the code base are of additional features with respect to the e-Tourist, such required. Additionally some of the functionality originally as location aware search, an implicit recommendation engine implemented in the application code is now supported by and a more interactive interface for trip planning. In the pa- the database itself which would lead to the deletion of sev- per we briefly explain the need for a new system, and present eral lines of code. The resulting conclusion was to write a the architecture and functionality of the e-Tourist2.0. new system using the latest tools and design patterns, while reusing the old code as much as possible. Keywords This rest of this article describes the need for a new system in trip planner; AS-IT-IC project; tourism; attractions; tourism Section 2, followed by a presentation of the new system Sec- database; recommendation engine; location aware search tion 3, where we present the features available through the application programming interface (API) of e-Tourist2.0. 1. INTRODUCTION In section 4 we present the system architecture and 5 con- Trip planners are applications that require a user to input a cludes the paper. number of predetermined parameters and then respond by offering the user a trip plan that can later be modified. The 2. WHY A NEW SYSTEM Austrian-Slovenian Intelligent Tourist-Information Center (AS- The required back-end functionality includes: IT-IC) platform provides a trip planner with a chat based user interface. It is built from three major components, each contributing to the final user experience. They are: 1. Trip planning 2. Location aware search 1. the front-end provided by a slightly modified version 3. Recommendation system of Rocket.Chat1 2. e-Tourist2.0 that fulfils the role of the back-end Initially e-Tourist was designed to plan trips visiting the coastal region of Slovenia.It was provided with a relatively 3. a conversational program (a bot) that takes user in- small list of tourist attractions in both regions and is capable puts, be they free form text input or button clicks, of planning a trip to either of them. The user can specify a from Rocket.Chat and generates responses, using data number of constraints like the exact start of their trip, and acquired from e-Tourist2.0 when necessary whether they want to eat along the way and the e-Tourist produces a nice trip recommendation. Based on how well In its function as the back-end for the AS-IT-IC platform, this works, we were convinced that all that needed doing was to add additional data with points for other regions of e-Tourist2.0 needs to support an assortment of queries over textual and geographical data (i.e. the ability to pro- Slovenia and Austria, spruce up the API so the bot would vide an answer to questions like “Which 5 points of inter- be able to use the system from outside, and we would have est are most similar to the Bled castle based on their de- our back end. This was presented in [1]. scriptions?” or “List all castles in the Gorenjska region”), a recommendation engine, that uses implicit data about user We are porting some of the trip planning capabilities of interest, as well as trip planning functionality. It also needs e-Tourist to e-Tourist2.0, and in this respect e-Tourist mostly meets our requirements, except for not being REST- 1https://rocket.chat/ ful, that is calls to the API needed session specific state with 272 them in order for it to return the correct result. Figure 1 2. Location aware search needs to be included. shows what the original user interface for trip planning looks 3. Proper database migrations have to be implemented. like. It shows the trip on a map and adds controls to add or remove points from it. 4. Database architecture needs to be reworked. 5. The way travel time was computed was very storage and time intensive. 6. e-Tourist is not REST-ful. 7. We need a more stateless way of user authentication. 8. Code base has to be updated to the newest framework versions. Besides the tree based structure mentioned in the intro- duction, that becomes unnecessary when storing geographic data correctly, the database schema of e-Tourist also failed to make use of several data structures offered by the Post- greSQL database engine that would make some queries sim- pler and more performant. The main problem with the database however was the lack of an initial migration that would create the relations required in the application. The authors assumed that future developers would use a copy of a pre created database for that, an approach that made cre- ating a new development environment as well as deployment Figure 1: an unnecessarily complicated task. e-Tourist trip planning interface As it turns out, the Because e-Tourist database stores each point e-Tourist does not comply with REST best prac- of interest in a tree-based data structure, based on its lo- tices, it is quite hard to use programmatically, which is es- cation – a point-of interest belongs to (or is a child in a sential for us since most of it is going to be used by the tree-based structure of) a settlement that in turn belongs Rocket.Chat bot, a program. The latter also means that to a region. This proves to be a very brittle way for storing session cookies are not a viable way for us to check user cre- larger amounts of points where some of the settlements have dentials and a different way of authenticating calls to the the same names and some areas of interest belong to multi- API is needed. ple settlements and even regions. Also the e-Tourist does not make it possible to ask queries based on actual distance, Finally the code used some features of an older version of the like “Which restaurants are within 1km of lake Bohinj?”. framework that were discarded in the newer ones. In order Therefore, a rewrite of the location aware search aspects of to be able to guarantee long term support for the project we the application would be required. decided that a newer framework version with longer support was required. In terms of the recommendation system e-Tourist requires users to rate the points of interest on a 1 to 5 scale or to Tallying up all of the above we figured it would take more input a lot of data about themselves like their education, work to make the necessary changes to the old system then age, gender so forth. It also requires data about points of writing a new one from scratch with an eye out for code interest that would be hard to acquire in an automated way, reuse whenever possible. like which age groups a point of interest is most appealing to or the level of education of the people who most enjoy their 3. FEATURES visit to the point of interest. Since the implementation of the In addition to the features already mentioned at the start of e-Tourist several modules have been developed that imple- the Section 2, the new e-Tourist2.0 supports: ment various recommendation algorithms and were released as open source software2. For reasons of stability, security 1. Finding similar points of interest and ease of maintenance we opted for one of those instead of using our own recommendation engine implementation. 2. A user interface that allows quickly adding points to a trip plan and deleting them from it by using a map According to the code analysis we reached the following con- clusions: 3. Exporting trips to Google Maps 4. Full text search for points of interest based on their 1. A recommendation system needs to be rewritten to descriptions, and limited by their location also include implicit information about user interaction 5. A recommendation engine that uses implicit data about with the system. user’s interaction with the system to find points that 2https://github.com/benfred/implicit/, http: might interest a particular user based on the user’s //surpriselib.com/ history 273 We will continue with a discussion of the features, what they 4. e-Tourist2.0 ARCHITECTURE do and why we need them. In order to make it easier to port code from e-Tourist to e-Tourist2.0 we chose to implement e-Tourist2.0 in the same framework as e-Tourist, Django5. Besides allowing us 3.1 Trip planning to more easily port code from the old project it also comes Trip planning means that given a list points of interest a with a built in administration interface that made imple- potential tourist wants to visit, the system can plan a route menting the user interface for tourist information providers on a map that visits all the points listed by the tourist. a lot easier, since we just had to customise the one provided The new e-Tourist2.0 does this by using the open source with the framework. routing machine [2] which uses open street map3data to plan the route. In addition to being able to plan the route, Our data storage is provided by the PostgreSQL6 database. e-Tourist2.0 is also capable of exporting that route to a By using the Postgis7 extension we were able to save geo- Google Maps link, so that users may conveniently follow graphical data and use it for several spatial queries. It also along using the Google Maps app on their devices. The trip supports full text search. In order to enable support for the planning user interface will also display potentially inter- Slovenian language we provided some language specific con- esting attractions near the route already chosen so that a figuration and files, while the German and English languages tourist may quickly add them to their trip plan. are supported out-of-the-box. 3.2 Recommendation 5. CONCLUSIONS Recommendations are a way to present the users with more We have presented a short description of the new of the relevant content based on their interests as shown e-Tourist2.0, describing the need for a new trip planner through their history of using the system. This will also implementation, its features and its architecture. While allow a registered user to simply ask the system “What is some of the e-Tourist code base was reused, the new interesting in Koroška?”, as well as provide additional sug- e-Tourist2.0 is mostly a new program. Most of the fea- gestions along the planned path. tures presented here are already fully functional, however, the program is not yet entirely complete and changes to Another use of a recommendation system is to compare at- existing features, or additional features are possible in the tractions based on user behaviour. This allows us to find future. similar attractions to the one picked by the user. Another way to find similar attractions is by comparing their descrip- 6. ACKNOWLEDGMENTS tions and e-Tourist2.0 uses both of them. This feature We thank students Tadej Petrič, Aljaž Glavač and Martin enables the users to quickly narrow in on what they want to Češnovar, who contributed to e-Tourist2.0 development. see or to just explore their options more conveniently. The work was co-funded by Cooperation Programme Inter- reg V-A Slovenia-Austria 2014-2020, project AS-IT-IC. 3.3 Full text search 7. REFERENCES Full text search is a way to quickly search a large database of [1] B. Cvetković, H. Gjoreski, V. Janko, B. Kaluža, documents for the ones containing given words or phrases. A. Gradišek, M. Luštrek, I. Jurinčič, A. Gosar, In e-Tourist2.0 this is coupled with some sentence anal- S. Kerma, and G. Balažič. e-turist: An intelligent ysis that attempts to produce more relevant results. It personalised trip guide. Informatica, 40(4):447, 2016. searches through attraction descriptions in German, English [2] D. Luxen and C. Vetter. Real-time routing with and Slovenian. openstreetmap data. In Proceedings of the 19th ACM SIGSPATIAL International Conference on Advances in All attractions and geographical features in e-Tourist2.0 Geographic Information Systems, GIS ’11, pages carry complete information about their location. Complete 513–516, New York, NY, USA, 2011. ACM. in the sense that regions and settlements are saved as poly- gons describing their borders, rivers are saved as lines de- scribing their entire flow and so on. This enables all sorts of location based search queries, as well as constraining other queries to certain locations. Examples of such queries are “List all museums in Slovenj Gradec” and “Find all attrac- tions similar to the Bled castle near Klagenfurt”. The most important feature from the standpoint of the tourist providers is the administration interface, which will allow them to add new points of interest to the system and make corrections to the data on those already there. To request administration access, a tourism provider should fill in a form available on the e-Tourist2.0 website4. 3 5 https://www.openstreetmap.org/ https://www.djangoproject.com/ 4 6 https://e-turist.docker-e9.ijs.si/ https://www.postgresql.org/ tourism-provider/request-admin 7https://postgis.net/ 274 Planning-based Security Testing for Chatbots Josip Bozic Franz Wotawa Institute for Software Technology Institute for Software Technology Graz University of Technology Graz University of Technology A-8010 Graz, Austria A-8010 Graz, Austria jbozic@ist.tugraz.at wotawa@ist.tugraz.at ABSTRACT natural language patterns, intelligent agents can be trained Chatbots are of increasing importance in modern day com- to learn from the communication with clients by relying on munication between users and industrial applications. For machine learning. In the long term, these smart chatbots example, providers of financial and medical institutions make refine their responses and provide better answers over time. use of intelligent agents in order to provide accessibility on a Advanced chatbots offer the client the possibility to authen- 24/7 basis. The human-like communication, often realized in ticate herself or himself, thus personalizing the connection an entertaining way, represents one of these advantages that between machine and person. Here the chatbot stores poten- chatbots offer. Eventually, chatbots make use of artificial tially sensible data from the user. In this case, the chatbot intelligence methods in order to learn from past communi- must guarantee user authenticity and data integrity. The cation interactions, to provide better and more personalized failure to fulfill such promises might result in personal and responses. Often chatbots are deployed as part of web ap- financial consequences for the user. plications. As a consequence, this makes them vulnerable to typical security attacks on websites. Planning-based tech- This requirement and the fact that chatbots can be deployed niques can help to identify security leaks for common attack as part of web applications leads to security issues, since web scenarios in a smart way. In this paper, we present such applications are known to be vulnerable to several hack- an approach that relies on artificial intelligence planning for ing attacks. Vulnerabilities like SQL injections (SQLI) and security testing of chatbots that are accessible using web cross-site scripting (XSS) are still very common [4], despite applications. security measures and new security testing approaches. It is interesting to know that the security issue in the context of Categories and Subject Descriptors chatbots have almost not been considered before. C.2 [Computer-Communication Networks]: General— Automated planning and scheduling, or simply planning, is Security and protection; D.2 [Software Engineering]: Test- a branch of artificial intelligence that was initially used in ing and Debugging; I.2 [Artificial Intelligence]: Distributed robotics and intelligent agents [17]. There planning is used Artificial Intelligence—Intelligent agents for guiding an agent by responding to encountered condi- tions. Some approaches have applied planning to security General Terms testing in specific domains as well [14, 16]. In order to Theory, Security contribute to the mentioned security issues, we introduce an automated planning-based security testing approach for Keywords chatbots in this paper. Planning, security testing, chatbots The paper is organized as follows. Section 2 introduces plan- ning to security testing of chatbots. Then, Section 3 gives 1. INTRODUCTION an overview about a test execution framework. Finally, Sec- ELIZA [21] is first known computer program that commu- tion 4 concludes the work and discusses potential goals for nicated in a natural language with a person, which was de- the future. veloped in 1966. Over the years, further improvements have been added to such similar applications, called chatbots [9, 20]. Chatbots are deployed in a stand-alone or online fash- ion, i.e., as part of websites in form of virtual assistants. 2. PLANNING FOR CHATBOTS Such programs offer the advantage of human-like communi- Planning has been already used in security testing to a small cation and as well an almost unlimited accessibility. Deploy- degree. The authors of this paper have applied planning for ing chatbots offer financial advantages for service providers testing of web applications (e.g. [7]) and the TLS proto- as well. They can be used in order to respond to customers’ col [6]. However, the application to chatbots is novel in inquiries, e.g., provide information about certain goods or this sense. The main motivation behind using planning for services without the need of human intervention. testing is the fact that attacks against applications can be depicted in form of a sequence of steps that could be applied In contrast to the initial versions of chatbots, where the pro- against every program. Actually, a plan acts as a blueprint grams usually responded to an inquiry according to stored for an attack. In this paper, we applied the planning speci- 275 fication from our previous work [5] but use it for identifying in it - s t a t u s two - status - si sq li vulnerabilities in the case of chatbots. xss - typ e get p ost h ead - m e t h o d u s e r n a m e - u s e r n a m e In general, the planning problem was initially given in [11] p a s s w o r d - p a s s w o r d ) and can be defined as follows. ( : p r e d i c a t e s ( i n I n i t i a l ? x ) ( i n A d d r e s s e d ? x ) Definition 1. A planning problem is defined by the tuple ( i n S e n t R e q ? x ) (I, G, A). A state is defined by a set of first order logic pred- ( i n R e c R e q ? x ) icates. I represents the initial state, the goal state is G and ( i n S Q L I ? x ) the set of actions is given by A. Every action a ∈ A com- ( i n X S S ? x ) prises a precondition and an effect. The functions pre(a) ( i n A t t a c k e d S Q L ? x ) and ef f (a) connect the individual preconditions and effects, ( i n A t t a c k e d X S S ? x ) respectively. ( i n F o u n d ? x ) ( E m p t y ? url ) ( F o u n d S c r i p t ? s c r i p t - s c r i p t ? re sp - If the precondition pre(a) of an action a is satisfied the cur- r e s p o n s e ) ) rent state S, then this action will be selected for the solution ( : a c t i o n S t a r t of the planning problem. The execution of this action will a : p a r a m e t e r s ( lead to a new state S0, namely S − → S0. This procedure ? x - s t a t u s will continue until the execution reaches the goal state G, ? url - a d d r e s s ) i.e. fulfills its preconditions. The program that reads the : p r e c o n d i t i o n ( and planning specification and searches for a solution according ( i n I n i t i a l ? x ) to a planning algorithm is called a planner. ( not ( E m p t y ? url ) ) ) : e f f e c t ( and ( i n A d d r e s s e d ? x ) Definition 2. The solution for the planning problem (I, G, A) is returned in form of a plan, which is given by a sequence ( not ( i n I n i t i a l ? x ) ) ) ) a a an−1 ( : a c t i o n S e n d R e q of actions ha 1 2 1, . . . , ani such that I −→ S1 −→ . . . −−−→ : p a r a m e t e r s ( a S n n−1 − − → G. ? x - s t a t u s ? se - status - se ? si - status - si ) The planning problem is implemented in the Planning Do- : p r e c o n d i t i o n ( i n A d d r e s s e d ? x ) main Definition Language (PDDL) [15]. Here, two specifi- : e f f e c t ( and cations have to be provided: ( i n S e n t R e q ? x ) ( not ( i n A d d r e s s e d ? x ) ) • Domain definition: Data that is present for every prob- ( a s s i g n ( s ent ? se ) 1) lem definition. ( s t a t u s i n i t two ) ) ) ( : a c t i o n P a r s e R e s p X S S C h e c k • Problem definition: Data that defines one specific prob- : p a r a m e t e r s ( lem. ? x - s t a t u s ? s c r i p t - s c r i p t ? re sp - r e s p o n s e ) PDDL supports a type-object hierarchy of data and uses : p r e c o n d i t i o n ( and it in conjunction with first-order logic predicates. Every ( i n R e c R e s p R X S S ? x ) object correspondents to a specific type, which relates to ( not ( F o u n d S c r i p t ? s c r i p t ? re sp ) ) ) variables and classes in object-oriented programming, re- : e f f e c t ( and spectively. The individual action definitions are built from ( F o u n d S c r i p t ? s c r i p t ? r esp ) parameters and pre- and postconditions, which are defined ( i n F o u n d ? x ) with one or more predicates. For example, an excerpt from ( not ( i n R e c R e s p R X S S ? x ) ) ) ) ) the the domain definition for chatbot testing is depicted be- low. Domain description in PDDL ( d e f i n e ( d o m a i n c h a t d o m a i n ) As can be seen, the PDDL definition encompasses, among ( : r e q u i r e m e n t s others, types, predicates and actions. Again, the individual : s t r i p s : t y p i n g : e q u a l i t y : f l u e n t s action definitions make use of the predicates and apply pa- : adl ) rameters in order to check if the predicate is valid. The spec- ( : t y p e s ification uses the parameter x to denote the current state of s t a t u s a d d r e s s s e r v e r status - si execution. As mentioned, the above domain, due to space status - se t ype e x p e c t r e s u l t u s e r n a m e reasons, does not include our entire specification. On the p a s s w o r d a c t i o n m e t h o d i n t e g e r sq li other hand, the problem definition is defined as follows. xs si r e s p o n s e s c r i p t ) ( : c o n s t a n t s 276 order to test for vulnerabilities. For example, model-based ( d e f i n e ( p r o b l e m c h a t p r o b l e m ) approaches usually rely on a model of the SUT [19, 18], ( : d o m a i n c h a t d o m a i n ) whereas fuzzing and combinatorial testing put emphasis on ( : o b j e c t s test case generation from a pentesting aspect [10, 13]. x - s t a t u s s - s e r v e r SQLI and XSS represent two common vulnerabilities for url - a d d r e s s many years and need further addressing for this reason. De- m - m e t h o d tailed information about these two vulnerabilities can be exp - e x p e c t found in [8] and [12], respectively. Chatbots, as already s c r i p t - s c r i p t mentioned, when deployed as part of a web application, in- re sp - r e s p o n s e ) herit the vulnerabilities as well. A scenario that depicts the ( : i nit entire planning-based security testing system is depicted in ( i n I n i t i a l x ) Figure 1. ( not ( E m p t y url ) ) ( M e t h o d pos t ) Attack vectors are malicious input strings that an attacker ( R e s p o n s e res p ) or tester submits against an application. For XSS, the list of ( not ( F o u n d S c r i p t s c r i p t res p ) ) attack vectors consists of JavaScript code, whereas malicious ( : g oal SQL statements are used for SQLI. As already mentioned in ( i n F i n a l x ) ) ) Section 2, a generated plan is used as an abstract test case. The reason for this is the fact that PDDL is limited with Problem description in PDDL regard to setting of concrete values for parameters. For this reason, we define a test execution framework, that encom- passes, among others, an executioner. This framework is The problem definition comprehends the definition of ob- implemented in Java and contains concrete Java methods jects and, most important, the initial state. This state rep- that correspond to the individual actions from PDDL. More resents the starting point from which the planner will start information about this mechanism can be found in [5]. the search. Modification of the initial state will result in the generation of a different plan. If no plan can be generated, The executioner reads the abstract plan and searches for the then the planner returns an error. A generated plan looks concrete counterpart of the individual actions. Then, HTTP as follows: requests are created with the help of HttpClient [1] and in- 0: S T A R T X URL stantiated with an attack vector. Then, the attack is carried 1: S E N D R E Q X SE SI out in form of the HTTP request. The SUT is a deployed 2: R E C R E Q X SI chatbot that encompasses a database. The chatbot has a 3: P A R S E X M U S E R N A M E P A S S W O R D T YPE user input field, e.g. an HTML element for textual inputs, 4: C H O O S E X S S X TY PE that represents the target for the executioner. The test or- 5: A T T A C K X S S X XS SI M UN PW acles specify what test output is expected and provide the 6: P A R S E R E S P X S S X S C R I P T RE SP final test verdict. We rely on our previously implemented 7: P A R S E R E S P X S S C H E C K X S C R I P T RE SP oracles from [7] for this purpose. After an attack, a parser 8: F I N I S H X reads the response from the SUT. It searches the HTML structure for critical vulnerability indicators, as specified in the oracles. In this scenario, we rely on jsoup [2]. The test- Generated plan for XSS ing process continues as long as the plan has been executed for every attack vector. As mentioned before, the plan is represented by a sequence of actions and corresponding parameters picked from the do- 4. CONCLUSION main definition. In our case, we used the planner Metric-FF In this paper, we introduced a security testing approach for [3]. Now, this sequence of steps acts as an abstract test case chatbots that relies on planning. After manually defining that will be executed by an executioner against the system the specification and generating the plan, a test execution under test (SUT). The purpose of the plan is to guide the implementation executes the plan in an automated manner. test execution process that, in the best case, will lead to the The approach is meant for testing of chatbots against two detection of a vulnerability. The main idea here is to apply common web vulnerabilities, namely XSS and SQLI. Under this plan against every chatbot that corresponds to the sce- the assumption that chatbots will play a major role in the nario as described in the next section. future, it remains important to address this issue. In the future, the proposed security testing approach will be evalu- ated against real-world applications and compared to other testing techniques. 3. SECURITY TESTING OF CHATBOTS Security plays a major role for every software system. Fail- ure to fulfill security requirements might lead to severe pri- Acknowledgments vate, financial and reputation consequences. For this rea- The research presented in the paper has been funded in son, programs have to be tested during the development part by the Cooperation Programme Interreg V-A Slovenia- lifecycle and after release of the software. Until now, many Austria under the project AS-IT-IC (Austrian-Slovene In- manual and automated approaches have been introduced in telligent Tourist Information Center). 277 Figure 1: Planning-based Chatbot Security Testing 5. REFERENCES Problem Solving. In Artificial Intelligence, pages [1] Apache HttpComponents - HttpClient. https: 189–208, 1971. //hc.apache.org/httpcomponents-client-ga/. [12] S. Fogie, J. Grossman, R. Hansen, A. Rager, and P. D. Accessed: 2018-09-06. Petkov. XSS Attacks: Cross Site Scripting Exploits [2] jsoup: Java HTML Parser. https://jsoup.org/. and Defense. Syngress, 2007. Accessed: 2018-02-02. [13] D. Kuhn, D. Wallace, and A. Gallo. Software Fault [3] Metric-FF. https://fai.cs.uni-saarland.de/ Interactions and Implications for Software Testing. In hoffmann/metric-ff.html. Accessed: 2018-09-06. IEEE Transactions on Software Engineering 30 (6), [4] OWASP Top Ten Project. 2004. https://www.owasp.org/index.php/Category: [14] A. Leitner and R. Bloem. Automatic Testing through OWASP_Top_Ten_Project. Accessed: 2018-01-31. Planning. Technical report, Technische Universität [5] J. Bozic and F. Wotawa. Plan It! Automated Security Graz, Institute for Software Technology, 2005. Testing Based on Planning. In Proceedings of the 26th [15] D. McDermott, M. Ghallab, A. Howe, C. Knoblock, IFIP International Conference on Testing Software A. Ram, M. Veloso, D. Weld, and D. Wilkins. PDDL - and Systems (ICTSS’14), pages 48–62, September The Planning Domain Definition Language. In The 2014. AIPS-98 Planning Competition Comitee, 1998. [6] J. Bozic and F. Wotawa. Planning the Attack! Or [16] A. M. Memon, M. E. Pollack, and M. L. Soffa. A How to use AI in Security Testing? In Proceedings of Planning-based Approach to GUI Testing. In First International Workshop on AI in Security Proceedings of the 13th International Software / (IWAIse), 2017. Internet Quality Week (QW’00), 2000. [7] J. Bozic and F. Wotawa. Planning-based Testing of [17] S. J. Russell and P. Norvig. Artificial Intelligence: A Web Applications. In Proceedings of the 13th Modern Approach. Prentic Hall, 1995. International Workshop on Automation of Software [18] I. Schieferdecker, J. Grossmann, and M. Schneider. Test (AST’18), 2018. Model-Based Security Testing. In Proceedings of the [8] J. Clarke, R. M. Alvarez, D. Hartley, J. Hemler, Model-Based Testing Workshop at ETAPS 2012. A. Kornbrust, H. Meer, G. OLeary-Steele, A. Revelli, EPTCS, pages 1–12, 2012. M. Slaviero, and D. Stuttard. SQL Injection Attacks [19] M. Utting and B. Legeard. Practical Model-Based and Defense. Syngress, Syngress Publishing, Inc. Testing - A Tools Approach. Morgan Kaufmann Elsevier, Inc., 30 Corporate Drive Burlington, MA Publishers Inc., 2006. 01803, 2009. [20] R. S. Wallace. The Anatomy of A.L.I.C.E. In ALICE [9] K. Colby. Artificial Paranoia: A Computer Simulation A.I. Foundation, 2004. of Paranoid Process. In Pergamon Press, New York, [21] J. Weizenbaum. ELIZA–A Computer Program For the 1975. Study of Natural Language Communication Between [10] F. Duchene, S. Rawat, J.-L. Richier, and R. Groz. Man and Machine. In Communications of the ACM KameleonFuzz: Evolutionary Fuzzing for Black-Box Volume 9, Number 1 (January 1966), 1966. XSS Detection. In CODASPY, pages 37–48. ACM, 2014. [11] R. E. Fikes and N. J. Nilsson. STRIPS: A New Approach to the Application of Theorem Proving to 278 Zbornik 21. mednarodne multikonference INFORMACIJSKA DRUŽBA – IS 2018 Zvezek F Proceedings of the 21st International Multiconference INFORMATION SOCIETY – IS 2018 Volume F Soočanje z demografskimi izzivi Facing Demographic Challenges Uredila / Edited by Janez Malačič, Matjaž Gams http://is.ijs.si 4. maj 2018 / 4 May 2018 Ljubljana, Slovenia 279 280 FOREWORD »FACING DEMOGRAPHIC CHALLENGES« It is the 11th annual conference “Facing demographic challenges” and it represents a milestone in our endeavor in several ways. Firstly, we managed to organize the conference in coordination with the meeting in the National Council of the Republic of Slovenia. The event had a major media effect with several conflicting viewpoints. Unfortunately, some scientific arguments were shadowed by non-scientific discussions, probably bringing this event to an undesired halt. Our motivation to help preserve European and Slovenian civilization, culture and position in the world was not accepted well by everybody. There were a couple of major demographic shifts in recent years in Slovenia: First, a decade ago the Slovenian birth rate was close to 1.2. Second, in recent years it has risen to around 1.6 and stalled there. Third, these events caused major discussion about the core scientific observations, presented also at the conference: a) The world is getting overpopulated and we need to bring the global birthrate towards the longterm ideal 2.1. While most of the world has stabilized its demographic growth, several parts like Africa are out of these terms. b) Europe and Slovenia with average birthrate around 1.6 will turn the current situation towards stagnation and degradation. c) It is important to incorporate mechanisms for decreasing birthrates in those countries with excessive birthrate, and for increasing birthrates in the developed world. d) It is important to understand that the exponential nature of demographics will hit hard unless treated in time. The ignorance might turn costly even to the extent of stagnation and degradation of world, European and Slovenian democracy. e) The influx of refugees has to be systematically taken care of by preventing mass exodus and continuing with a relatively small immigration. f) The question why demographic scientific revelations were being threatened as if having political or ideological orientations, remains unanswered. However, the impression is that the scientific knowledge is not as welcome as, for example, a decade ago. It seems that in recent times ideology is taking over science and interests of citizens. At the conference, we constantly emphasized that among often misunderstood issues is the effect of the European “demographic winter”. Governments, public opinions and media often fail to understand the importance of population in the domains of economics, politics and other components of societal and individual standard of living. Studies show that the demographic challenges are directly linked to education, work and employment, health care, financial problems, retirement and other important topics of everyday life. In plain words: with fertility rates around 1.6, the quality of life in Slovenia is significantly lowered compared to safe, say 1.8. Imagine just two statistical data: nearly 20% of Slovenian women do not experience maternity, while today’s men have 50% less sperm than 50 years ago, thus being severely hampered in their ability. Other problems included overload due to too many elderly, which is in itself a major civilization achievement; decreasing active work span, exodus of young educated generation (8,000 yearly in recent years, around 1,000 of the smartest 2,000 in the generation). 281 The "Facing Demographic Challenges" final conference deals with all questions related to the demography and population development, in particular: - fertility - economic aspects - mortality/health care - anthropological aspects - migrations - sociological aspects - population ageing - historical aspects - family - population projections - solidarity between generations - mathematical/computational models - gender relations - demography of national minorities - moral/ideological influences - theological aspects - (rural) planning in new demographical conditions - reforms We present our analyses and hypotheses in the top academic environment, openly presenting the worrisome future trends. Not only that, based on national, European and global studies, we also discuss various potential solutions and propose them to our leading politicians and political institutions. It is not that demographic science has solutions to all dilemmas, in particular in measures to raise the too-low fertility rate in the developed countries. New scientific studies in particular in relation to artificial intelligence indicate that there might be some rather simple and inexpensive solutions to solve the abovementioned problems. We continue to pursue an ambitious goal – to enlighten Slovenian and European leaders with demographic problems and knowledge to enable them to govern the society better. Now more than ever it is becoming clear that the voices of the demographic science should be carefully studied. Janez Malačič and Matjaž Gams 282 PROGRAMSKI ODBOR / PROGRAMME COMMITTEE Janez Malačič, predsednik Matjaž Gams, organizator Drago Čepar Christian Gostečnik Majda Černič Istenič Boštjan Kerbler Karin Kasesnik Dušan Kidrič Marko Krevs Tomaž Merše Mari Osredkar Janja Pečar Janja Povhe Jože Ramovš Jože Sambt Milivoja Šircelj Petronela Vertot Božidar Voljč MEDNARODNI ORGANIZACIJSKI ODBOR KONFERENCE / INTERNATIONAL PROGRAMME COMMITTEE Raimondo Cagiano de Azevedo, predsednik Enrico Todisco Guillaume Wunsch Serge Feld Catherine Withol de Wenden Antoine Pecoud Nadine Zielonke 283 284 Koliko nas manjka? Zakaj Vlada skriva obseg primanjkljaja ljudi v delovni starosti? Drago Čepar Tomažičeva ulica 34 1 000 Ljubljana Tel.00 386 41 677 850 drago.cepar@gmail.com POVZETEK kar za tretjino, zaradi česar se bo zmanjšalo tudi število Avtor s pomočjo statističnih podatkov o številu rojstev in živorojenih otrok….. Do leta 2030 bo tako število rojstev odseljevanju državljanov RS poskuša prikazati razsežnosti upadalo. « [2, str. 16] primanjkljaja ljudi v delovni starosti in poskuša pokazati, da je Ustavimo se pri skupini mlajših od 25 let, ki se postopoma primanjkljaj večji, če želimo ohraniti razmerje med zavarovanci in zaposluje. V letih 1991 do 2015 se nam je rodilo 264 tisoč otrok upokojenci. Z analizo novejših vladnih dokumentov ugotavlja, da premalo za enostavno obnavljanje prebivalstva. Ti otroci bi bili vlada pregled nad primanjkljajem ima, vendar njegove razsežnosti konec leta 2015 stari od 0 do 25 let. Z meddržavnimi priselitvami skriva. Vlada bi morala objaviti razsežnosti primanjkljaja, njegove ( V letih 1991 – 2016 je bil meddržavni selitveni prirast skupno srednjeročne projekcije, oceniti gospodarske in družbene 84 090 oseb.) je prišlo tudi nekaj otrok, ki so bili rojeni od posledice, časovno in finančno konkretizirati svoje ukrepe na tem začetka leta 1991 do konca leta 2015. Na osnovi starostne področju in oceniti njihov domet. porazdelitve teh priseljencev, lahko pa pridemo do dokaj zanesljive ocene, da jih je bilo okrog 18 tisoč. Primanjkljaj Ključne besede mlajših od 25 let je torej leta 2016 znašal 246 tisoč, kar je Demografija, rodnost, obnavljanje prebivalstva, primanjkljaj povprečno skoraj deset tisoč letno, in predstavlja 28% od števila rojstev, selitveni prirast, delovna starost, število zavarovancev, zavarovancev decembra 2016. število upokojencev, Bela knjiga o pokojninah, Strategija dolgožive družbe, Strategija razvoja Slovenije 2030, vladni 2.2 Zaradi odseljevanja državljanov RS ukrepi. Leta 2012 so odselitve za 5 450 oseb presegle priselitve. Negativna bilanca se je do leta 2016 povečevala in leta 2016 dosegla številko 5955; v letih 2012 1. UVOD -2016 se je odselilo 28 437 slovenskih državljanov več kot se jih je priselilo, torej povprečno Podnaslov vsebuje trditev, ki jo bom v nadaljevanju dokazal. 5 687 letno. Ob enakih gospodarskih pogojih brez pozitivnih Poglejmo, koliko več bi nas moralo biti v delovni starosti, da bi ukrepov je torej v prihodnjih letih pričakovati dodatni ne bila ogrožena gospodarstvo in blaginja, ter kaj o tem pravita, in primanjkljaj državljanov RS na ravni 6 tisoč oseb letno. česa ne povesta, vlada in ministrstva. 2.3 Za ohranitev razmerja med zavarovanci 2. PRIMANJKLJAJ in upokojenci Primanjkljaj po mnenju avtorja izvira iz prenizkega števila rojstev Po podatkih ZPIZ je bilo število zavarovancev konec leta 1916 pred desetletji in iz odseljevanja v zadnjih nekaj letih; če enako 889 tisoč. Število uživalcev pokojnin se je od leta 2007 do vzamemo za kriterij ohranitev razmerja med zavarovanci in januarja 2017 povečalo od 519 na 617 tisoč, torej letno za upokojenci, pa tudi povečevanje števila upokojencev. povprečno skoraj 11 tisoč, in se bo še povečevalo, » saj naj bi se v 2.1 Zaradi prenizkega števila rojstev obdobju od leta 2013 do 2060 življenjsko pričakovanje ob rojstvu za ženske povišalo za 5,6 let, za moške pa za 7,2 let.« Leta 1980 smo v Sloveniji ob rodnosti 2.11 otrok na žensko [2] Če bo zadnjič imeli dovolj rojstev za dolgoročno enostavno obnavljanje letna rast števila upokojencev ostala enaka 11 tisoč, bo za ohranitev razmerja med številom zavarovancev in upokojencev, prebivalstva. Letni primanjkljaj rojstev - to je razlika med da bi se torej ohranilo finančno zdravje javnih blagajn in številom, potrebnim za enostavno obnavljanje in dejanskim obstoječa raven blaginje, potrebno vsako leto številom dodatnih 15 tisoč - je potem naraščal, dosegel številko 13 tisoč letno, zadnja leta pa znaša okrog sedem tisoč letno. Če seštejemo letne zavarovancev. primanjkljaje do leta 2016 [1] dobimo seštevek 331 tisoč otrok, ki 3. KAJ PRAVI VLADA? O ČEM MOLČI IN bi leta 2016 dopolnili od 0 do 35 let, kar predstavlja 37% od 889 tisoč zavarovancev decembra 2016. KAJ SKRIVA? Ker se letni primanjkljaji seštevajo v ogromne zneske in presegajo Letni primanjkljaj je v letih 2011 - 2016 znašal 7 do 8 tisoč otrok, letno število rojstev, ki je bilo v obdobju 1990 - 2 000 med 18 in kar bi lahko pomenilo, da lahko ob nespremenjenih pogojih brez 23 tisoč, bi pričakovali, da bo ta demografska luknja, ki se od dodatnih ukrepov za zvišanje rodnosti, tolikšen letni primanjkljaj četrtine veča na preko tretjine števila zavarovancev, osrednja tema pričakujemo tudi v prihodnjih letih. Vendar bo ob odsotnosti vladnih razvojnih dokumentov, vsaj tistih iz zadnjih dveh let, ko pozitivnih ukrepov zadeva verjetno slabša in se bo število rojstev so vsi trije zapisani primanjkljaji dobro znani. Pričakovali bi, da zmanjšalo, saj demografi opozarjajo na » padec števila žensk v bodo zato podali konkretne, finančno ovrednotene ukrepe in starosti 25-34 let do leta 2030, in sicer v primerjavi z letom 2010 ocene, kaj od ukrepov lahko pričakujemo, ter kaj nas čaka, če ne 285 bodo uspešni. Poglejmo si tri vladne dokumente iz let 2016 in od kje ga bomo dobili. Strategija kot » krovni okvir razvoja 2017. države« (str. 49) bi morala naslavljati poglavitne naloge in izzive. Zavedanje problema na splošni deklarativni ravni sicer ni 3.1 Bela knjiga popolnoma odsotno, saj na strani 10 beremo, da » Nadaljnje Daje delne nižje podatke in napačne izračune. Na strani 12 razvojne možnosti tako omejujejo nizka produktivnost, preberemo: » Vsaka generacija mladih, ki sedaj vstopa na trg neprilagojenost demografskim spremembam, …)«, na strani 12, dela, je torej za okrog 20% oziroma 5 tisoč oseb manjša, kot bi pa, da » Demografske spremembe torej vodijo v relativno hitro bila sicer – in to vsako leto! Zato bomo imeli iz tega naslova v zmanjševanje zmogljivosti aktivnega prebivalstva, kar zaradi prihodnjih desetletjih za okrog 25 krat po 5 tisoč, torej skupaj pomanjkanja ustrezne delovne sile lahko tudi pomembno zmanjša okrog 75 tisoč manj oseb v delovni starosti, kot bi jih imeli sicer. « sposobnost za hitrejši gospodarski napredek, ki je pogoj za Dokument se izogne navajanju visokih številk tako da: ne poda nadaljnje izboljšanje življenjskega standarda prebivalstva. « ocene celotnega primanjkljaja rojstev , ampak le tisti del, ki ga Vendar je še to zapisano olepševalno, kot ovira za večje pripisuje odlaganju rojstev; celó iz vprašljivo ocenjenega letnega izboljšanje in hitrejši razvoj, ne pa kot huda grožnja sedanji ravni primanjkljaja 5 tisoč oseb izračuna 25 letni primanjkljaj v višini blaginje. 75 tisoč in ne 125 tisoč. Mediji so leto pred volitvami nehali objavljati svarila EU o 3.2 Strategija dolgožive družbe najslabši starostni sestavi Slovenije med vsemi članicami EU. Vlada je leta 2017 sprejela Strategijo dolgožive družbe [3], v Namesto načrtov, kako se lotiti primanjkljaja, ki nam že letos pije kateri na strani 11 ugotavlja padanje deleža prebivalstva v starosti kri, še bolj pa jo bo v naslednjih letih, poslušamo vladno 20–64 let ter potrebo po spodbujanju » priseljevanja delovne sile« predvolilno zgodbo o uspehu, višji rasti, proračunskem presežku, in » aktivnosti v vseh življenjskih obdobjih«. Zapiše, da to povečanju pokojnin, novih dela prostih dnevih, in poročila o spodbujanje tudi ob neto priseljevanju 4 000 oseb letno ne bo drugih predvolilnih golažih koalicije. Kot tisti, ki je padal iz 100 dovolj za zaustavitev padanja razmerja med aktivnim in nadstropnega nebotičnika in se je pri petdesetem nadstropju neaktivnim prebivalstvom, in da bo to padanje postalo omejitveni hvalil, da mu gre za zdaj zelo dobro. dejavnik za gospodarsko rast. Pove torej, da bo 4 tisoč premalo, ne pa, koliko bi bilo dovolj. Iz Slike 3 na strani 16 se z grafa da 4. SKLEP odčitati, da se bo starostna skupina 20-64 let od leta 2015 do 2040 Vlada torej navaja probleme in področja ukrepov, ne pa velikosti zmanjšala za 280 tisoč, kar je primerljivo našim izračunom, problemov in njihovih posledic, niti cene in pričakovanega učinka vendar je to podano le slikovno; bralec se mora potruditi, da se ukrepov. Kjer navaja ocene primanjkljaja ljudi v delovni starosti dokoplje do številčne ocene. in priseljevanja, navaja mnogo prenizke in pove, da ne zadoščajo; ne pove pa, koliko bi zadoščalo. Namesto, da bi številčno Med ukrepi v poglavju Zagotavljanje delovne sile (str. 39) prikazala svojo zapuščino, pred volitvami prodaja zgodbo o najdemo tudi prizadevanje za povečanje priseljevanja, zmanjšanje odseljevanja državljanov , spodbujanje vračanja uspehu. izseljenih državljanov, zagotavljanje možnosti za integracijo Omenjeni graf v Strategiji je moč narisati le iz številčnih tujcev … Vendar manjka konkretizacija in ovrednotenje cene ter podatkov. Če vlada zapiše, da nekaj ni dovolj, ve, koliko je učinkov ukrepov. Trditev, da bo to »omejevalo možnosti za dovolj. Vlada torej ve več, kot pove. Kakšen bolj realen podatek zagotavljanje in povečevanje blaginje prebivalstva«, zavaja v tudi uide. Tako je ministrica Anja Kopač Mrak povedala « Po razumevanje, da sedanja raven blaginje ne bo ogrožena. Bralec ne ocenah UMAR se bo delež aktivnega prebivalstva do leta 2020 v dobi občutka ogromnosti in usodnosti prihajajočih sprememb in povprečju zmanjšal za osem tisoč ljudi na leto. « [6] nujnosti ukrepanja. Vlada torej ima ocene primanjkljaja in s tem povezanega 3.3 Strategija razvoja Slovenije 2030 priseljevanja, ne le po obsegu, ampak verjetno tudi po državah Vlada jo sprejela 7. 12. 2017, z osrednjim ciljem kakovostno izvora. Ve namreč, da se države izvora dosedanjega priseljevanja življenje za vse [4]. Od petih strateških usmeritev ( vključujoča, praznijo: na strani 12 Bele knjige namreč preberemo: » Zadnje zdrava, varna in odgovorna družba;učenje za in skozi vse projekcije Eurostat-a predpostavljajo neto migracije od 4000 do življenje; visoko produktivno gospodarstvo, ki ustvarja dodano 6000 prebivalcev letno več do leta 2060. Ob upadanju neto vrednost za vse; ohranjeno zdravo naravno okolje; visoka stopnja migracij od leta 2010 dalje se postavlja vprašljivost te sodelovanja, usposobljenosti in učinkovitosti upravljanja) nobena predpostavke. « določno ne naslavlja pomanjkanja ljudi v delovni starosti. Iz Pravočasna dosegljivost ključnih demografskih parametrov je nobenega od dvanajstih razvojnih ciljev, niti iz množice 30 številčnih ciljnih kazalnikov, nujna za gospodarski razvoj. Če parlamentarne stranke vladi ki naj bi pomagali slediti njihovo dopuščajo, da jih omalovažuje ali morebiti celo prikraja, morebiti uresničevanje, ni videti zavedanja ogromnosti prebivalstvenega Državni svet lahko doseže ali vsaj pripomore, da vlada brez primanjkljaja, oziroma je razvidno zamolčevanje tega problema; uporabe kake inovativne poštevanke takoj objavi primanjkljaj izjema je kazalnik stopnje aktivnosti prebivalstva, ki naj bi se povečala od sedanje 70 na več ljudi v delovni starosti v naslednjem mandatu (in v naslednjih kot 75, in posredno naslavlja 10, 15, 20, 25 letih) za ohranitev razmerja med številom problem premajhnega števila zavarovancev. Med kazalniki zavarovancev in številom upokojencev, pri sedanji starostni vsekakor manjkajo totalna mera rodnosti, obseg odseljevanja sestavi prebivalstva, napovedanem podaljševanju življenjske državljanov Slovenije, obseg priseljevanja tujcev, razmerje med dobe, sedanji ravni odseljevanja državljanov ter sedanji zavarovanci in upokojenci. Celo pri cilju 5 Gospodarska pokojninski in drugi zakonodaji;tudi projekcije upokojevanja po stabilnost (str. 32) je problematika primanjkljaja ljudi v delovni starosti popolnoma odsotna. Kot bi do podrobnosti načrtovali letih. Oceni naj gospodarske in družbene posledice tega primanjkljaja za obseg gospodarstva, prihodke proračuna, lesno podjetje, pa bi niti ne omenili, koliko lesa potrebujemo in pokojnine, zdravstvo, šolstvo, …. Ukrepe za zmanjševanje 286 primanjkljaja naj konkretizira, oceni njihovo ceno, doseg in 5. REFERENCE učinek; pove naj, kolikšen obseg primanjkljaja namerava reševati [1] D. Čepar, ‘Koliko nas manjka? Migranti – grožnja ali s priseljevanjem, ter kako in iz katerih držav izvora bo uresničila priložnost?’, Slovenski čas, februar 2017. tak obseg priseljevanja. Razkrije naj načrte za dvig sedanje prenizke rodnosti, katere posledice bomo čutili v gospodarstvu [2] Ministrstvo za delo, družino, socialne zadeve in enake čez 25 let - ukrepe za vzpodbujanje odločanja za življenje v možnosti, ‘Bela knjiga o pokojninah’, 2016. mladih družinah. [3] Republika Slovenija, ‘Strategija dolgožive družbe’, 2017. Do teh podatkov imamo pravico državljani, politične stranke, ki [4] Vlada RS, ‘Strategija razvoja Slovenije 2030’, 2017. preko parlamenta in vlade usmerjajo razvoj, in vsi drugi tudi v [5] K. B., ‘Starejši delavec ne sme biti zgolj strošek’, Delo, 16. Državnem svetu zastopani dejavniki. December 2017. 287 Demografski trendi v svetu in Sloveniji Kako preprečiti izumiranje slovenskega naroda? Matjaž Gams Jozef Stefan Institute Jamova 39, 1000 Ljubljana, Slovenia matjaz.gams@ijs.si POVZETEK predvidoma 11,2 (EUROSTAT). Delež Slovencev v svetu se zmanjšuje z V prispevku so analizirani osnovni demografski trendi doma in po 0,6 promila leta 1950 na 0,2 promila leta 2015 in po svetu s poudarkom na izumiranju slovenskega naroda in na projekcijah na 0,09 promila leta 2100. Od druge svetovne vojne relacije, ki najmočneje vplivajo na to. Predlagani so tudi do 2100 se bo torej delež Slovencev v svetu zmanjšal za 6-krat. potencialni ukrepi kot način debate v pomoč pri reševanju Pomemben je še en pojav, opazen s slike 1. Medtem ko je leta problematike. Ključnega pomena pa je, da se slovenska javnost in 2015 skoraj 60% vse svetovne populacije prebivalo v Aziji in ko politika zavesta problemov, preden bo prepozno. je bilo leta 1950 le 9,1% Afričanov, bo leta 2100 44% Azijcev in 39% Afričanov. V času od druge svetovne vojne do 2100 se bo ABSTRACT delež Afričanov povečal za več kot štirikrat, medem ko se bo delež Evropejcev zmanjšal za skoraj štirikrat (EUROSTAT). We analyze basic demographic trends worldwide and in Slovenia. Posledice demografskih premikov so očitne tudi v ekonomskem The emphasis is on the forthcoming extinction of the Slovenian in drugih pogledih. Čeprav zahodna civilizacija preko svojega nation. Several relations, reasons, and possible improvements are NATO pakta ostaja dominantna vojaška sila, se ekonomsko identified. The key factor is that the media and governing gledano Kitajska prebija na mesto svetovno najpomembnejše acknowledge the trends and the final outcome unless major ekonomije, še značilnejše pa so projekcije srednjega razreda. Slika changes are introduced. 2 kaže izrazito zmanjševanje srednjega razreda v ekonomskem globalnem deležu. V resnici absolutno število srednjega razreda v Keywords zahodni civilizaciji rahlo upada, medtem ko se bo npr. na Artificial intelligence, demography, economy, world, Slovenia Kitajskem povečal na 700 milijonov leta 2030. Ko pa številčnost Ključne besede: srednjega razreda pomnožimo z ekonomskimi učinki, dobimo sliko 2. Trenutno po tem kriteriju prevladuje Evropa, ki pa jo Umetna inteligenca, demografija, ekonomija, svet, Slovenija dohiteva Kitajska, medtem ko projekcije kažejo, da bo leta 2030 Indija po tem kriteriju prehitela vse druge države. Slika 2 nazorno kaže, kako slabo nas o teh pojavih obveščajo svetovni mediji, saj 1. UVOD je večina bralcev ob tovrstnih podatkih običajno presenečena. Demografski pojavi so razen v primeru vojn, katastrof ali epidemij običajno dolgotrajni in zato pogosto nepomembni za javnost ter tudi politike, ki najpogosteje razmišljajo le o postopkih izvolitve, kar pomeni časovni razpon nekaj let. Posledica tega je, da so dolgoročne demografske napovedi v javnosti sprejete z nezainteresiranostjo in tudi skepso. Poglejmo sliko 1, ki kaže pričakovane spremembe v deležu svetovne populacija po celinah. Leta 1950, tj. po drugi svetovni Slika 2: Delež svetovne potrošnje srednjega razreda. Pomembno se je tudi zavedati, da je srednji razred nosilec kvalitete življenja: zgornji razred preveč obremenjuje okolje, spodnji razred pa nima dovolj kvalitetnega življenja. Premajhna rodnost vpliva na poslabševanje kvalitete življenja v ekonomskem Slika 1: Deleži populacije kontinentov v svetu. in drugih ozirih (Lee .. Sambt 2014). Iz svetovnih analiz je očitno, da se dogajajo izjemne vojni, je bilo Evropejcev skoraj 22 odstotkov. Leta 2015 jih je demografske spremembe v relativno mirnem obdobju človeške bilo le še 10%, leta 2100 jih bo le še 6%. Leta 1950 je bilo 2,6 zgodovine, kjer ni svetovnih vojn ali življenjsko nevarnih milijarde ljudi na svetu, 2015 7,4 milijarde in leta 2100 jih bo epidemij. Po spletu se pojavlja vrsta teorij, kaj se dogaja. Nekateri 288 celo trdijo, da gre za genetsko propadanje bele rase, kar se včasih Limits to Growth« je najbolj odmevno predstavil Rimski klub leta povezuje z raznimi »izmi« - rasizmi, šovinizmi, feminizmi, 1972 (Meadows 1972), ponovno izdano 26. novembra 2017. političnimi in drugimi ideologijami, in pozivanju k nasilju. Računalniške predikcije, temelječe na eksponentnih rasteh (še Znanost mora zato skrbno paziti, da ne prekorači meje starejši vir: Malthus; novejše EUROSTAT), so leta 1972 objektivnosti. Drugi celo trdijo, da gre za namerno uničevanje predstavili na konferencah v Moskvi in Riu de Janeiru. Izdanih je bele rase, kar se vidi npr. v zmanjševanju količine sperme pri bilo preko 30 milijonov knjig. Leta 2004 je izšla nova verzija: mladih moških (Slika 3). Dodaten »dokaz« naj bi bilo »The Limits to Growth: The 30-Year Update« (EUROSTAT). sistematično izigravanje tovrstnih informacij v medijih. To sicer Čeprav so bile prve napovedi o skorajšnjem pomanjkanju nafte in ne bi bilo nič novega, saj so v javnosti mediji pogosto drugih naravnih resursov pretirane, saj se npr. v Ameriki po sistematično zatirali znanstvena in strokovna spoznanja (npr. dolgoletnem zmanjševanju proizvodnje zadnja leta z drobljenjem nevarnost kajenja, globalno segrevanje, politični oporečniki itd.). skrilavcev hitro povečuje, je očitno, da so na vsakem omejenem Kljub temu se zdi povsem neverjetno, da bi lahko nekdo prostoru meje eksponentne rasti slej ko prej dosežene. Zaradi obvladoval medije v toliko državah in toliko različnih političnih narave eksponentne rasti, ki je našim možganom relativno tuja, saj usmeritvah. Kakorkoli, vsaj desetletje so posamezne objave kazale je v naravnem okolju skorajda ni, pa ljudje brez strokovnih študij na alarmanten trend zmanjševanja spermijev pri mladih moških, tega ne uspemo dojeti. . pri čemer so predlagale različne vzroke za to, npr. pretesne Najpomembnejša zaključka novih študij sta dva: spodnjice. Širša skupnost in predvsem mediji pa so se tem - Če se bo rast ljudi in industrije nadaljevala, bo pripeljala do raziskavam bolj ali manj posmehovali. Poglavitni znanstveni strmega »vrha« okoli leta 2070 in pripeljala do propada, tj. pomislek je bil, da so bile študije narejene na malem številu velikega padca populacije in gospodarstva primerov in tipično v eni državi. Šele leta 2017 je vrsto raziskav – z ustreznimi mehanizmi, predvsem zmanjševanjem demografske združil dr. Levine, vodja Environmental Health Track pri Hebrew rasti, bo možno preiti v trajnostno stanje. University-Hadassah skupaj z dr. Shanna H. Swanovo, profesorico v New Yorku in s tem prodrl v javnost (Levine 2016). Glede na razne študije ljudje že sedaj vsako leto porabimo precej Slika 3 opisuje pojav pretežno v Ameriki, vendar je značilen za več virov, kot jih zemlja letno proizvede. To velja zlasti za fosilne ves razviti svet (Evropa, Amerika), medtem ko ne velja za manj vire, ki so nastajali nekaj sto milijonov let, smo jih pa skoraj razvite kraje, recimo Afriko. Študija sama ne poda razlage, kaj je polovico že potrošili. Podobno smo preoblikovali že polovico vzrok za tako radikalno zmanjševanje, ki bi v nekaj generacijah svetovnega površja, s tem da smo uničili 5% površine. Tudi v lahko pripeljala do radikalnega zmanjšanja reprodukcijske Sloveniji nismo bistveno na boljšem glede normalnega odnosa do sposobnosti oz. kot pravi avtor: »sedaj je le kanarček v rudniku«. narave. Medtem ko imamo med največ kilometri avtocest na Najverjetnejša se zdi razlaga, da to povzročajo snovi, tj. prebivalca in kvadratnimi metri veletrgovin na prebivalca, smo onesnaževalci. glede preskrbe padli na mizerni nivo: medtem ko pride na Slovenca 800 m2 obdelovalnih površin, bi jih potrebovali 2000, zato smo na dnu Evrope po tem kriteriju. Ne glede na to pospešeno gradimo tako ceste kot nove tovarne na najbolj kvalitetnih površinah. Medtem ko se pred 40 leti nisi mogel poleti peljati na morje, ne da se ne bi vsaj enkrat ustavil in umil šipo, se sedaj lahko voziš cel teden. Študij in objav o tem ni, morda nekaj malega o izginjanju čebel, pri čemer nihče točno ne ve, podobno kot pri zmanjševanju sperme, kaj to povzroča. Še en »kanarček v rudniku«? Še bolj neprijeten je neposreden vpliv na okolje in živa bitja. Slika 4 kaže sovpadanje rasti človeškega prebivalstva z upadanjem / izumiranjem živalstva. Študije kažejo, da je - izumiranje vrst 100x hitreje kot normalno (de Vos 2014) - v 40 letih 50% živali manj na svetu - v 27 letih v Nemčiji 75% manj letečih žuželk. Strokovnjaki govorijo o šestem velikem izumiranju na našem planetu (Kolbert 2016). Če torej dosegamo meje rasti človeške populacije, ali se rast Slika 3: Količina spermijev se je v generaciji moških zmanjšala za človeške populacija ustavlja in zakaj? Odgovor je nenavaden, saj polovico. število ljudi hitro raste (http://www.worldometers.info/world- population/), kljub temu pa se svetovna rast prebivalstva ustavlja. Kaj je torej razlog za velike demografske spremembe? Poglavitna Število otrok zadnjih 10 let je precej konstantno. Svetovno raziskovalna teza tega prispevka je, da se človeška populacija prebivalstvo se v seštevku povečuje predvsem zaradi rasti na približuje trajnostni zmogljivosti tega planeta in da je potrebno določenih geografskih lokacijah in zaradi povečevanje življenjske ustaviti človeško demografsko rast. dobe. Rast pa se je globalno skoraj ustavila (Pearce 2008). 2. SMO DOSEGLI KAPACITETE PLANETA? Meje rasti, angleško »https://en.wikipedia.org/wiki/File:Club_of_Rome_Logo.svgThe 289 race (also Caucasoid or Europid)https://en.wikipedia.org/wiki/Cau casian_race - cite_note-2« (EUROSTAT)). Slika 4: Eksponentna rast prebivalstva zadnje stoletje sovpada z uničevanjem živalstva in okolja. V letih 1950-55 je bila rodnost (število otrok na žensko) 5. V Slika 5: Razmerje med prebivalci bele in črne polti Vir: National letih 2005-2010 je bila rodnost 2,5 in je sedaj padla na 2,4. Z policy institute. rodnostjo 2,5 bi v 10 generacijah svetovna populacija narasla na Slika 5 kaže glede na sliko 1 pričakovano spreminjanje razmerja 40 milijard, v 13 generacijah pa na skoraj 80 milijonov. Z rodnostjo 5 pa bi v 10 generacijah prišli na 40.000 milijard, v 13 med prebivalci bele in črne polti. Slika 6 kaže razmerja med generacijah bi prišel en prebivalec na 1 m2 površine, v 40 populacijskimi skupinami v Ameriki – leta 2040 bo manj kot generacijah pa 1 na kg Zemlje. Število Slovencev bi v 10 polovica belcev in do 2100 manj kot 20%, če se bo sedanji trend generacijah ob sedanjih trendih padlo na 3.5% prebivalstva v nadaljeval. Sloveniji – brez povečane emigracije. Morda bi kdo dejal, da se zgodovina aboriginov in Indijancev ponavlja, ampak potrebno se je zavedati drugačnih okoliščin: v 3. IZUMIRANJE SKUPIN LJUDI prvih dveh primerih je šlo za nasilno kolonizacijo civilizacijsko bolj napredne skupine ljudi, v sedanjem primeru izumiranja Izumiranje oz. praktično izumiranje določenih skupin ljudi je belcev pa gre za mirni proces s podobnimi posledicami. znano iz nedavne človeške zgodovine. Če pa gremo nekoliko dlje, Nepreverjena hipoteza tega prispevka je, da se belci »zavedajo« je Neandertalec živel v Evropi od 230.000 do 30.000 let nazaj. Bil preobremenjenosti planeta, da je potrebno ustaviti svetovno rast je zelo podoben nam, sodobnim ljudem, in še danes ima večina prebivalstva, in da imajo zato manj otrok. svetovne populacije razen Afričanov 1-2% neandertalskih genov. Še dalj je živel Homo floresiensis v Indoneziji (Zimmernan 2010), pradavna verzija malega človeka (hobit), ki je izumrl pred cca 10.000 leti. Leta 1492 je Krištof Kolumb odkril obe Ameriki, poseljeni z Indijanci. Ocena populacije se giblje nad 50 milijoni. V 150 letih pa je bila smrtnost skoraj 90% in sedaj je Indijancev manj kot 2 milijona v deželi s skoraj 330 milijoni, tretji najštevilčnejši državi sveta. Precejšen del Indijancev v Severni Ameriki je v rezervatih, ki so verjetno preprečili popolno izumrtje. Podobno se je zgodilo avstralskim aboriginom – pred kolonizacijo jih je bilo okoli 1,25 milijona, leta 1930 le še 50.000. Otok Mavricijus so po letu 1500 odkrili Evropejci in se nekaj časa borili med seboj za prevlado. Danes je Hindujcev / Indijcev 40%, Kreolov (Afro, Afro-Indijcev) 22%, indijskih muslimanov 16%, Tamilcev (južnih Indijcev) 4 ,5%, drugih Kreolov 3%, Kitajcev 2%, belih Francozov 1%, belih Južnoafričanov 1,5%. Slika 6: Populacijske skupine v ZDA. Belcev bo manj kot 50% Za primere evropskega dogajanja ni treba iti daleč. Pred nekaj leta 2040, če se razmerja ne spremenijo. Čeprav gre za objektivne desetletji je bila večina ljudi na ulicah Londona evropskega strokovne analize, se je potrebno ograditi od rasizma in podobnih izvora, danes pa jih je le približno polovica. Matere brez nezaželenih pojavov, zato je obzirnost pri tolmačenju številk matičnega državljanstva v evropskih državah rodijo od 10% do nujna. 30% otrok, ta delež pa je v zadnjih 10 letih zrasel za nekaj odstotkov. Kljub temu je večinsko prebivalstvo od 70-90% v Ameriška dogajanja so zanimiva, ker so podrobneje analizirana – večini razvitih zahodno usmerjenih držav (ZDA, Evropa, EU sestoji iz več nacionalnih držav, vsaka s svojimi študijami, Avstralija) evropskega izvora. Svetovno gledano je rodnost 2,4, Amerika pa je enotna. V letu 2016 je bilo vsega 60,2 rojstev na medtem ko je rodnost belcev 1,5. (Opomba: v svetovni strokovni vsakih 1000 žensk v starosti od 14 do 44 in rodilo se je 3,85 literaturi in v medijih se namesto izraza »belec« pogosteje milijonov otrok, 2% manj kot leta 2016. Pri tem se je uporabljajo drugi izrazi, npr. »evropskega izvora« ali »Caucasian proporcionalno rodilo 50% več otrok v družinah z manj kot $10.000 v primerjavi s tistimi z $200.000 in več. Strokovnjaki 290 ugotavljajo, da srednjeročno in dolgoročno tako stanje kulturno in 5. DISKUSIJA ekonomsko ni vzdržno. Predsednik Trump je sicer povečal davčne olajšave za otroka s $1.000 na $2.000, trenutno še brez učinka na Hipoteza tega prispevka je, da so se ljudje nezavedno odločili, da povečano rodnost. Mimogrede – študije avtorja ugotavljajo, da je potrebno ustaviti rast prebivalstva, na kar vrsto let opozarja tudi finance niso ključna komponenta za večjo rodnost. stroka. Podobno se je odločila Kitajska, kjer je partijsko vodstvo Razmere v Evropi so daleč slabše, saj je rodnost na naši celini najprej zapovedalo 1,5 otroka in sedaj 2 otroka na žensko. V 1,56 (približno kot slovenska), medtem ko je ameriška 1,76. Pri razvitem svetu ni možno zapovedati števila rojstev na tak, tem pa je treba upoštevati tako veliko imigracijo kot tudi bistveno ekspliciten način, zato pa mediji, sistem vrednot, sistem delovanja večjo rodnost pri prišlekih. Pri tem ameriški demografi bijejo plat družbe avtomatsko vodijo v zmanjševanje prebivalstva, ki je sedaj zvona in opozarjajo, da jih mediji ignorirajo. Vseeno občasno za belo raso v razvitem svetu izrazito z 1.5 otroka na žensko. pridejo na prvo stran poglavitnih medijev, kar se v Sloveniji še ni Ostali deli sveta, predvsem Afrika, tem smernicam ne sledijo, zgodilo (Fox news 2018). Še slabša so razmerja pri voditeljih. ampak vztrajajo pri visoko rodnosti. Posledično prihaja in bo še Voditelji 8 ključnih evropskih inštitucij (komisija EU, VB, bolj prihajalo do demografskega emigracijskega pritiska, ki bo Nemčija, Francija, Italija, Belgija, Luksemburg, Nizozemska) vršilo silovit pritisk predvsem na Evropo, ki je geografsko imajo skupno 2 otroka, leta 1951 pa so imeli ti voditelji 32 otrok. povezana z Azijo in Afriko (Gams 2015). Ob prepuščanju To ni nedolžno, kot morebiti izgleda na prvi pogled. Taki voditelji demografskim množicam bosta tako evropska kot slovenska pač ne skrbijo ne za otroke in ne za primerne družinske razmere, civilizacija relativno kmalu izginili oz. se transformirali, genetsko hkrati pa ljudje volivci volijo take politike, kar kaže na pa se bo evropsko prebivalstvo korenito spremenilo. prevladujočo miselnost. Demografi ugotavljajo, da je to posledica Politiki in javnost gledajo na tovrstna vprašanja ideološko, načina življenja in vrednot – zasledovanje kariere in užitkov politično, medtem ko je pogled v tem prispevku predvsem namesto dolgoročnih ciljev in naravnosti. Pojav je opisan že leta strokovno demografski, skozi število ljudi in zakonitosti, ki jih 1994 (Postman 1994) in se samo še potencira. prinašajo množice. Te demografske analize so dokaj nesporne, saj so številke dokaj nesporne – en človek je pač en človek – in 4. SLOVENSKA POPULACIJA omogočajo dokaj zanesljive analize vzrokov in napovedi. Zanimivo je primerjati analize, ki smo jih naredili z metodami Slovenska demografska gibanja smo analizirali v vrsti lastnih umetne inteligence, z drugimi strokovnimi analizami in potezami programov in smo jih primerjali s programi EU in EUROSTAT slovenske politike. Napovedi domačih in tujih strokovnjakov so v (Gams 2007, 20012). Izkazalo se je, da so predikcije zelo vseh primerih dokaj podobne in kažejo, da bo ob nespremenjenih podobne, kar pomeni, da je ob sedanjih predpostavkah prihodnost pogojih najverjetneje prišlo do napovedanega scenarija. Seveda pa dobro predvidljiva. Seveda pa lahko pride do spremembe razmer, se razmere lahko spremenijo in recimo na primeru migracije v kar vpliva na nadaljnje spremembe. Analize so objavljene v Evropo je težko napovedati, kako se bo Evropa odločila – povsem prejšnjih referatih konference »Soočanje z demografskimi izzivi«, odprla meje, jih priprla ali celo hermetično zaprla. Oba ekstrema ki je z dobrim desetletjem strokovnih analiz vodilno slovensko najbrž nista dobrodošla. srečanje na tem področju. Pri analizi slovenske politike in delovanja ministrstev opazimo Projekcije kažejo, da bo okoli leta 2100 v Slovenji le še pol večje razlike glede na naše in tudi druge strokovne analize. Na Slovencev po narodnosti. Podrobnejše analize so dokaj decidivno papirju vsi skrbijo za idealno rodnost 2.1, v praksi pa sprejemajo odgovorile, kaj so ključni vzroki za upadanje rojstev in s kakšnimi zakone, ki negativno vplivajo na rodnost (Gams 2007, 2008, ukrepi bi lahko povečali število slovenskih rojstev oz. preprečili 2011, 2012, 2013, 2017). Med njimi so: izumrtje Slovencev. - Preobremenjenost žensk. Sodobna ženska ima težko otroke, Č ker potem nima časa hoditi v službo in delati kariero. Če pa e na kratko povzamemo: je nezaposlena, pa nima sredstev za kvalitetno življenje. - S Slovenci se dogaja podobno kot z belci in Evropejci: z Nima tudi varnega okolja, ker jo lahko moški kadarkoli brez rodnostjo okoli 1,5 nam sledi demografsko nazadovanje in s posledic zapusti. Zato so predvsem mlade ženske pri iskanju tem povezano ekonomsko, marginalizacija, degradacija varnega materinstva najbolj diskriminirana skupina, vsaj po obstoječe kvalitete življenja in delovanja, nazadovanje mnenju avtorja. slovenske kulture in civilizacije. Spremembe ne bodo hipne, - Zmanjševanje pomena usmeritev, ki nagovarjajo k večjemu so pa že dokaj opazne. številu rojstev. - Zmanjševanje pomena in vloge družine, padanje ugodnosti in - Ključen je sistem vrednot in vpliv neoliberalno-globalističnih medijev, ki vsiljujejo manjšo rodnost, kar je dobrodošlo na posledično deleža porok, ukinjanje tradicionalnih vlog v svetovnem nivoju za krotitev preštevilčne populacije, za družini, forsiranje liberalnega in hedonističnega pogleda na Slovenijo in Evropo pa predstavlja čedalje večji problem. svet, promoviranje feminizma in raznih alternativnih oblik družine (Gams 2013) brez varnosti za otroke in mame, - Slovenski mediji, javnost, politika sledijo vzoru forsiranje egoističnega potrošništva in ukinjanje skupnosti, neoliberalnih medijev, ignorirajo opozarjanja in forsirajo ukinjanje stabilnih in kvalitetnih okolij za ženske – mame. take vrste družbo in vrednot in ukrepov politike, da - Sprejemanje ukrepov v smislu neoliberalnega globalizma in zmanjšuje slovensko rodnost. pretiranega potrošništva tudi na področju medčloveških odnosov, predvsem uničevanju vseh skupnosti od družin do Že sedaj pa je po podatkih statističnega urada SURS držav. (http://www.stat.si/StatWeb/Field/Index/17/104) 15% mladih od 0 - Ignoriranje pomena premajhne rodnosti, odhajanja najboljših do 14 let, 19,4% starejših od 65 let in 5,9% tujih državljanov. mladih v tujino in drugih demografskih problemov. Vsako leto iz Slovenije odide nekaj tisoč najsposobnejših v populaciji, ki ima manj kot 20.000 otrok (Dnevnik 2016). Pa se glede tega le malo razburjamo, veliko pa je govora o tem, da je 291 potrebno biti odprt v svet, da ne smemo ljudi nasilno zadrževati in [7] Gams. M., Krivec. J. 2008. Demografski pojavi in vzroki podobno. Recimo v nedavni debati v Državnem svetu so mi nekoliko drugače. V: MALAČIČ, Janez (ur.), GAMS, Matjaž uradni predstavniki zatrjevali, da odhajanje v tujino ni nič (ur.). Slovenija pred demografskimi izzivi 21. stoletja : slabega, saj vse nadomestimo z migranti. Ampak, če gre zbornik 11. mednarodne multikonference Informacijska pomemben del najsposobnejših mladih v tujino, to vsakemu družba - IS 2008. normalnemu posamezniku pomeni siromašenje bazena mladih [8] Gams. M., Krivec. J. 2007. Analiza vplivov na rodnost. V: prodornih in pametnih, polnih energije in poleta. Globalizem ne MALAČIČ, Janez (ur.), GAMS, Matjaž (ur.). Slovenija pred pomeni samo koncentracijo kapitala, ampak tudi kadrov, demografskimi izzivi 21. stoletja : zbornik 10. mednarodne posledično pa siromašenje vseh na obrobju. Finančne posledice so multikonference Informacijska družba - IS 2007, str. 18-23. precej hujše kot samo vlaganje v šolanje, ki znaša okoli 300.000 v posameznika. Problem je v tem, da se le malokdo vrne v [9] Gams, M. 2007. Ekonomski in demografski samomor Slovenijo. Če bi se vračali, bi bilo to zelo dobrodošlo po več Slovenije. Finance, ISSN 1318-1548, 9.03.2007, št. 47, 12. kriterijih. [10] Gams, M. 2007. Trije otroci bi nas rešili : demografska V zaključku: Hvalevredno je, da se rast človeške populacije gibanja Slovencev. Znanost : štirinajstdnevna znanstvena ustavlja. Pritisk na Evropejce, belce in planet pa ostaja oz. se priloga časnika Delo, 04.10.2007, letn. 49, št. 229, str. 19. povečuje, to pa je zaskrbljujoče. Zlasti je ta pritisk velik na Slovenijo, ki zaradi svoje majhnosti ne zmore dalj časa zdržati [11] Gams, M. 2007. Osnovna demografska gibanja. V: MALAČIČ, Janez (ur.), GAMS, Matjaž (ur.). tega trenda, ne da bi prišlo do marginalizacije slovenske Slovenija pred demografskimi izzivi 21. stoletja : zbornik 10. mednarodne populacije, civilizacije, kulture, jezika. multikonference Informacijska družba - IS 2007, str. 24-27. Zakaj slovenski politiki, mediji, množice ne začnejo biti plat zvona zaradi migracije, premalo rojstev, grozeče marginalizacije [12] Fox news 2018 Slovencev kot narodnosti in državljanstva (Gams 2017)? Jasnega http://www.foxnews.com/opinion/2018/05/20/heres-why-it- odgovora avtor po desetletju raziskav še vedno nima. Zdi se, da so matters-that-americans-are-having-fewer-children-than-ever- ljudske mase in politiki tako pod vplivom neoliberalnih before.html globalističnih pogledov svetovnih elit, da niso sposobni razumeti, [13] Postman M. 1994. The Disappearance of Childhood, First kaj je za Slovenijo in Slovence, tako državljane in državljanke kot vintage book edition. po narodnosti (slovenski, hrvaški, italijanski, srbski itd.) dobro in kaj ne. [14] Lee, R., Mason, A., Sambt, J., et al. Is low fertility really a problem?: population aging, dependency, and consumption. Science, ISSN 0036-8075, 10. Oct. 2014, vol. 6. REFERENCE 346, iss. 6206, str. 229-234, doi: 10.1126/science.1250542. [1] Gams, M. 2017. Globalizem in prednosti superinteligence za [15] Levine, H., Swan, S.H. 2016. Is dietary pesticide exposure Republiko Slovenijo, Prenova Evrope, Posvetovanje related to semen quality? Positive evidence from men Prispevki za slovenski nacionalni program II, urednik Tine attending a fertility clinic, Human Reproduction, Volume 30, Hribar, Slovenska akademija znanosti in umetnosti. str. 118- Issue 6, 1 June 2015, Pages 1287–1289. 128. [16] Meadows, Donella H; Meadows, Dennis L; Randers, Jørgen; [2] Gams. M., Krivec. J. 2008. Demographic analysis of fertility Behrens III, William W (1972). The Limits to Growth; A using data mining tools. Informatica : an international Report for the Club of Rome's Project on the Predicament of journal of computing and informatics, 2008, vol. 32, no. 2, Mankind (PDF). New York: Universe str. 147-156. Books. ISBN 0876631650. [3] Gams, M. 2015. Begunci 2015 - vrh evropske demografske [17] Kolbert. E. 2014, The Sixth Extinction: An Unnatural ledene gore. V: MALAČIČ, Janez (ur.), GAMS, Matjaž History, Bloomsbury, February 11. (ur.). Soočanje z demografskimi izzivi : zbornik 18. [18] Zimmerman., W. F. 2010. Homo floresiensis: the "hobbits" mednarodne multikonference Informacijska družba - IS of Flores Island (What Every Human Should Know). 2015 , Ljubljana, str. 12-14. [19] De Vos, J. M., Joppa, L.N., Gittleman, J.L., Stephens, P.R., [4] Gams, M., Grabnar, J., Vidulin, V. 2013. Vpliv pravic Stuart L., S. L. Pimm, S.L., 2014. Estimating the Normal istospolno usmerjenih na stopnjo rodnosti. V: MALAČIČ, Janez (ur.), GAMS, Matjaž (ur.). Background Rate of Species Extinction, Article first Soočanje z demografskimi published online: 26 AUG 2014 DOI: 10.1111/cobi.12380. izzivi : zbornik 16. mednarodne multikonference Informacijska družba - IS 2013, 7.-8. oktober 2013, [20] Pearce, F. 2008. Peoplequake, Mass Migration, Ageing Ljubljana, str. 13-17. Nations and the Coming Population Crash, Transworld Publishers. [5] Vidulin, V., Gams, M. 2012. Slovenske demografske projekcije in analize. V: MALAČIČ, Janez (ur.), GAMS, [21] EUROSTAT https://ec.europa.eu/eurostat/web/population- Matjaž (ur.). Soočanje z demografskimi izzivi : zbornik 15. demography-migration-projections/population-projections- mednarodne multikonference Informacijska družba - IS data 2012, str. 14-18. [22] Meadows, D., Randers, J., and Meadows, D. A Synopsis: [6] Gams, M., Krivec, J. 2011. Slovenske demografske Limits to Growth: The 30-Year Update projekcije in analize. V: MALAČIČ, Janez (ur.), GAMS, http://donellameadows.org/archives/a-synopsis-limits-to- Matjaž (ur.). Soočanje z demografskimi izzivi v Evropi : growth-the-30-year-update/ zbornik 14. mednarodne multikonference Informacijska [23] Dnevnik, Slovenijo letno zapusti več kot 8000 Slovencev, družba - IS 2011, str. 14-18. 2016, https://www.dnevnik.si/104274 292 KOMUNICIRANJE S SPLOŠNO JAVNOSTJO O ZDRAVJU IN ZDRAVILIH: Razumevanje besedil z vidika starejših oseb Karin Kasesnik NIJZ - SPC Trubarjeva 2, 1000 Ljubljana +386-1-58-63-908 karin.kasesnik@nijz.si POVZETEK Determining readability of texts, analyzing appeals in the texts Zdravje je ena največjih vrednost. Informacije o zdravju in and assigning of importance to specific attributes is a part of zdravilih nas spremljajo na vsakem koraku; nekatere informacije researching health and medicine texts. The results of the research načrtno iščemo, druge pa nas dosežejo, ne da bi posebej achievements may lead to the measures for improving poizvedovali za njimi. Poleg ustreznih sporočil o zdravju in comprehension of information. Special emphasis must be zdravilih je razpoložljivih veliko informacij, ki niso ustrezne, ali dedicated to the vulnerable groups, including older persons. so celo zavajajoče. Dobro razumevanje informacij o zdravju in Population ageing is an important part of demographic changes zdravilih je bistvenega pomena za vedenje oseb, ki zagotavlja and also represents a challenge for the public health system. učinkovito in varno zdravljenje. Appropriate comprehension of health and medicines information Starejše osebe se pogosto soočajo s težavami pri pridobivanju contributes to an efficient maintaining of health and treatment. informacij o zdravju in zdravilih, velikokrat zaradi zdravstvenih razlogov. Pojavljajo se lahko tudi težave z razumevanjem pisnih Keywords sporočil o zdravju in zdravilih. Poleg besedil z opisom zdravil, ki Health, Health and medicines information, Efficient and safe se predpisujejo na recept, je pomembno tudi ustrezno treatment, Demographic changes, Older persons, Readability, razumevanje besedil, ki opisujejo zdravila brez recepta. Določitev Appeals, Perceived importance of attributes. berljivosti besedil, analiza apelov v besedilih in pripisovanje pomembnosti specifičnim atributom so del raziskovanja besedil o zdravju in zdravilih. Rezultati raziskovalnih dosežkov lahko vodijo v ukrepe za izboljšanje razumevanja informacij. Poseben 1. UVOD poudarek mora biti namenjen ranljivim skupinam, vključno s V sedanjosti se vsakodnevno soočamo z velikim številom starejšimi osebami. informacij, vključno z informacijami o zdravju in zdravilih. Za nekaterimi izmed informacij načrtno poizvedujemo, veliko Staranje prebivalstva je pomemben del demografskih sprememb informacij pa ne iščemo, niti jih ne potrebujemo, pa nas vseeno in predstavlja tudi izziv za sistem javnega zdravja. Ustrezno dosežejo. Pomembno je izbrati informacije, ki so zanesljive, razumevanje informacij o zdravju in zdravilih prispeva k uporabne in bodo imele pozitiven vpliv na naše zdravje. Kar pa ni učinkovitemu ohranjanju zdravja in zdravljenju. vedno enostavno. Ključne besede Informacijam o zdravju in zdravilih pripisujemo velik pomen. Zdravje, informacije o zdravju in zdravilih, učinkovito in varno Zdravje je opredeljeno kot ena največjih vrednost. Rezultati zdravljenje, demografske spremembe, starejše osebe, berljivost, raziskave (1) kažejo, da je ovrednotenje zdravja v skupini mladih apeli, zaznana pomembnost atributov. žensk z leti naraščalo; ženske srednjih let pa so zdravju pripisale večjo vrednost kot moški srednjih let. Tudi v slovenski raziskavi ABSTRACT (2) je zdravje opredeljeno kot najpomembnejša vrednota mladih. Health is one of the most important values. We are accompanied Na eni strani se srečujemo s sporočili, ki so ustrezna in se jim by health and medicines information at every step; some pripisuje pozitiven vpliv na zdravje, pa tudi s sporočili, ki so information is intentionally searched for and other information neuporabna, ali celo zavajajoča. Posebej internet je neobvladljiv reaches us without a demand. Besides appropriate health and vir informacij, ki ga velik del prebivalstva, predvsem pa mlade medicines messages, a lot of inappropriate information or even osebe, uporablja za sporazumevanje. Poleg informacij, ki misleading information is available. Good comprehension of pozitivno vplivajo na naše življenje in zdravje, je nemalo takih, ki health and medicines information is essential for patients’ imajo negativen, lahko celo škodljiv in ogrožajoč vpliv. Raziskava behavior which is in accordance with efficient and safe treatment. (3) kaže, da kakovost spletnih informacij o zdravju vpliva na usklajeno komunikacijo med zdravnikom in pacientom; iz te Older persons are often confronted with the difficulties in ugotovitve izhaja pomen zdravnikovega usmerjanja pacientov k obtaining health and medicines information, many times due to pomembnim virom informacij o zdravju. the health issues. Also the difficulties with comprehension of written health and medicines messages may appear. Besides the Tudi bolj tradicionalni viri informacij o zdravju in zdravilih, kot texts, describing the prescribed medicines, also a proper so na primer pisni, tiskani viri, so kljub zakonodajnim zahtevam comprehension of texts, describing OTC medicines is important. (4) nadzorovani le v omejenem obsegu. 293 Razumevanje sporočil o zdravju in zdravilih se zdi samoumevno. Poleg tega je izdaja zdravil brez recepta obsežna. Skladno s Uvedle so se izboljšane vsebine določenih uradnih virov o podatki za leto 2016 je bila vrednost trga zdravil na recept v zdravilih, kot so na primer navodila za uporabo zdravil. Vendar iz Sloveniji 201,0 milijona evrov, vrednost trga zdravil brez recepta različnih virov izhaja veliko vsebin, ki so zasičene z medicinsko pa 27,9 milijonov evrov, se pravi je razmerje znašalo približno 7 terminologijo, se lahko interpretirajo na zelo različen način, ali pa proti 1 (6). Skupno je trg zdravil brez recepta in sorodnih izdelkov so celo zavajajoče. Izkušnje kažejo, da strokovnjaki pri dajanju v prvi polovici tega leta dosegel 54,4 milijona evrov, kar je pojasnil o zdravljenju z zdravili brez recepta menijo, da je pacient pomenilo 1,0 % rast glede na leto prej. V največji meri so se nasvet prejel od drugega strokovnjaka, ali pa se zanašajo, da je prodajala zdravila brez recepta za lajšanje bolečine. natančno prebral navodila za uporabo. Obseg svetovanja je odvisen tudi od razpoložljivega časa, zavedanja pomena informacij o zdravju in zdravilih ter od drugih dejavnikov. 2.2 Način informiranja starejših oseb o V raziskavah so med merila kakovosti informacij zajeli zdravju in zdravilih natančnost, popolnost, berljivost, oblikovanje, jasnost informacij Starejše osebe pogosto zbolevajo zaradi več sočasnih bolezni in in vključenost referenc. Raziskovalci (5) so pri pregledu zbirk jemljejo tudi več zdravil hkrati. Zaradi spremljajočih bolezni podatkov o člankih ugotovili, da avtorji kar v 70 % člankov starejše osebe težje spremljajo informacije o zdravju in zdravilih. ugotavljajo, da kakovost informacij o zdravju ni ustrezna, Poleg tega velik del starejših oseb ni seznanjen z napredno zaključki 22 % raziskav kažejo nevtralne rezultate, le iz 9 % tehnologijo, ki bi jim olajšala spremljanje informacij. raziskav pa izhaja pozitivna ocena kakovosti informacij. Pri raziskovanju iskanja informacij o zdravju oseb so ugotovili, da Razumevanje informacij se povezuje z odločitvami, namerami in starejši od 55 let ocenjujejo zdravstvene delavce kot najbolj vplivom na vedenje. Težave nastopijo, ko je vedenje neustrezno zanesljiv vir informacij o zdravju (7). Raziskovalci zaradi tega do te mere, da se poslabša zdravje. Neustrezno jemanje oziroma poudarjajo pomen orodij za izboljšanje komuniciranja med ravnanje z zdravili brez recepta lahko vodi v slabšo učinkovitost osebami te starosti in zdravstvenimi strokovnjaki. Obenem zdravil in/ali povečano tveganje zaradi pojava neželenih učinkov omenjajo pomen pomoči osebam iz tega starostnega segmenta, da ali medsebojnih učinkovanj med zdravili. bodo uporabljale in v večji meri zaupale zanesljivim spletnim Glede na demografske spremembe, katerih pomembni del je virom. staranje prebivalstva, se potrebujejo strategije in ukrepi za poučevanje starejših oseb o informacijah o zdravju in zdravilih, vključno z usmerjanjem k zanesljivim virom informacij. Starejše 3. BERLJIVOST IN DOLOČANJE osebe pogosto jemljejo več zdravil hkrati, različne bolezni pa jim onemogočajo primerno razumevanje informacij RAZMERJA MED APELI Na osnovi V svetovnem merilu je objavljenih več raziskovanja bi bilo koristno ustvarjati gradiva, ki bodo z večjo raziskav o berljivosti, s razumljivostjo pripomogla k učinkovitejšem poudarkom na področju zdravja in izobraževanja. u in varnejšemu zdravljenju. Berljivost se navezuje na branje in razumevanje besedila. Če je sposobnost branja znatno pod nivojem berljivosti dokumenta, se razumno predvideva, da posameznik ni sposoben popolnoma 2. RAZISKOVANJE INFORMACIJ O razumeti besedila (8). VIRIH INFORMACIJ O ZDRAVJU IN ZDRAVILIH 3.1 Določitev berljivosti besedil in opredelitev apelov v besedilih o zdravju in zdravilih 2.1 Izhodišča za raziskovanje sporočil v Razvili so več formul za določitev berljivosti, ki vsebujejo različne gradivih o zdravilih brez recepta spremenljivke. Ena bolj uporabljenih formul je Flescheva formula za določitev berljivost besedila oziroma Flesch- Zdravila brez recepta so opremljena z navodili, ki jih pregledajo Kincaidova formula za določitev stopnje berljivosti (9). Na osnovi strokovnjaki iz pooblaščene uradne ustanove. Iz navodil za te formule se berljivost določi iz števila besed v stavkih oziroma uporabo zdravil brez recepta lahko razberemo podatke o števila zlogov v besedah. S stopnjo berljivostjo določimo, kateri učinkovitosti, jemanju zdravil, indikacijah, možnih neželenih stopnji izobrazbe ustreza zadevno besedilo. Če ugotovljena učinkih, kontraindikacijah, medsebojnem učinkovanju z drugimi stopnja berljivosti besedila presega priporočeno stopnjo, zdravili in druge pomembne podatke. Zdravila brez recepta lahko predvidevamo, da razumevanje besedila ne bo ustrezno. medsebojno učinkujejo tudi z zdravili na recept. Vsebina Obstajajo tudi druge formule določanja berljivosti besedil, ki promocijskih gradiv o lastnostih zdravil brez recepta pa se le upoštevajo dodatne spremenljivke, na primer število črk, delež delno nadzoruje s strani uradnih ustanov; to velja za tiskana lažjih in delež zahtevnih besed ter druge gradiva, še v večji meri pa za spletne informacije. spremenljivke. Smisel določanja berljivosti na področju izobraževanja je, da so Izhajali smo tudi iz opažanja, da izdajo zdravil brez recepta učbeniki prilagojeni večinoma ustrezni stopnji berljivosti, kar se povezuje z spremlja manj pojasnil kot izdajo zdravil, ki se večjim razumevanjem besedila. Tudi pri gradivih z informacijami predpisujejo na recept. Deloma pacienti prejmejo nasvete o zdravljenju z zdravili brez recepta v lekarni. Čeprav b o zdravju in zdravilih velja povezanost med stopnjo berljivosti in i morali razumevanjem besedila, ki nadalje vpliva na jemanje in ravnanje z navodila za uporabo prebrati, vsi pacienti tega napotila ne upošteva zdravili. Stopnje berljivosti besedila morajo biti čim bližje jo. priporočeni stopnji berljivosti besedil. Ustrezne stopnje berljivosti 294 pa so seveda eden izmed dejavnikov za zagotavljanje dobrega Rezultati (14) kažejo, da so respondenti primerjalno večjo razumevanja besedila. pomembnost pripisovali skupini atributov s tveganji in sicer Rezultati raziskave slovenskih gradiv so pokazali, da berljivost 49,77, glede na skupino atributov, ki opisujejo prednosti jemanja zdravil brez recepta, to je učinkovitost besedil o zdravju in zdravili ne ustreza priporočenim stopnjam oziroma prikladnost (10). Podobne ugotovitve so objavili tuji raziskovalci (11). jemanja (20,62 oz. 18,64). Ceni za razliko od rezultatov drugih Stopnje berljivosti slovenskih besedil so bile neustrezne tako pri raziskav (15) respondenti niso pripisovali velike pomembnosti. To analizi gradiv, namenjenih strokovni javnosti, kot tudi gradiv, razlagamo s primerjalno nizko ceno zdravil brez recepta in namenjenih splošni javnosti obenem z manjšo razliko med cenami zdravil (10). (14), glede na primerljivo raziskavo (15). Pomembno je analizirati tudi apele v besedilih in njihovo razmerje, za oceno vpliva na razumevanje sporočila. V raziskavi Iz teh preliminarnih rezultatov (14) izhaja velika razlika v (12) smo določili razmerje med apeli o učinkovitosti in apeli o preferencah med atributi o prednostih zadevnih zdravil brez tveganjih zdravil. Raziskovalci smo pri analizi slovenskih besedil recepta in atributi, ki opisujejo tveganja. Tem rezultatom glede ugotovili, da apeli o prednostih analiziranih zdravil brez recepta zaznane pomembnosti atributov o tveganjih pritrjujejo tudi niso bili uravnoteženi z apeli z opisanimi tveganji teh zdravil (12). rezultati primerljive, objavljene raziskave (16). 3.2 Berljivost z vidika starejših oseb 4.2 Zbolevanje in izdaja zdravil brez recepta glede na spol, starost in izobrazbo Raziskovalci ugotavljajo (13), da starejši odrasli pogosto težko razumejo pisna gradiva o zdravju in zdravilih. Težave z respondentov razumevanjem besedil povezujejo s posledičnim poslabšanim Respondenti so v spletni anketi odgovarjali glede pogostnosti zdravstvenim stanjem in več sprejemi v bolnišnico. Obenem ta nakupa zdravil brez recepta v lekarni (14). Največji delež in sicer težava zadeva tudi zdravstveni sistem, zaradi visokih stroškov za 75,3 % oseb je ta zdravila nabavilo do nekajkrat letno, kar je zdravstveno varstvo. Raziskovalci poudarjajo pomen izboljšanega skladno z odgovori o zbolevanju zaradi virusnih bolezni od enkrat razumevanja besedil, z uporabo z zdravjem povezanih gradiv, ki do trikrat letno, ki je znašalo 78,9 %. Manjši delež respodentov je vsebujejo ustrezne stopnje berljivosti. zdravila brez recepta nabavil enkrat ali večkrat na mesec. 92,3 % Raziskava (13) je pokazala, da je razumevanje z zdravjem respondetov, ki so kupovali zdravila brez recepta večkrat na povezanih besedil odvisno od več dejavnikov. Predlagajo dve mesec, je sodilo v starostno skupino nad 50 let, vsi pa so imeli strategiji za izboljšanje razumevanje pisnih, z zdravjem povezanih podiplomsko izobrazbo. besedil. Prva strategija zajema uporabo kratkih besed in kratkih Med osebami, ki so prebrale navodilo za uporabo, jih je bilo malo stavkov, da bi izboljšali berljivost, skladno s Fleschevo formula za več kot polovica starejših od 50 let in malo manj kot polovica s določanje berljivosti. Pri starejših odraslih je treba upoštevati tudi podiplomsko izobrazbo (14). možnost zmanjšane sposobnosti pomnenja, kar lahko vodi do težav z razumevanjem. Druga strategija je ponavljanje pomembnih besed, fraz in zamisli, da bi povečali celovitost besedila. Opisani strategiji se morata uporabljati sočasno, da se 5. ZAKLJUČEK doseže pričakovani učinek. Izsledki raziskav kažejo pomembnost rezultatov za ustvarjanje strategij z namenom povečanja razumevanja besedil o zdravju in zdravilih. Razumevanje besedil vpliva na odločitve pacientov, 4. PRIPISOVANJE POMEMBNOSTI jemanje in ravnanje z zdravili ter posredno na učinkovitost in ATRIBUTOM O UČINKOVITOSTI IN varnost zdravljenja. Rezultati raziskav kažejo, da besedila o zdravju in zdravilih TVEGANJIH ZDRAVIL BREZ RECEPTA pogosto niso prilagojena razumevanju oseb. Posebno pozornost je Raziskovalci (14) smo izvedli pilotno raziskavo, s katero smo treba nameniti ranljivim skupinam prebivalcev, vključno s določali preference 85 respondentov. Opredeljeni atributi so starejšimi osebami, ki jim spremljajoče bolezni nemalokrat opisovali prednosti: to so pretežno atributi, ki v analiziranih onemogočajo ustrezno razumevanje besedil. Rezultati raziskav so besedilih opisujejo učinkovitost in delno prikladnost jemanja odlična osnova za ukrepanje, ki obsega spremljanje pisnih besedil zdravil brez recepta, ki so indicirana za lajšanje simptomov in drugih vrst komuniciranja ter prilagoditev vsebin z namenom virusnih bolezni. Naslednja skupina atributov je opisovala boljšega razumevanja. morebitna tveganja, povezana z jemanjem teh zdravil; zadnja Zaradi staranja prebivalstva v razvitih družbah se bo predvidoma skupina atributov je opisovala ceno. Atributi so vzorčeni deli povečeval problem razumevanja informacij o zdravju in zdravilih. besedila iz navodil za uporabo zdravil brez recepta. Nadaljnje raziskovanje bo imelo raziskovalni pomen, pa tudi pomen za zdravje prebivalstva, posebej ranljivih skupin 4.1 Atributi o učinkovitosti in tveganjih prebivalstva. zdravil brez recepta ter ceni Bodoči raziskovalni napori bodo predvidoma usmerjeni k analizam besedil o zdravju in zdravilih, tudi na večjih vzorcih in z S preliminarnimi rezultati smo dobili podatke o delnih vključevanjem dodatnih skupin zdravil, ob hkratnem sodelovanju preferencah (14). Uporabljali smo conjoint metodo, s katero z drugimi ustanovami. respondenti skladno s svojimi preferencami pripisujejo pomembnost posameznim atributom. 295 6. LITERATURA [1] Lau, R. R., Hartman, K. A., in Ware, J. E. 1986. Health as a [10] Kasesnik, K., in Kline, M. 2011. Analyzing readability of value: Methodological and theoretical considerations. Health medicines information material in Slovenia. Southern Med Psychology, 5(1), 25-43. Review, 4;2, 33-40. [2] Musil, B. Stabilnost in sprememba v vrednotah slovenske [11] Williamson, J. M., Martin, A. G. 2010. Analysis of patient mladine: Tradicija in samoekspresivnost proti tekmovalnosti information leaflets provided by a district general hospital by in optimizmu. http://www.irdo.si/skupni-cd/cdji/cd-irdo- the Flesch and Flesch-Kincaid method. Int J Clin Pract, 64, 2011/referati/a-musil.pdf. 13, 1824-1831. [3] Laugesen, J., Hassanein, K. in Yuan, Y. 2015. The Impact of [12] Kasesnik, K., Kline, M., Gammie, T., in Babar, Zaheer-Ud- Internet Health Information on Patient Compliance: A Din. 2016. Analyzing medicines information of over-the- Research Model and an Empirical Study. J Med Internet Res, counter medicines leaflets in Slovenia. Akademija MM, Vol. 17(6), e143. XIII, 26, 9-22. [4] Pravilnik o oglaševanju zdravil. Uradni list RS, št. 105/08, [13] Liu, C., Kemper, S., in Bovaird, J. A. 2009. Comprehension 98/09 – ZmedPri, 105/10 in 17/14 – ZZdr-2. of Health-related Written Materials by Older Adults. Educ http://www.pisrs.si/Pis.web/pregledPredpisa?id=PRAV8761. Gerontol, 35(7), 653–668. [5] Eysenbach, G., Powell, J., Kuss, O., in Sa, E.-R. 2002. [14] Kasesnik, K., Toroš J., in Kline, M. 2017. Poročilo s Empirical Studies Assessing the Quality of Health preliminarnimi rezultati pilotne raziskave o vedenju Information for Consumers on the World Wide Web: A uporabnikov zdravil brez recepta in pripisovanju Systematic Review. JAMA, 287(20), 2691-2700. pomembnosti atributom. [6] Trg zdravil in ostalih OTC izdelkov v kanalu lekarn v prvi [15] Halme, M., Linden, K. in Kääriä, K. 2009. Patients’ polovici leta 2016. Preferences for Generic and Branded Over-the-Counter https://www.kemofarmacija.si/wps/wcm/connect/SL/Domov/Z Medicines: An Adaptive Conjoint Analysis Approach. a+medije/Aktualno/Landing/Trg+zdravil+in+ostalih+OTC+iz Patient, 2(4), 243-255. delkov+v+kanalu+lekarn+v+prvi+polovici+leta+2016. [16] Fraenkel., L., Bogardus, S. T., Concato, J., Dick, R., in [7] Chaudhuri, S., Le, T., White, C., Thompson, H., in Demiris, Wittink, D. 2004. Treatment Options in Knee Osteoarthritis. The Patient’s Perspective. G. 2013. Examining Health Information–Seeking Behaviors Arch Intern Med, 164 (12), 1299- of Older Adults. Comput Inform Nurs, 31(11), 547-553. 1304. [8] Schutten, M., in McFarland A. 2009. Readability levels of health-based websites: from content to comprehension. Int Electron J Health Educ, 12, 99-107. [9] Flesch, R. 1948. A new readability yardstick. J Appl Psychol, 2, 221-233. 296 STANOVANJSKE RAZMERE STAREJŠIH V SLOVENIJI IN SPREJEMLJIVOST RAZLIČNIH OBLIK BIVANJA HOUSING CONDITIONS OF THE ELDERLY IN SLOVENIA AND THE ACCEPTANCE OF DIFFERENT HOUSING FORMS Boštjan Kerbler Richard Sendi Maša Filipovič Hrast Urbanistični inštitut Republike Urbanistični inštitut Republike Univerza v Ljubljani, Fakulteta za Slovenija/Urban Planning Institute of Slovenija/Urban Planning Institute of družbene vede/University of the Republic of Slovenia the Republic of Slovenia Ljubljana, Faculty of Social Sciences Trnovski pristan 2 Trnovski pristan 2 Kardeljeva ploščad 3 1000 Ljubljana, Slovenija/Slovenia 1000 Ljubljana, Slovenija/Slovenia 1000 Ljubljana, Slovenija/Slovenia Tel: +386 1 4201338; fax: +386 1 Tel: +386 1 4201338; fax: +386 1 masa.filipovic@fdv.uni-lj.si 4201330 4201330 bostjan.kerbler@uirs.si IZVLEČEK 1. UVOD Prebivalstvo Slovenije se pospešeno stara. Zaradi soočanja s Za zahodno družbo je značilno, da se vse bolj stara. Problem povečevanjem finančnih potreb, med drugim tudi za zagotavljanje staranja prebivalstva je, bolj kot drugod po svetu, ustrezne stanovanjske oskrbe in storitev za starejše je problematika izpostavljen v Evropi. Slovenija glede tega ni izjema. Še več, postala vroča politična tema. Z vidika naraščanja deleža starejših v slovenska družba se stara celo hitreje od evropskega družbi in njihove stanovanjske oskrbe je v Sloveniji še posebej povprečja. Po zaskrbljujoče to, da smo do sedaj razvijali predvsem institucionalno projekcijah Statističnega urada Republike obliko stanovanjske oskrbe za starejše, ki je med Slovenije [1] se bo delež starejših od 65 let do leta 2060 vsemi oblikami stanovanjske oskrbe najdražja. V zelo omejenem obsegu so za povečal na 31,6 %, kar bo nad evropskim povprečjem, delež starejše na voljo še najemniška in oskrbovana stanovanja, druge starih nad 80 let pa bo do takrat že presegel potrojitev. Z stanovanjske ponudbe pa skoraj ni. Po drugi strani je za Slovenijo vidika naraščanja deleža starejših v družbi in njihove značilna visoka lastniška zasedenost stanovanj. V članku so stanovanjske oskrbe je v Sloveniji še posebej zaskrbljujoče predstavljene bivalne razmere starejših ljudi v Sloveniji in odnos do to, da je za Slovenijo značilna visoka lastniška zasedenost izbranih oblik bivanja, s katerimi bi lahko v Sloveniji povečali stanovanj in da prevladuje institucionalna oblika raznovrstnost stanovanjske ponudbe za starejše. stanovanjske oskrbe za starejše. Po osamosvojitvi smo Ključne besede izvedli zelo radikalno privatizacijo nekdanjih javnih najemnih stanovanj in, kot pravi Mark Stephens [2], postali staranje prebivalstva, starejši ljudje, stanovanja, bivanje »superlastniki stanovanj«. Po Richardu Sendiju et al. [3] ima bivanje starejših v lastnem domu sicer številne pozitivne ABSTRACT učinke (npr. občutek domačnosti, socialne vezi), prinaša pa tudi številne ovire (npr. stroški vzdrževanja nepremičnine, The population of Slovenia is rapidly aging. Due to the increase in financial needs, including the provision of adequate housing and grajene ovire v bivalnem okolju, dostop do storitev). Po services for the elderly, the problem has become an important drugi strani smo do sedaj za bivanje v starosti gradili political issue. With regard to the provision of housing for the predvsem domove za starejše, vendar pa je ta stanovanjska increasing share of the elderly in Slovenia, it is especially alarming oblika med vsemi oblikami stanovanjske oskrbe najdražja. that Slovenia has so far primarily developed institutional housing Poleg tega zmogljivosti v domovih za starejše zadoščajo le for the elderly, which is the most expensive among all forms of za okoli 5 % prebivalcev Slovenije, starih 65 let in čeprav se housing. To a very limited extent, rental housing and assisted-living kapacitete stalno povečujejo, v prihodnje ne bo mogoče facilities are also available to the elderly, but there are nearly no zadostiti vse večjemu povpraševanju po institucionalni oskrbi other forms of housing. On the other hand, Slovenia has a high level of home owners. The article presents the living conditions of [4]. Poleg omenjenih oblik, drugih oblik stanovanjske oskrbe older people in Slovenia and the attitudes towards selected different za starejše pri nas skoraj ni. V zelo omejenem obsegu so na forms of living, which could increase the diversity of housing for voljo sicer še oskrbovana stanovanja in najemniška older people in Slovenia. stanovanja za starejše. Premalo različnih oblik bivanja za starejše in drugih instrumentov povezanih s tem, odsotnost Key words raznolikosti storitev oskrbe za starejše in vse večje ageing population, elderly, housing, living povpraševanje po institucionalnih oblikah bivanja (zlasti po bivanju v domovih za starejše) so pri nas že začeli sprožati 1 297 vse resnejše probleme. V Evropski uniji že nekaj časa preložili na kasnejši čas. To se sklada s spoznanji relativno prevladuje spoznanje, da problema ne bo mogoče reševati le nizki mobilnosti prebivalstva Slovenije [5], kar potrjujejo z do sedaj uporabljanimi modeli stanovanjske in druge tudi rezultati raziskave, po kateri je 63 % anketirancev oskrbe za starejše, ampak bo treba razviti nove rešitve, ki prebivalo v obstoječem domu več kot 30 let, kar 84 % pa jih bodo učinkovitejše in finančno manj zahtevne. V raziskavi še ni nikdar razmišljalo o selitvi. To je v prvi vrsti posledica smo zato preučili, katere tovrstne rešitve bi bile primerne za močne navezanosti starejših na svoje nepremičnine in Slovenijo oziroma kako sprejemljive bi bile te rešitve za domače bivalno okolje. Kar 91 % jih je izrazilo, da so starejše ljudi v Sloveniji. navezani oziroma zelo navezani na svojo nepremičnino, več kot tri četrtina pa, da so navezani ali močno navezani tudi na 2. RAZISKAVA svojo sosesko oziroma kraj, v katerem bivajo. Največ vprašanih je odgovorilo, da se v svojem domačem okolju V letih 2014–2017 sta Urbanistični inštitut Republike počutijo samostojne in neodvisne in da se v tem okolju Slovenije in Fakulteta za družbene vede izvedla raziskavo z znajdejo (95 %). Ob močni navezanosti in nizki želji po naslovom Inovativne oblike bivalnih okolij za starejše v preselitvi je skrb zbujajoče dejstvo, da je 47 % vprašanih Sloveniji, ki jo je financirala Agencija Republike Slovenija potrdilo, da s sedanjimi prihodki ravno še shajajo 13 % pa, za raziskovalno dejavnost. Cilj je bil opredeliti inovativna da se le težko ali izjemno težko preživljajo. bivalna okolja za starejše, ki bi ustrezala potrebam, željam in navadam starejših ljudi v Sloveniji in bi omogočala, da bi Pri vprašanju: »Starejši ljudje se včasih znajdejo v situaciji, starejši čim dlje časa ostali aktivni člani družbe, hkrati pa bi ko ne bi mogli več sami skrbeti zase niti s pomočjo bila finančno vzdržnejša za starejše ljudi in slovensko sorodnikov; za vsako izmed naštetih možnosti povejte, kako družbo nasploh. Podatke za analizo sta sodelujoči sprejemljiva bi bila za vas?«, so starejši izbirali med organizaciji pridobili z anketiranjem, ki ga je novembra 2015 izvedel Center za raziskovanje javnega mnenja in množičnih nekaterimi bolj poznanimi in manj poznanimi bivanjskimi rešitvami. Po pričakovanjih so bile bolj znane oblike komunikacij. Šlo je za telefonsko anketiranje, t. i. CATI stanovanjske oskrbe bolje sprejete med starejšimi. Najbolj tehnika anketiranja, v katerega je bilo vključeno prebivalstvo sprejemljivo bi bilo za njih bivanje v obstoječem Slovenije, staro 50 let in več, obeh spolov, iz vseh stanovanju/hiši s podporo (sprejemljivo 68 % anketirancem), statističnih regij in tipov naselij (mestnih in podeželskih). Končnih veljavnih anket je bilo 930, ki predstavlja kar potrjuje, spoznanje o navezanosti starejših na svoj dom. Enak delež starejših je izrazil naklonjenost tudi za preselitev reprezentativni vzorec anketirane populacije. Anketa je v dom za starejše, ki je najbolj znana in najbolj razširjena obsegala vprašanja povezana z bivalnimi razmerami starejših in odnosom starejših do morebitne preselitve, njihovimi oblika bivanja v Sloveniji. Sledi oskrbovano stanovanje (49- stališči do različnih bivalnih okolij za starejše s poudarkom odstotna sprejemljivost) in bivanje v obstoječem stanovanju/hiši z obiskovanjem dnevnega centra (46- na oblikah bivanja, ki so v slovenskem okolju manj pogoste odstotna sprejemljivost). Večjo zadržanost so vprašani oziroma še niso prisotne, ter vprašanja povezana z odnos pokazali do drugih predstavljenih rešitev, ki so v našem vprašanih do različnih stanovanjskih rešitev, ki omogočajo pridobitev dodatnih prihodkov iz lastništva nepremičnine. okolju manj poznane ali nepoznane. Tako je bilo bivanje v večgeneracijski stanovanjski stavbi sprejemljivo za 30 % anketirancev, gospodinjska skupnost za 27 % vprašanih, 3. REZULTATI sobivanje starejših v skupnosti je bilo sprejemljivo za 24 % Kot je bilo pričakovano iz poznane lastniške strukture v starejših, medtem ko je bilo bivanje z drugo družino ali Sloveniji, kjer prevladuje lastništvo stanovanja, je bilo med posameznikom oziroma bivanje pri oskrbniški družini vprašanimi 97 % lastnikov, od tega v skoraj treh četrtinah sprejemljivo desetini anketirancev ali manj. Se pa je lastnikov hiš in v eni četrtini pa lastnikov stanovanj. Pri tem izkazalo, da so mlajše generacije starejših bolj odprte za je z vidika dostopnosti in grajenih ovir zaskrbljujoče, da je različne/inovativne oblike bivanja. Drugače je pri različnih kar 84 % tistih, ki živijo v stanovanju, biva v nadstropju ter možnostih in stanovanjskih rešitvah, ki omogočajo da med tistimi, ki živijo v stanovanju, velik delež (70 %) pridobitev dodatnih dohodkov iz lastništva stanovanja biva v zgradbi brez dvigala. Z vidika vzdrževanja oziroma hiše (na primer obratna hipoteka, prodaja z nepremičnin je skrb zbujajoče predvsem to, da po eni strani odloženim prenosom lastninske pravice, prodaja in hkratni 92 % starejših živi v stanovanjih z vsaj dvema sobama povratni najem/rentni odkup, prodaja stanovanja in nakup oziroma jih 80 % biva na več kot 50 kvadratnih metrih, po manjšega stanovanja, oddaja dela nepremičnine najemniku drugi strani pa več kot dve tretjini starejših živi v eno ali itd.). Za večino vprašanih so bile namreč vse predstavljene dvočlanskem gospodinjstvu oziroma štiri desetine je takih, ki možnosti nesprejemljive [6]. Ugotovljeno podpira tudi živijo v gospodinjstvih z manj kot tisoče evrov mesečno. podatek, da so med vsemi vidiki, kaj vprašanim pomeni Vse to pomembno vpliva na kakovost bivanja v starosti. stanovanje, najnižje ocenili mnenje, da je njihovo Zahteva namreč nujen razmislek o ustrezni prilagoditvi stanovanje/hiša finančna naložba. grajenega bivalnega okolja in o zmožnostih starejših za vzdrževanje lastniških nepremičnin. To je še posebej pomembno zato, ker bi radi starejši ljudje čim dlje časa bivali v svojih stanovanji in prehod v drugo obliko bivanja 2 298 4. SKLEP učinkovit in razumljiv način ozaveščati o različnih oblikah bivanja za starejše ter možnostih in stanovanjskih rešitvah, ki Rezultati raziskave so pokazali, da stanovanjske razmere omogočajo pridobitev dodatnih dohodkov iz lastništva starejših niso posebej vzpodbudne, predvsem z vidika stanovanja oziroma hiše. Seznanjati bi jih bilo treba o dostopnosti in morebitnih grajenih ovir ter z vidika potreb po primerih dobrih praks iz domačega, slovenskega okolja – vzdrževanju in finančnih zmožnosti starejših pri tem. Po starejše bi morali nagovarjati drugi starejši, drugi strani starejši izkazujejo močno navezanost na svoja ki imajo glede drugačnih oblik bivanja pozitivne izkušnje. Ozaveščanje bi domača bivalna okolja in zelo nizko stopnjo želje po preselitvi. Z namenom, da se ugodi željam in potrebam moralo potekati na lokalni ravni, med ljudmi, ki se med seboj poznajo. Vsakemu bi moralo biti zato omogočeno, da bi v starejših ljudi, je zato treba razmišljati, kako starejšim domačem okolju od lokalnih informatorjev pridobil vse omogočiti, da bi lahko čim dlje časa ostali v svojih znanih domačih bivalnih okoljih, da bi bilo njihovo bivanje čim bolj ustrezne informacije o možnostih bivanja in prilagajanju stanovanj. To bi po našem mnenju pripomoglo, da bi se s kakovostno ter da bi čim dlje časa ohranili svojo časom spremenila miselnost, ukrepi in pobude od »zgoraj samostojnost, neodvisnost in da ostali aktivni člani svoje navzdol« in od »spodaj navzgor« pa bi dosegli sinergijske skupnosti. To je mogoče doseči s prilagajanjem grajenega učinke in bi pozitivno vplivali eden na bivalnega okolja in storitev. Starejšim ljudem bi moralo biti drugega. omogočeno, da si svoja stanovanja arhitekturno prilagodijo za bivanje v starosti. Podobno kot za povečanje energetske 5. ZAHVALA učinkovitosti stavb (kot posledica blaženja podnebnih Zahvaljujemo se Agenciji Republike Slovenije za sprememb in povečanih potreb po energiji), bi morala raziskovalno dejavnost za financiranje temeljnega Slovenija subvencije namenjati tudi prilagoditvam stanovanj raziskovalnega projekta Inovativne oblike bivanja za starejše za lažje bivanje v starosti (kot posledica na izzive ljudi v Sloveniji, ki je potekal med letoma 2014 in 2017 pod demografskih sprememb v družbi). Menimo, da bi bil vložek šifro J5–6824 in temeljnega raziskovalnega projekta Model za fizične prilagoditve domačega bivalnega okolja zn za staranje starejših v domačem bivalnem okolju v Sloveniji, atno nižji, kot pa če bi se morali starejši zaradi neprimernega ki je potekal med letoma 2017 in 2020 pod šifro J5–8243. V stanovanja (predčasno in pod prisilo) preseliti v projekt sta bila vključena Urbanistični inštitut Republike institucionalno bivanje. Ustrezen državni organ (po našem Slovenije in Fakulteta za družbene vede. mnenju je to ministrstvo za okolje in prostor), bi moral sestaviti seznam podjetij, ki bi takšne prilagoditve lahko 6. REFERENCE opravljala. Ne seznam bi bila uvrščena le tista podjetja, za [1] Statistični urad Republike Slovenije (2017). Prebivalstvo. katere bi se na podlagi ustreznih strokovnih preverjanj Ljubljana. potrdilo, da so verodostojna in primerno usposobljena za [2] Stephens M. (2003). Globalisation and Housing Finance naloge, ki bi jih morala izvajati. Seznam bi moral biti Systems in Advanced and Transition Societies. Urban narejen za vsako slovensko občino posebej, tako da bi lahko Studies, 40(5–6), str. 1011–1026. pri prilagoditvah stanovanj sodelovala lokalna podjetja [3] Sendi, R., Černič Mali, B., Jakoš, A., Filipović, M. oziroma bi starejši ljudje imeli možnost izbirati med podjetji (2003). Stanovanjske potrebe upokojencev in drugih in ljudmi, ki jih poznajo in jim zaupajo. Glede priprave starejših ljudi. Ljubljana, Urbanistični inštitut Republike takšnega seznama je urbanistični inštitut, ki ima za to Slovenije. potrebno znanje, seznanil direktorat na ministrstvu za okolje [4] Kolmančič, K. (2010) Problematika dolgotrajne oskrbe v in prostor. Inštitut bi tudi pripravil ustrezne predloge za Republiki Sloveniji s poudarkom na domovih za starejše spremembo zakonodaje in pravilnikov ter priprave občane. Ljubljana. strokovnega priročnika za prilagoditve stanovanj in [5] Mandič, S. (2015). Stanovanjske razmere in mobilnost stanovanjskih objektov za potrebe varnega in samostojnega starejšega prebivalstva – Slovenija v primerjalni bivanja starejših v domačem bivalnem okolju. Vzporedno z perspektivi. In: Majcen, B. (ur.) Značilnosti starejšega zagotavljanjem možnosti za bivanje v domačem bivalnem prebivalstva v Sloveniji: prvi rezultati raziskave SHARE, okolju, bi bilo treba omogočati vzpostavitev različnih oblik str. 183–195. Ljubljana: Inštitut za ekonomska bivanja za starejše ljudi, pri tem pa bi bilo treba izhajati iz raziskovanja. njihovih potreb in želja. Raznovrstnost oblik stanovanjske [6] Kerbler, B., Filipovič Hrast, M., Mandič, S., Sendi, R., oskrbe za starejše je bila na predlog urbanističnega inštituta Černič Mali, B., Mrzel, M. (2017). Inovativne oblike in ministrstva za delo družino in socialne zadeve ter enake bivalnih okolij za starejše ljudi v Sloveniji. Ljubljana, možnosti vključena tudi v strategijo dolgožive družbe. Univerza v Ljubljani, Arhiv družboslovnih podatkov. Strategija kot naslednji korak predvideva pripravo ustreznih akcijskih načrtov, pri katerih bi urbanistični inštitut sodeloval kot strokovna podpora. Menimo še, da bi bilo treba za razširitev raznovrstnosti stanovanjske oskrbe starejših nujno spremeniti ali vsaj dopolniti stanovanjski zakon in v njem izbrane oblike bivanja ustrezno opredeliti. Nujno pa je treba starejše in tudi mlajše ljudi začeti na 3 299 Družina kot sistem in pomen družinske terapije Urška Kranjc Jakša CTIS – Center za terapijo, izobraževanje in svetovanje Livarska ulica 9, 1000 Ljubljana Elektronska pošta: ukranjcjaksa@gmail.com POVZETEK močnejši od njegovih posameznih delov ali podsistemov« [5]. Vsak posameznik se v družini razvije, oblikuje in odide v svet z Družina je tako vedno več kot le vsota družinskih članov, saj je doto, ki jo prejme od svojih najbližjih. Čeprav so ti odnosi ključni vsaka družina organska celota, živ organizem, ki je tudi notranje, za naše poznejše interakcije z okoljem, so samo ogrodje, ki ga sistematično organiziran [5]. lahko dograjujemo. Sistemska teorija razume družino kot sistem Tako kot so elementi v sistemu povezani, saj vsak del v sistemu odnosov, katerega člani vzajemno vplivajo drug na drugega, vpliva na celoten sistem [4], tako so med seboj povezani tudi družina pa kot celota na vsakega posameznika. Prvi odnos s starši družinski člani. Vsaka sprememba, ki se zgodi enemu članu ima ključno vlogo za razvoj možganov in zdravo psihično družine, vpliva tako na celoto kot na vse druge družinske člane. strukturo. Nefunkcionalni vzorci in razne travmatične izkušnje Razni dogodki, ki se dogodijo v družini, npr. bolezen, smrt, posameznikov se lahko uspešno predelujejo z družinsko relacijsko poroka, rojstvo, izguba službe ipd., vedno vplivajo na celotno terapijo, ki tako pripomore tudi k urejanju odnosov na ravni družino. Če družina želi obstati, mora te spremembe sprejeti, se osnovne družbene celice. prilagoditi, preoblikovati oz. se drugače organizirati. Ključne besede Če želimo zares razumeti družino, moramo vedno opazovati odnose med vsemi člani. Posameznega vedenja nekega člana Družina, odnosi, razvoj možganov, relacijska družinska terapija družine ne moremo popolnoma razumeti, če ne razumemo delovanja celotnega družinskega sistema. Na tem temelji tudi 1. SISTEMSKO RAZUMEVANJE družinska terapija, pri kateri vedno opazujemo in razumemo DRUŽINE individualno simptomatiko v kontekstu zakonitosti delovanja cele družine [12]. Vsi prihajamo iz družine. Vsakemu izmed nas se, ko pomisli na svojo družino, prebudijo različna občutja, od prijetnih občutij Družinski sistem deluje preko manjših delov, t. i. podsistemov, ki veselja, zadovoljstva in sreče pa vse do težkih občutij tesnobe, so hierarhično urejeni in razmejeni [5]. Hierarhija je pomembna zavrženosti, nesposobnosti, nehotenosti in zapuščenosti. Ko za funkcionalno delovanje družinskega sistema [12], tako kot je govorimo o družini, ne govorimo samo o posameznikih, ki jo hierarhija pomembna za funkcionalno delovanje različnih sestavljajo. Družina je namreč veliko več. Je najpomembnejši skupnosti, mest, okrajev, države itd. [4]. Hierarhija in sistemska prostor človeškega razvoja. Tu gre najprej za odnos med staršema, razmejitev temeljita na jasnih pravilih in vlogah, kar omogoča za odnos med starši in otroci, nato za odnos med vrstniki, za razvoj in delovanje družinskega sistema v obliki čustvene odnos do šole in družbe [10]. Torej lahko rečemo, da je družina stabilnosti in zmožnosti predvidevanja vedenja družinskih članov sistem odnosov, ki zaznamujejo delovanje tako posameznika kot [12]. skupnosti [12] oziroma družina je sistem odnosov, v katerem njeni člani vzajemno vplivajo na drugega in v katerem družina kot 2. ODNOSI V DRUŽINI celota vpliva na vsakega družinskega člana. To je sistemski pogled na družino, ki se je uveljavil po razmahu splošne sistemske Lahko se vprašamo, zakaj včasih nikakor ne more priti do teorije in njenem prenosu na področje psihologije. Splošna spremembe, čeprav bi bila ta dobra. Kaj je tisto, kar nekaterim ljudem preprečuje, da bi se rešili nefunkcionalnih vzorcev sistemska teorija, ki je temeljila v biologiji, matematiki, tehniki in oziroma da se ne bi vedno znova zapletali v odnose, ki niso fiziki, je izhajala iz ugotovitev, da različni pojavi v svetu nosijo v sebi značilnosti sistema, ki je organiziran v celoto in sestavljen iz primerni? Odgovor lahko najdemo v razumevanju oblikovanja navezanosti v odnosu in pripadnosti družinskim odnosom [12]. med seboj odvisnih delov. Komunikacija in samoregulacija sta bistvena za delovanje sistema, ta pa procesira informacije, se na Ljudje potrebujemo odnose, ker je pripadnost vrojena potreba spremenjene pogoje prilagodi, kar vpliva na nadaljnje vedenje in s vsakega posameznika [12]. Pripadnost pa je tudi temelj tem na njegovo samovzdrževanje [4]. Ugotovitve, ki jih je družinskega sistema [10]. Prvi odnos, ki ga ne moremo izbrati, prinesla splošna sistemska teorija, so prevzele različne veje prav tako ga tudi nikoli ne moremo zapustiti, je odnos s starši znanosti, tudi psihiatrija in preko nje družinska terapija. S tem je [12], zato je tudi tako pomembno, kako varen, zaupen in iskren je prišlo do spremembe paradigme v razumevanju družine, ki je bil ta odnos. To pa ne pomeni, da se naš razvoj konča z odnosi, ki prenesla opazovanje in razumevanje iz individualno orientirane jih oblikujemo s starši, saj se razvijamo celo življenje [12] paradigme k raziskovanju družine kot sistema elementov, ki stalno oziroma se razvijamo v vseh odnosih, v katere vstopamo v svojem sodelujejo in medsebojno vplivajo [15]. življenju. Vendar je ravno odnos s starši tisti, ki nas najbolj Najpomembnejša predpostavka sist zaznamuje in vpliva, na kakšen način bomo vstopali v vse emskega pristopa je sistemska celovitost, kar pomeni, da je »sistem kot celota vedno večji in prihodnje odnose, v njih vztrajali ali jih zapuščali. V odnosih se namreč rodijo samozavedanje, vrednote, spomin ali stališča, prav 300 tako se v odnosi rodijo tudi dvomi in strah [12]. Odnos, ki ni minuto proizvedejo 250 000 novih celic, do rojstva pa ustvarijo varen, morda celo zaznamovan z zlorabami ali travmatičen, milijardo nevronov. Pri štirih letih so otrokovi možgani že 95 % odnos, v katerem ne dobimo občutka pripadnosti, je za otroka ena strukturno razviti [2]. V prvih mesecih življenja se razvije tudi najhujših travmatičnih izkušenj, ki pa ne vpliva samo na tega navezanost, medtem ko se sposobnost regulacije afektov, ki otroka in na njegovo nadaljnje življenje, ampak se prenaša celo v temelji na navezanosti, najbolj razvije med prvim in drugim letom naslednje generacije. Sistem namreč toliko časa ponavlja iste starosti [1]. Otroški možgani so veliko bolj dojemljivi in vzorce odnosov, dokler neuresničena potreba po pripadnosti ne bo prilagodljivi kot možgani odraslega človeka, so pa tudi zelo zadovoljena [10]. senzitivni in zelo ranljivi, predvsem kar se tiče zlorabe in zanemarjenosti, saj lahko pustita na otrokovih možganih John Bowlby je bil prvi, ki je uporabil pojem navezanost. Svoja nepopravljive vtise in lahko celo povzročita, da se v določenih raziskovanja je začel z opazovanjem živalskega sveta in ugotovil, segmentih sploh ne razvijejo oziroma zaostanejo v razvoju. da ima vsaka žival sposobnost, da prepozna svojo mater in se Travmatične oz. kronične izkušnje, ki jih dojenček doživi v času zateče k njej [10]. Bowlby je šel še korak dlje in svoja odkritja hitrega razvoja možganov, tako lahko pustijo trajne, nepopravljive apliciral na človeški svet. Ob opazovanju dojenčkov in otrok v vtise na strukturi in mehanizmu delovanja možganov [2,11]. sirotišnicah in bolnišnicah je ugotovil, da so dojenčki po rojstvu tako nebogljeni, da lahko preživijo samo pod pogojem, če je Pri zgodnjem razvoju možganov po rojstvu je bolj vključena prisotna odrasla oseba, ki jim nudi varnost in skrb [8]. desna hemisfera možganov, kjer pa se nahaja tudi center Bowlbyjeve ugotovitve, da je za normalni razvoj otroka nujna možganov za odzivanje na stres. Zato lahko travmatične izkušnje prisotnost odrasle osebe, na katero se lahko naveže, so trčile ob do prizadenejo ravno te strukture možganov, ki so odgovorne za takrat trdno zasidrano Freudovo teorijo o nagonskih temeljih regulativne mehanizme in za odgovor možganov na stres [10]. To otrokovega razvoja [7]. Trdil je celo, da je mati tista, ki mora biti lahko pomembno vpliva tudi na regulacijo stresa in posledično na fizično prisotna, da bo omogočen zdrav otrokov razvoj. Kljub kakovost življenja v kasnejšem obdobju. Možgani se oblikujejo na temu da danes Bowlbyjevih ugotovitev glede škodljivosti podlagi tega, koliko so bili uporabljeni, koliko novih izkušenj materine deprivacije ne moremo jemati kot izključen razlog za dobijo in koliko se te izkušnje ponavljajo. Utrdijo se tiste nezdrav razvoj otroka, saj je poleg njene prisotnosti za razvoj nevronske poti in povezave, ki jih največkrat uporabimo. Če pomembna tudi čustvena kvaliteta odnosov v družini [7], pa je izkušenj ni oziroma če so izkušnje neprimerne, se možganski dejstvo, da je premik paradigme ugodja kot motivacijske sile na centri ne povezujejo in začnejo propadati [2]. Pri otroku, ki je pomembnost odnosov ustvaril novo polje za raziskovanje večkrat podvržen stresu, se bo utrdil tisti odgovor na stres, ki ga psihološke sfere človeške duševnosti. bo otrok največkrat uporabil. Veliko otrok je soočenih s kroničnim stresom zaradi nasilja v družini, revščine ter fizične ali Pri navezanosti gre za vrojen sistem v možganih, ki otroka duševne bolezni staršev. Dolgotrajna izpostavljenost stresu v tem motivira, da poišče bližino staršev ali drugih primarnih skrbnikov zgodnjem obdobju razvoja možganov lahko posamezniku ter z njimi vzpostavi komunikacijo. S takšnim vedenjem si otrok povzroči trajne spremembe, te pa lahko vodijo v pogosta fizična poveča zmožnosti za preživetje [14]. Vendar ko govorimo o obolenja [10]. odnosih v družini in o razvoju posameznika, zmožnost preživetja ni dovolj. Raziskave so pokazale, da je oblika navezanosti, ki jo 4. DELO Z DRUŽINAMI – RELACIJSKA otrok vzpostavi zgodaj v otroštvu, povezana s kasnejšimi procesi regulacije emocij, razvojem samozavedanja in razumevanja sebe DRUŽINSKA TERAPIJA [10], v osnovi pa gre pri navezanosti za njegovo zaščito pred Pomoč otrokom se začne s pomočjo njihovim staršem. Odnosi so nevarnostjo. Zato navezanost delimo na varno in nevarno namreč vedno vzajemni, kar pomeni, da starši ne morejo navezanost. Otrok se lahko varno naveže, ko se mati nanj uglasi, zagotoviti razvoja otroka, če se ob njem tudi sami ne razvijajo kar pomeni, da vedno znova prepoznava njegove potrebe in nanje [12]. Včasih starši ne zmorejo sami in potrebujejo pomoč. ustrezno odgovarja. Občutek varnosti in zaščite je za otroka Družinska terapija je primeren način, s katerim lahko pomagamo najpomembnejši dejavnik za njegov zdrav čustveni in socialni družinam v stiski, saj poskuša med seboj povezati odnose, vedenje razvoj [7]. Prav tako pa varna navezanost omogoča možnost in dogodke. Družino razume kot dinamičen sistem, ki se razvija v raziskovanja oziroma celo velja, da bolj kot je večji občutek okolju in času ter ima svoj življenjski cikel, v katerem se dogajajo varnosti, širši je krog raziskovanja [1]. Otrok namreč ve, da mu različne prilagoditve glede na pričakovane in nepričakovane skrbniki ob stiski nudijo varno zatočišče, hkrati pa tudi varno stresorje [13]. Družinska terapija se s simptomatiko posameznika izhodišče, od koder lahko raziskuje okolico [7]. vedno sooča v kontekstu celotne družine. Pomembno je, da se z družinami, zakonci in posamezniki dela že ob prvih znakih nemoči, stisk in težav, s katerimi se sami ne zmorejo soočiti. 3. RAZVOJ MOŽGANOV Predvsem pa je nujno in smotrno, da začnemo z njimi delati že Najnovejše raziskave na področju nevropsihologije kažejo, da je pred pojavom raznih psihopatologij oziroma bolezenskih stanj. za razvoj možganov ključnega pomena odnos s starši. Možgani se Raziskave namreč kažejo, da so travmatične izkušnje v otroštvu lahko namreč v polnosti razvijajo takrat, ko je aktivno prisoten pomemben dejavnik za razvoj raznih psihiatričnih bolezni v nekdo tretji [3]. Razvoj možganov je od samega začetka zelo kasnejšem življenju, od posttravmatske stresne motnje, depresivne odvisen od odnosov, ki jih ima otrok s starši, predvsem z materjo. motnje in anksionzne motnje [1]. Dotiki nežnosti, ljubkovanje, dojenje, način, kako se mu mama približa, skratka občutja, ki jih otrok dobi od svoje mame v Na družinske odnose pa ne vplivajo samo sedanje potrebe in intimnih trenutkih, so tisti prvi impulzi, ki se zapišejo v njegov pričakovanja družinskih članov, ampak tudi preteklost organski, implicitni spomin in ki postanejo osnova za nadaljnje družinskega sistema ter pričakovanje prihodnosti [12]. Včasih je navezovanje stikov z drugimi [6]. ravno zaradi travmatične preteklosti, ki jo je doživel nek posameznik, težko delovati v sedanjosti in takrat je primeren čas Ko se rodimo, so možgani najbolj nediferenciran organ v telesu za vstop v terapevtski odnos. Ta nam pomaga spoznati, da [1]. Možgani se zelo hitro razvijajo in že v prenatalni dobi vsako 301 preteklosti ne moremo spremeniti, prav tako ne moremo vplivati [3] Gostečnik, C. 2010. Neprodirna skrivnost intime. Ljubljana: na prihodnost, lahko pa v sedanjih odnosih vplivamo na odzive in Brat Frančišek in Frančiškanski družinski inštitut. doživljanje [12]. S sprejetjem in osmislitvijo preteklosti lahko [4] Gotečnik, C. 2010. Sistemske teorije in praksa. Ljubljana: spremenimo svoje odzive v sedanjih odnosih. Brat Frančišek in Frančiškanski družinski inštitut. Družinska terapija sega na različna področja delovanja in se [5] Gostečnik, C. 2011. Inovativna relacijska družinska terapija. dokazano učinkovito sooča s partnerskimi problemi, konflikti v Ljubljana: Brat Frančišek in Frančiškanski družinski inštitut. družini, odvisnostmi, zlorabami, nasiljem, problemi v šoli, osebnostno rastjo, žalovanjem itd. Relacijska družinska terapija [6] Gostečnik, C. 2012. Govorica telesa v psihoanalizi. Ljubljana: Brat Frančišek in Frančiškanski družinski inštitut. [5], je terapevtski model, ki je znanstveno preverjen v mnogih raziskavah in dokazano učinkovit v klinični praksi [9] in ima [7] Erzar, T. in Kompan Erzar, K. 2011. Teorija navezanosti. danes pomemben položaj med različnimi vrstami terapije v Celje: Celjska Mohorjeva družba. slovenskem prostoru. [8] Hazen, C. in Shaver P. R. 1994. Attachment as an Organizational Framework for Reaserch on Close 5. ZAKLJUČEK Relationships. Psychological Inquiry 5, št. 1: 1–22. DOI= Pomen zdravega temelja v družini je nedvomno med https://doi.org/10.1207/s15327965pli0501_1 najpomembnejšimi pogoji za normalen razvoj človeške psihične [9] Jerebic D. in Jerebic S. 2017. Bližina osrečuje. Celje: Celjska strukture. Razni nefunkcionalni in razboleli odnosi v družini Mohorjeva družba. imajo lahko pogubne vplive na vse posameznike te družine, prav [10] Kompan Erzar, K. 2006. Ljubezen umije spomin. Ljubljana. tako pa tudi na ljudi, s katerimi ti potem vzpostavljajo odnose. Terapija je način, da se posamezniki kar se da temeljito soočijo s [11] Maroney, D. I. 2003. Recognizing the Potential Effect of svojo travmatično izkušnjo in vzorci, ki jim onemogočajo uspešno Stress and Trauma on Premature Infants in the NICU: How delovanje. Pomaga jim lahko, da vzpostavijo zdrav in urejen Are Outcomes Affected? Journal of Perinatology : Official odnos s svojimi bližnjimi in okolico ter ustvarijo pogoje za Journal of the California Perinatal Association 23, št. 8: 679– uspešno in polno življenje. Terapevtska obravnava namreč nujno 83. DOI=10.1038/sj.jp.7211010 pomeni tudi soočenje z družinsko izkušnjo, saj je ta temelj [12] Poljak Lukek, S. 2017. Ko odnosi postanejo družina: posameznikovega delovanja, katerega težave pogosto izvirajo iz oblikovanje edinstvenosti in povezanosti skozi razvojna izkušenj v ožjem družinskem okolju. V tem pogledu družinska obdobja družine. Celje: Celjska Mohorjeva družba. terapija pomembno prispeva k urejanju odnosov na ravni posameznika, pa tudi družine kot osnovne družbene celice s [13] Rus Makovec, 2006. Dva konteksta izraza "družinska pozitivnimi posledicami tako na okolje kot na prihodnje družinske terapija": specifična paradigma in psihoterapevtska generacije. Zdravo družinsko okolje pa gotovo vpliva tudi na modaliteta. Anthropos 38, št. 1/2: 161–172. blaginjo celotne družbe. [14] Siegel, D. 2012. The developing mind: how relationships and the brain interact to shape who we are. 2. izd. New 6. LITERATURA IN VIRI York: Guilford Press. [1] Cvetek, R. 2010. Bolečina preteklosti. Celje: Celjska [15] Žabkar, Blaž. 2017. Vloga očeta pri vzgoji, problematično Mohorjeva družba. vedenje otrok in relacijska družinska terapija. Doktorska disertacija. Teološka fakulteta Univerze v Ljubljani. [2] Gostečnik, C. 2008. Relacijska paradigma in travma. Ljubljana: Brat Frančišek in Frančiškanski družinski inštitut. 302 Dolgotrajna oskrba – zahtevna demografska naloga in izjemna priložnost za krepitev sodobne solidarnosti Jože Ramovš Ksenija Ramovš Maja Rant Inštitut Antona Trstenjaka za Inštitut Antona Trstenjaka za Inštitut Antona Trstenjaka za gerontologijo in medgeneracijsko gerontologijo in medgeneracijsko gerontologijo in medgeneracijsko sožitje sožitje sožitje (00386)1 433 93 01 (00386)1 433 93 01 (00386)1 433 93 01 joze.ramovs@guest.arnes.si ksenija.ramovs@guest.arnes.si maja.rant@gmail.com Marta Ramovš Veronika Mravljak Andoljšek Ana Ramovš Inštitut Antona Trstenjaka za Inštitut Antona Trstenjaka za Inštitut Antona Trstenjaka za gerontologijo in medgeneracijsko gerontologijo in medgeneracijsko gerontologijo in medgeneracijsko sožitje sožitje sožitje (00386)1 433 93 01 (00386)1 433 93 01 (00386)1 433 93 01 marta.ramovs@inst- veronika.mravljak@gmail.com ana.ramovs@inst- antonatrstenjaka.si antonatrstenjaka.si Ana Gorenc Vujović Ajda Svetelšek Inštitut Antona Trstenjaka za Inštitut Antona Trstenjaka za gerontologijo in medgeneracijsko gerontologijo in medgeneracijsko sožitje sožitje (00386)1 433 93 01 (00386)1 433 93 01 ana.gorenc.vujovic.iat@gmail. c ajda.svetelsek@inst- om antonatrstenjaka.si POVZETEK implementation of healthy ageing and intregenerational solidarity Članek izhaja iz podatkov o prihajajoči demografski krizi staranja programs. It stands on the anthropological knowledge and prebivalcev ter iz dejstva, da je razvoj sistema humane in finančno explains how fast advancements are needed, due to shortfalls that vzdržne dolgotrajne oskrbe starostno onemoglih, kronično bolnih were made on the field of interpersonal relations in the past, when in invalidnih ljudi skupaj z uvajanjem sodobnih programov za economic, medicine and life-ecpectancy development was a zdravo staranje in za krepitev medgeneracijske solidarnosti priority. At the present time there is enough knowledge on this osnovna naloga za reševanje te krize. Na osnovi antropoloških topic to adress the issue and at the same time this societal need is spoznanj trdi, da je nagel razvoj na teh treh področij nujen zaradi the main anthropological energy resource for dealing with the velikega zaostanka na področju kakovosti medčloveškega sožitja tasks, that the demographic crysis places on us. The first part of za materialnim razvojem in za razvojem na področju zdravja in the article summarizes today's knowledge on long-term care. In dolžine življenja. Zaradi novih znanj je pospešen razvoj the second part the 12 thesis on the development of national long- medgeneracijskega sožitja danes možen, obenem pa je ta razvojna term care system are stated. Slovenian government was potreba glavna notranja antropološka energija za reševanje nalog unsuccesseful in acomplishing the formation and implementation demografske krize. V prvem delu članka so povzeta današnja of long-term care legislations twice (2010 and 2017). For the slovenska in svetovna spoznanja o dolgotrajni oskrbi, v drugem je purpose of compensating for the lost time, the gerontological navedenih 12 tez o oblikovanju sodobnega nacionalnega sistema thesis are suggested to help the politics and administration on za dolgotrajno oskrbo; po dveh neuspešnih poskusih slovenske preparation and obtainment of quality long-term care system in vlade (2010 in 2017), da bi s sprejetjem Zakona o dolgotrajni Slovenia. oskrbi začela dohitevati četrt stoletni zaostanek za evropskimi sosedi na tem področju, so te teze pomoč gerontološke stroke Keywords demographic crisis, long-term care, intergenerational solidarity politiki in upravi, da pripravi in sprejeme kakovosten sodoben sistem dolgotrajne oskrbe tudi v Sloveniji. 1. UVOD Ključne besede Zagotoviti humano in finančno vzdržno dolgotrajno oskrbo demografska kriza, dolgotrajna oskrba, medgeneracijska starostno onemoglim, kronično bolnim in invalidnim ljudem je solidarnost ena od osnovnih nalog za rešitev demografske krize ob staranju prebivalstva v 21. stoletju; enako pomembni in s prvo neločljivo ABSTRACT povezani sta: poskrbeti učinkovite programe za zdravo staranje ter The article is based on the data about forthcomming demographic za krepitev solidarnosti med generacijami (Svet EU, 2005). Te tri crysis. The fact is, that the primary focus for overcoming this naloge so med seboj neločljiva komplementarna celota. Sedanja crysis should be on the development of humane and financially demografska kriza ni samo zahtevna naloga, ampak tudi ugodna sustainable system for long-term care and also on the priložnost za hiter razvoj evropske kulture na področju kakovosti 303 sožitja in sodelovanja v družini, službi in družbi; ta razvoj sedaj  družinskih in drugih neformalnih oskrbovalcev z dobro zaostaja vsaj za eno stoletje za materialnim in tehnološkim usposobljenimi in sodobno organiziraziranimi razvojem ter za razvojem telesnega zdravja in podaljševanja strokovnimi oskrbovalci in oskrbovalnimi ustanovami, življenja. Razmere za pospešen razvoj kakovosti sožitja so danes ugodnejše kakor kdajkoli doslej, za ta razvoj imamo danes na  javnih virov v krajevni skupnosti in državi ter voljo toliko znanj kakor nikdar doslej v zgodovini (Ramovš,  lastnih virov in zmožnosti človeka v dolgotrajni oskrbi ured., 2013; 2017). Nadaljnji razvoj družbe in naše evropske in njegovih najbližnjih. kulture kot celote – Slovenci in vsak od evropskih narodov pa tudi za razvoj svoje nacionalne kulture – je verjetno motiv z največ Tako sestavljena in na oskrbovančevo osebo usmerjena notranje antropološke energije, ki ga imamo na voljo za uspešno dolgotrajna oskrba je tudi najučinkoviteljši način za preprečevanje rešitev demografske krize. Razvoj humane in finančno vzdržne pogostega nehotenega verbalnega in fizičnega nasilja ter dolgotrajne oskrbe je odločilen za nadaljnji razvoj humanosti zanemarjanja v domačem okolju in oskrbovalnih ustanovah. evropske in slovenske družbe. Danes se hitro razvija IKT za svetovanje oskrbovalcem, za fizično pomoč pri dolgotrajni oskrbi, za spodbujanje in pomoč pri 2. SPOZNANJA O DOLGOTRAJNI ohranjanju, krepitvi in rehabilitaciji oskrbovančevih zmožnosti ter OSKRBI za varovalno in opozorilno kontrolo njegove varnosti in oskrbe. Dolgotrajna oskrba je pomoč sorazmerno bolj zdravih ljudi tistim, ki ne morejo sami opravljati ene ali več od sedmih osnovnih Enako pomembna družbena naloga, kakor vzpostavljanje in vodenje humane in finanačno vzdržne dolgotrajne oskrbe, je vsakodnevnih opravil, ki so: vstajanje, osebna higiena, oblačenje, gibanje, hranjenje in jemanje zdravil, izločanje in dnevno organizirana javna pomoč za zdravo staranje preostalim 75 % navezovanje človeškega stika, ter pri opravljanju instrumentalnih upokojenske generacije, ki ne potrebujejo dolgotrajne oskrbe. Za opravil za življenje, ki so: kuhanje, čiščenje stanovanja, pranje in ohranjanje njihove samostojnosti se danes naglo razvijajo zlasti: likanje perila, nakupovanje, denarni posli, prevoz in druga za  ozaveščanje in javne spodbude za prilagajanje lastnega človeka pomembna opravila. Odvisnost pri opravljanju osnovnih stanovanja za starostno pešanje, in funkcionalnih opravil je merilo za vstop v sodobni sistem socialne varnosti za dolgotrajno oskrbo, ki ga evropske države  organizirani programi za krepitev telesnega, duševnega uvajajo po letu 1990, Slovenijo pa s četrt stoletnim zaostankom ta in socialnega zdravja z zdravo prehrano, gibanjem, naloga še čaka; leta 2010 in 2017 sta bila predloga zakona o spanjem, osvajanjem veščin za preprečevanje padcev v dolgotrajni oskrbi v javni obravnavi (Republika Slovenija, 2010; starosti, življenje s povišanim krvnim tlakom, Vlada RS, 2017). Oba predloga sta imela nekaj dobrih strani in sladkorjem v krvi, razvijajočo se demenco in drugimi nekaj usodnih pomanjkljivosti – temeljiti analizi zadnjega kroničnimi nenalezljivimi boleznimi, predloga je bila posvečena celotna zadnja številke slovenske  medgeneracijsko povezovanje in kakovostno osebno znanstveno-strokovne revije Kakovostna starost letnika 2017 povezovanje starejših za preprečevanje osamljenosti, (Kakovostna starost, 2017), strokovno poglobljena stališča je revija zavzela tudi ob predlogu leta 2010 (Inštitut, 2010), ves čas  usposabljanje za lepše komuniciranje in sožitje v pa objavlja svetovna in domača spoznanja ter dobre izkušnje o družini in z drugimi, za prenašanje svojih življenjskih dolgotrajni oskrbi. Iz teh virov ter iz preglednih del o dolgotrajni izkušenj in spoznanj na starejših na mlajše in obratno oskrbi v Evropi (Leichsering in sod., 2013) navajamo nekaj ter za iskanje smisla starosti. osnovnih spoznanj o dolgotrajni oskrbi. Po reprezentativni raziskavi (Ramovš, 2013, ured.) opravlja v Sodobni sistem dolgotrajne oskrbe odgovarja na navedeno Sloveniji dolgotrajno oskrbo v domačem okolju nad 220.000 odvisnosti od pomoči pri vsakdanjih opravilih starostno neformalnih oskrbovalcev, ki nekaj ur tedensko ali celodnevno onemoglih, kronično bolnih in invalidnih ljudi. Danes v Evropi brezplačno oskrbujejo starega, kronično bolnega ali invalidnega potrebuje dolgotrajno oskrbo 25 % ljudi, ki so stari nad 65 let, v človeka. Med njimi je velika večina – 200.000 družinskih Sloveniji je to okrog 80 tisoč ljudi. Četrtino od njih oskrbujejo oskrbovalcev, to je sorodnikov, in dobrih 20.000 drugih, največ ustanove (domovi za stare ljudi), tri četrtine pa domači in drugi sosedov, nekaj prostovoljcev in prijateljev. Po podatkih uradnega neformalni oskrbovalci na njihovem domu. V prihodnjih letih se evropskega združenja za neformalne oskrbovalce Eurocarers, ki bo delež ljudi, ki so stari nad 85 let trikratno povečal. Ker v tej ne upoštevajo, koliko časa in kako pogosto oskrbujejo, pa je starosti večina potrebuje pomoč pri enem ali več vsakdanjih neformalnih oskrbovalcev kar 20 % evropske populacije opravilih, se bo tudi delež prebivalstva v dolgotrajni oskrbi vsaj (Yghemonos, 2018), v Sloveniji torej 400.000. Družinski podvojil. oskrbovalci oskrbujejo v Sloveniji 60.000 starejših ljudi doma, 6.000 oskrovancem pomagajo po nekaj ur tedensko poklicni Programi, metode in sistem dolgotrajne oskrbe so danes med oskrbovalci v organizaciji javne pomoči na domu. Domovi za najhitreje razvijajočimi se področji. Usmeritve pri tem so: stare ljudi oskrbujejo 20.000 starih ljudi, v njih je blizu 10.000 1. oskrbovanje v domačem okolju, zaposlenih. Za kakovost življenja v oskrbovalni ustanovi je odločilno tudi sodelovanje domačih in prostovoljcev. 2. usposabljanje in razbremenilna pomoč družinskim in drugim neformalnim oskrbovalcem, Raziskave in vsakdanje izkušnje kažejo, da je pomoč svojcem tako za oskrbovalce kakor za oskrbovance lahko zelo dobra 3. komplementarno povezovanje v celoto vseh virov izkušnja. Naša reprezentativna raziskava o potrebah, zmožnostih oskrbe: in stališčih nad 50 let starih prebivalcev Slovenije (Ramovš, ured., 2013) je pokazala, da ostaja svojcem v lepem spominu na oskrbovanje predvsem izkazana hvaležnost oskrbovanca, etično 304 zadoščenje ob opravljeni dolžnosti, lastni občutki zadovoljstva, materialno-tehničnemu razvoju in s tem telesnemu zdravju in veselja, topline, sreče in koristnosti, poglobitev odnosa s svojcem podaljševanju življenja, pri tem pa ji je povsem zaostal razvoj in/ali med družinskimi člani. Oskrbovanci pa navajajo zlasti medčloveškega sožitja in celovite človekove osebnosti. Socialni naslednje lepe spomine na čas oskrbovanja: na svojca, ki je zanje kapital desettisočletne tradicije z njenimi vzorci solidarnosti za skrbel, na njegovo prijaznost, potrpežljivost, požrtvovalnost ipd., razvoj in preživetje v poljedeljsko-obrtniško-stanovskih družbenih pa tudi na dobro oskrbo, na dejstvo, da so jim svojci pomagali in razmerah, se v tem času izčrpava, zato EU v Zeleni knjigi o odzivu na poglobitev odnosa. na demografske spremembe (2005) govori o nujnosti razvijanja »nove solidarnosti med generacijami«. Podvojena potreba po Isti raziskovalni podatki kažejo tudi pereče potrebe oskrbovalcev oskrbi starih ljudi je za to vsaj tako dobra priložnost, kakor je bila pri oskrbovanju. Svojci navajajo kot največje težave zlasti v tradicionalni družbi skrb za otroke, katerih se je tedaj rodilo naslednje: pomanjkanje veščin in znanja o negi, o boleznih, o nekajkrat več na žensko kakor danes. Učenje aktivnega komunikaciji z bolnim človekom, pomanjkanje časa za oddih, človekovanja in občestvovanja se dogaja v pretežni meri ob spopadanje z lastnimi zdravstvenimi težavami in duševnimi človeški vzgoji otrok in osebni pomoči nemočnim v njihovi stiskami (npr. soočanje z nemočjo, strah pred poslabšanjem). bolezni, onemoglosti, invalidnost ali materialni revščini; edini Mnogi se v vlogi družinskega oskrbovalca znajdejo čez noč, način učenja človekovanja in občestvovanja je pristno človeško nekateri za svojce skrbijo praktično 24 ur dnevno. V Sloveniji, pomagati ljudem v njihovih težavah in od njih hvaležno prejemati kjer še nimamo sodobnega sistema za dolgotrajno oskrbo, so pomoč. Samo razvita osebna solidarnost pri večini ljudi (empatija, družinski oskrbovalci prepuščeni sami sebi. Brez podpore komuniciranje, sprejemanje sebe in ljudi z našimi stvarnimi politike, strok in širše javnosti so v današnjih življenjskih razvojnimi možnostmi in mejami) omogoča ob naglem razmerah težko kos tej zahtevni nalogi. materialnem in tehnološkem razvoju tudi razvoj humane človeške Dolgotrajno oskrbovanje – in vsako domače, prostovoljsko ali družbe, posameznikom pa doživljanje srečnosti in smisla poklicno delo za ljudi v potrebi in stiski – ima dve razsežnosti: celotnega življenja, sožitja in razvoja. 1. je storitev ali opravilo, ki drugemu stvarno zadovolji Dolgotrajna oskrba je torej danes največja priložnost za razvijanje potrebo, ki je sam ne more, sodobne solidarnosti in z njo celotnega kakovostnega sožitja v družini, službi in družbi. Da bomo v Sloveniji in Evropi to 2. je osebni odnos človeka s človekom. priložnost izkoristili, moramo zavestna usmeriti veliko osebne in Storitev lahko bolje, hitreje in ceneje opravimo s poklicnimi družbene pozornosti na usposabljanje vsega prebivalstva za veščinami in tehnološkimi orodji, zlasti s sodobno IKT; ta bo v odnosno človeško oskrbovanje in hvaležno sprejemanje oskrbe. prihodnje lahko v celoti kakovostno opravljala oskrbovalne storitve. Pri opravljanju storitve je komunikacijski kanal odprt 3. TEZE O SODOBNEM SISTEMU enosmerno od oskrbovalca do oskrbovanca – ta je res uporabnik DOLGOTRAJNE OSKRBE storitve. Inštitut Antona Trstenjaka za gerontologijo in medgeneracijsko Druga bistvena razsežnost oskrbovanja je človeški odnos med sožitje dela od osamosvojitve Slovenije raziskovalno na tem nemočnim, ki za preživetje potrebuje pomoč, in (bolj) moč področju. Razvija in izvaja na terenu po Sloveniji in tujini nim, ki mu zmore to narediti. Človeški odnos z obojestransko odprto sodobne programe za zdravo staranje ter za usposabljanje komunikacijo ohranja človeško dostojanstvo nemočnega in oskrbovalcev, zlasti družinskih. Ves čas sodeluje z močnega. gerontološkimi in medgeneracijskimi strokovnjaki v Sloveniji in po svetu. Na tej osnovi smo povzemamo naslednja spoznanja o Močni ob človeški pomoči nemočnemu (otroku, bolniku, sodobni humani in finančno vzdržni dolgotrajni oskrbi; poleg umirajočemu ...) razvija svojo temeljno človeško zmožnost domačih in tujih raziskovalnih spoznanj in uspešnih praks smo jih solidarnosti, sočutja, empatije, vživljanja v drugega in oblikovali tudi ob zgoraj navedenih analizah obeh slovenskih sodoživljanja z njim, človeško občilnost. Te človeške zmožnosti predlogov zakona o dolgotrajni oskrbi, nekatera pa so bila so pogoj za vsako kakovostno komuniciranje in sodelovanje v oblikovana na 1. slovenski konferenci o družinski in drugi družini, službi in družbi, razvijamo pa jih lahko samo v praksi neformalni oskrbi, ki jo maja 2018 organiziralo Ministrstvo za pomoči nemočnim, ki ni brezosebno opravljanje storitve in ne zdravje RS, izvedel pa Inštitut; prispevki so objavljeni v 2. in 3. premoč močnejšega človeka nad slabotnejšim človekom – človeka številki revije Kakovostna starost 2018 (http://www.inst- sta namreč oba enako. antonatrstenjaka.si/tisk/kakovostna- starost/stevilka.html?ID=201802). Spoznanja naštevamo v obliki Nemočni s storitvami, ki mu jih opravijo drugi, telesno preživi in tez. se razvija; toda na tej ravni bo v prihodnje lahko opravila vse pametna tehnologija, danes pa storitveno oskrbuje profitno 1. Pogoj za vzpostavitev sodobnega sistema dolgotrajne organiziran oskrbovalni trg. Na človeški ravni je odločilen osebni oskrbe je politična odločitev, da za dolgotrajno oskrbo odnos med oskrbovalcem in oskrbovancem, pri katerem namenimo najmanj 1,3 % BDP, kakršno je povprečje oskrbovalec doživlja, da se ob tem, ko daje (storitev) potrebnemu, evropskih držav; v Sloveniji je to le 0,9 % BDP. sam človeško razvija. Zaradi tega odnosa oskrbovanec poleg storitve za telesno preživetje, ohranja in dobiva tudi duševno, 2. Sistem mora izhajati iz celovitega gledanja na socialno in duhovno moč, da ostaja celovito človeško bitje s oskrbovanca, ki potrebuje predvsem pomoč pri kljubovalno močjo duha v svoji omejen opravljanju vsakodnevnih opravil, in šele nato iz logike osti. storitev zdravstvenih, socialnih in drugih strok ter Množična potreba po oskrbovanju v času staranja prebivalstva je ustanov. Čeprav so storitve eden od bistvenih vidikov zato izjemna priložnost za humani razvoj tisočletne evropske oskrbe, izhajanje iz njih (kakor je to bilo v predlogu kulture, ki se je zadnji dve stoletji posvetila uspešnemu zakona o dolgotrajni oskrbi) onemogoča celovito 305 sodobno humano oskrbo, razvoj njene kakovosti in 8. Učinkoviti možnosti za preprečevanje nenujne potrebe finančno vzdržnost ob večanju potreb po oskrbi v po dolgotrajni oskrbi sta usposabljanje za zdravo prihodnje. staranje ter strokovna in finančna pomoč pri prilagajanju hiš in stanovanj za samostojnost ob 3. Sistem mora povezati v dopolnjujočo se celoto vse starostnem pešanju, boleznih in drugih oviranostih. glavne vire: družinske in druge neformalne oskrbovalce, ki oskrbujejo 75 % potrebnih, znanje in metode vseh 9. Za krepitev sodobne solidarnosti med generacijami pri relevantnih strok in oskrbovalnih ustanov, primerno oskrbi potrebnih članov družbe je smiselno ob motivacijo lokalne skupnosti, da poskrbi za svoje sprejemanju nacionalnega sistema za dolgotrajno oskrbo občane, ter državo, ki je regulatorka in garant uvesti po zgledu nekaterih razvitih držav za mlade med humanega, razvojno naravnanega in finančno vzdržnega 18. in 30. letom starosti socialno leto organiziranega sistema dolgotrajne oskrbe. Sodobni sistem dolgotrajne služenja v skupnosti (mnogi ga opravljajo prav v oskrbe je lahko human in finančno zdržen le, če sistemu dolgotrajne oskrbe) ter socialno služenje ljudi, sistemsko povezuje neformalno in formalno oskrbo v ki v času brezposelnosti prejemajo socialno pomoč dopolnjujočo se sinergično celoto. skupnosti. 4. V sistemu je treba povezati v celoto zdravstveni, 10. Usodna zapreka pri vzpostavljanju sodobnega sistema socialni, arhitekturno-infrastrukturni in vzgojno- dolgotrajne oskrbe je prevlada kateregakoli lobija, ki kulturni vidik oskrbovanja. Nepogrešljivo celoto vede ali nevede enači ta stvarni nacionalni interes s sodobne dolgotrajne oskrbe sestavljajo: znanja in parcialnim interesom svoje skupine, pa najsi je to metode zdravstvenih in socialnih strok, arhitekturno kapital, katera od zdravstvenih ali socialnih strok, urejanje bivalnih prostorov, ki so primerni za starostno upravni aparat, zavarovalnica ali politična stranka. pešanje, sodobni tehnični pripomočki pri oskrbovanju Kakovostno staranje – vključno s humano oskrbo v ter vzgojni, medijski in ostali kulturni vplivi na onemoglosti – je temeljna potreba slehernega človeka, razumevanje in krepitev medgeneracijske solidarnosti v pereča naloga ob staranju prebivalstva je skupna naloga današnjih razmerah. Učinkovit sistem dolgotrajne naše države, obvladovanje demografskega staranja oskrbe lahko vzpostavimo in razvijamo le, če so v njem slovenskega naroda pa življenjski interes vsakega, ki se vse navedene stroke enovita celota, njihovi politični in razvija v moči slovenske kulture. Ti skupni interesi in upravni resorji v državi in občini pa intersektorsko naloge so dobra priložnost za sodelovanje vseh povezani. političnih, strokovnih in civilnih deležnikov. 5. Dolgotrajna oskrba mora biti decentralizirano umeščena 11. Svojci – družinski oskrbovalci so danes najšibkejši člen v krajevno skupnost, enako kakor je otroško varstvo in med deležniki dolgotrajne oskrbe. Ker v Sloveniji in osnovno zdravstvo; država mora celovit sistem vsej Evropi pa opravljajo nad 75 % celotne dolgotrajne dolgotrajne oskrbe regulirati z zakonodajo, jamčiti oskrbe in ker so glavni nosilec človeškega odnosa z njegovo organiziranost in enakost v dostopnosti, voditi oskrbovancem, so nepogrešljivi del kakovostne in vzdržen sistem financiranja, zagotavljati minimalne finančno zdržne dolgotrajne oskrbe. Pogoj za kakovost standarde kakovosti, izvajati mora kontrolo nad nacionalnega sistema dolgotrajne oskrbe je intenzivni delovanjem izvajalcev in skrbeti za razvoj sistema. razvoj sodobnih programov za razbremenitev družinskih oskrbovalcev: njihovo usposabljanje, krajevni programi 6. Individualna upravičenost do financiranja iz sistema za pomoč in pravna zagotovitev njihovih pravic. dolgotrajne oskrbe omogoča – ob sodobni informacijski Usodna posledica zanje, za razvoj oskrbovalne stroke in tehnologiji, dobrih kriterijih za vstop, zagotovljenih oskrbovalnih programov nastane, če politika v boju za standardih oskrbovanja in učinkoviti kontroli nad moč izkorišča njihovo nevidnost in ranljivost. izvajalci – minimalno porabo sredstev za administracijo. To se doseže z dobro organizacijo obstoječih javnih Dolgotrajna oskrba je v času velikega povečanja deleža ljudi, ki služb za vstopanje v javno mrežo. Stroški za so odvisni od dolgotrajne oskrbe, glavna priložnost za nagel administracijo otežujejo finančno vzdržnost sistema, razvoj kakovosti sožitja in sodelovanja v sodobni družbi, v kateri predvsem pa onemogočajo razvoj njegove vsebinske in postajajo neuporabni tradicionalni vzorci solidarnosti. metodične kakovosti. Koncesije za oskrbovalne ustanove in organizacije (poleg dovoljenja za delo in 4. ZAKLJUČEK kontrole storitev) po Evropi ne uporabljajo, saj Sodobna ureditev humane in finančno vzdržne dolgotrajne oskrbe zmanjšajo pobudo skupnosti in povečajo škodljivo je pogoj za uspešno reševanje današnje demografske krize in za birokratizacijo centraliziranega sistema. nadaljnji human razvoj družbe. Brez dobre rešitve te naloge ni 7. Vlada mora skupaj s pripravo in sprejemanjem sistema rešljivo niti kakovostno in zdravo staranje velike deleža starejših in Zakona o dolgotrajni oskrbi poskrbeti za ljudi ob staranju baby boom generacije, niti vzpostavitev nove permanentno in kakovostno informiranje in ozaveščanje medgeneracijske solidarnosti ob koncu tradicionalne družbe. javnosti o dolgotrajni oskrbi v okviru celote sodobnih Evropske države sistematično rešujejo to nalogo od leta 1990, potreb in možnosti na področju staranja in s tem Slovenijo večinoma še čakajo, med drugim tudi uvedba in povezane krepitve solidarnosti med generacijami. O zakonska ureditev sodobnega sistema dolgotrajne oskrbe. Na delovanju nacionalnega sistema dolgotrajne oskrbe osnovi raziskovalnih spoznanj in dobrih praks v tujini in doma mora biti vzpostavljeno zelo dobro informiranje vseh ima vlada dobre možnosti, da to področje uredi ne le kot prebivalcev. zagotavljanje oskrbovalnih storitev, ampak ob njih celovito učenje vseh generacij za novo, posttraciionalno solidarnost med ljudmi. 306 To je najboljša in verjetno edina možnost, da kakovost [6] Ramovš, Jože. 2017. Sožitje v družini. Celje – Ljubljana: medčloveškega sožitja in sodelovanja v družini, službi in družbi Mohorjeva družba – Inštitut Antona Trstenjaka za doseže razvoj na materialnem področju, v telesnem zdravju in gerontologijo in medgeneracijsko sožitje. dolžini življenja. [7] Republika Slovenija – MDDSZ (2010). Predlog zakona o dolgotrajni oskrbi in zavarovanju za dolgotrajno oskrbo. 5. REFERENCE http://www.mddsz.gov.si/fileadmin/mddsz.gov.si/pageupload [1] Inštitut Antona Trstenjaka in Firis Imperl. 2010. Izjava o s/dokumenti__pdf/ZDOZDO_uvod__cleni_in_obrazlozitve_ zakonu o dolgotrajni oskrbi. Kakovostna starost, 13/2, 3-4. -_12.03.2010.pdf http://www.inst-antonatrstenjaka.si/tisk/kakovostna- [8] Yghemonos Stecy, Ogrin Alenka in Ramovš Marta. 2018. starost/clanek.html?ID=832 Neformalna oskrba kot politični izziv. Kakovostna starost, [2] Kakovostna starost. Revija za gerontologijo in 21/2, 70–86. medgeneracijsko sožitje (od 1998 naprej). Inštitut Antona [9] Svet EU. 2005. Zelena knjiga “Odziv na demografske Trstenjaka za gerontologijo in medgeneracijsko sožitje; spremembe: nova solidarnost med generacijami. Bruselj. http://www.inst-antonatrstenjaka.si/tisk/kakovostna- starost/predstavitev.html [10] Vlada RS (2017). Zakon o dolgotrajni oskrbi in obveznem zavarovanju za dolgotrajno oskrbo. [3] Leichsenring Kai, Billings Jenny in Nies Henk (Ur.). 2013. https://www.irssv.si/upload2/20102017_o_Z_o_dolg__oskrb Long-Term Care in Europe. Improving Policy and Practice. i_JR.pdf Hampshire: Palgrave Macmillan. [4] Ramovš, Jože (Ur.). 2017. Kakovostna starost, 20/4. [5] Ramovš, Jože (Ur.) 2013. Staranje v Sloveniji – Raziskava o potrebah, zmožnostih in stališčih nad 50 let starih prebivalcev Slovenije. Ljubljana: Inštitut Antona Trstenjaka. 307 Močna družinska medicina je najboljši način zdravstvene obravnave starajočega se prebivalstva doc. dr. Janez Rifel, dr. med. spec. druž. med. Katedra za družinsko medicino, Medicinska fakulteta Ljubljana, Univerza v Ljubljani Poljanski nasip 58 1000 Ljubljana T+ 386 4386915 janez.rifel@mf.uni-lj.si POVZETEK Zdravstveni sistemi težijo k fragmentaciji, poudarku na 1.2 Sprememba vrednot bolnišnicah in komercializaciji zaradi zakonov ponudbe in Zdravstveni sistemi težijo k fragmentaciji, poudarku na povpraševanja. Ljudje pričakujejo, da zdravstvena politika bolnišnicah in komercializaciji zaradi zakonov ponudbe in zmanjšuje neenakosti in pospešuje solidarnost. povpraševanja. Zdravstveni sistem je odsev globalne potrošniške družbe. Vendar pa se vedno bolj krepijo prepričanja da Države, ki so bolj orientirane v primarno zdravstveno varstvo potrošniški tip zdravstvenega sistema ni ustrezen odgovor na imajo nižje stopnje smrtnosti in boljše zdravstvene izide. Ljudje, resnične potrebe ljudi. Zdravje je vedno bolj čislan del ki imajo dobro primarno zdravstveno oskrbo v 10 do 15% vsakdanjega življenja. Ljudje želijo zdravstven sistem, ki bo v verjetneje poročajo o dobrem zdravju v primerjavi z ljudmi, ki le- njih videl posameznike s pravicami in ne samo stranke te nimajo. zdravstvenih programov. Ljudje so pripravljeni spoštovati Družinska medicina je v zadnjih desetletjih v Sloveniji dosegla zdravstvene delavce vendar želijo da so tudi sami spoštovani v nesluten razvoj in je priznana tako v slovenski kot tudi ozračju medsebojnega zaupanja [3]. Ljudje pričakujejo da mednarodni medicinski stroki. Hitremu strokovnemu, zdravstvena politika zmanjšuje neenakosti in pospešuje znanstvenemu in akademskemu razvoju družinske medicine pa ne solidarnost. V večini družb se ljudje strinjajo, da bi moral vsak sledi tudi močan položaj družinske medicine v zdravstvenem in imeti dostop do kakovostne zdravstvene oskrbe, ko jo potrebuje, političnem sistemu v Sloveniji. ne da bi zaradi tega bankrotiral ali padel v revščino. 1.3 Primarna zdravstvena oskrba Ključne besede Države, ki so bolj orientirane v primarno zdravstveno varstvo Družinska medicina, starostniki, zdravstveni sistem. imajo nižje stopnje smrtnosti in boljše zdravstvene izide [2]. Ljudje, ki imajo dobro primarno zdravstveno oskrbo v 10 do 15% verjetneje poročajo o dobrem zdravju v primerjavi z ljudmi, ki ...brez zdravja nima nič nobene vrednosti, niti denar niti karkoli nimajo dostopa do dobre primarne zdravstvene oskrbe [4]. Ena drugega... raziskava je pokazala, da če na vsakih 10000 prebivalcev ZDA Demokrit, peto stoletje pred Kristusom dodamo enega družinskega zdravnika se to pokaže v 5,3% boljših zdravstvenih kazalnikih (celotna smrtnost, smrtnost zaradi srčno- ...ohranitev zdravja je... ...brez dvoma prva žilnih bolezni, smrtnost dojenčkov, nizka porodna teža, dobrina in osnova vseh drugih dobrin v tem življenju... pričakovana življenska doba in samoocena zdravja) [5]. Pregled René Descartes, Discours de la Méthode, 1637 literature je pokazal da tako količina kot tudi kakovost oskrbe na primarni ravni vpliva na zdravje populacije in na delovanje celotnega zdravstvenega sistema. Primarna raven zdravstvene oskrbe preprečuje bolezni in smrti ne glede na to ali je oskrba 1. UVOD boljša zaradi večjega števila zdravnikov, zaradi boljše dostopnosti Zdravstveni sistemi se spontano ne nagibajo k vrednotam osnovne do primarne zdravstvene oskrbe ali zaradi večje kakovosti storitev zdravstvene dejavnosti in ne odgovarjajo učinkovito na nove na primarni ravni [5]. zdravstvene izzive. To je še posebej resnično, kadar ni prisotnih močnih voditeljev ali trdne politične volje. V današnjem svetu so 1.4 Družinska medicina v Sloveniji trije trendi, ki spodkopavajo zdravstveni sistem: osredotočenost V Socialistični federativni republiki Jugoslaviji je primarna raven na bolnišnice, fragmentacija in nebrzdana komercializacija medicine slonela na zdravstvenih domovih, kjer so bili zaposleni zdravstva. Bogate države so običajno lahko delno ublažile zdravniki. Prvi zdravstveni dom v Sloveniji je bil sicer negativne učinke teh trendov. Manj bogate države, kjer nadzor ustanovljen že leta 1926 v Lukovici pod vplivom Andrije nad zdravstvenim sistemom bolj šepa, so imele zaradi tega več Štamparja. Leta 1961 je bila prvič uvedena specializacija iz težav [1,2]. splošne oziroma družinske medicine v Zagrebu, leta 1974 je bil 308 ustanovljena katedra za splošno medicino na Šoli javnega zdravja Zagrebu, leta 2017 pa je v Zagrebu izšel še angleški prevod Andrija Štampar (Andrija Štampar je bil leta 1946 predsednik učbenika iz leta 2013. prve Svetovne zdravstvene skupščine, odločevalskega organa Od leta 2011 poteka projekt referenčnih ambulant, kjer se Svetovne zdravstvene organizacije). Od leta 1966 je bila zdravniku družinske medicine in srednji medicinski sestri v ožjem specializacija splošne medicine tudi v Sloveniji, trajala je tri leta, timu ambulante družinske medicine pridruži še diplomirana 2 leti v bolnišnici, eno leto pa je bil podiplomski tečaj iz javnega medicinska sestra, s čemer se okrepi preventiva v družinski zdravja, vodil ga je tedanji Nacionalni inštitut za javno zdravje. medicini. Prvi specialist splošne medicine v Sloveniji je bil leta 1966 Janez Schrott. V Sloveniji so bili neuspešni poskusi ustanovitve katedre Leta 2017 je dekan medicinske fakultete v Ljubljani postal za splošno oziroma družinsko medicino leta 1950 in 1975. Leta profesor Igor Švab, predstojnik katedre za družinsko medicino. 1966 pa se je ustanovila sekcija zdravnikov splošne medicine pri First Page Copyright Notice Slovenskem zdravniškem društvu. Leta 1983 je bila v Sloveniji prvič organizirana učna delavnica splošne medicine. Leta 1988 je 2. ZAKLJUČEK bil prvič izdan priročnik Nujna stanja, leta 1992 pa prvi učbenik Družinska medicina je v zadnjih desetletjih v Sloveniji dosegla splošne medicine. nesluten razvoj in je priznana tako v slovenski kot tudi Leta 1995 se končno na MF v Ljubljani ustanovi katedra za mednarodni medicinski stroki. Hitremu strokovnemu, družinsko medicino, sekcija splošne medicine se leta 1998 znanstvenemu in akademskemu razvoju družinske medicine pa ne preoblikuje v Združenje zdravnikov družinske medicine, ki sledi tudi močan položaj družinske medicine v zdravstvenem in postane največji organizator skupaj s katedro strokovnih srečanj političnem sistemu v Sloveniji. Od leta 2005 je bilo že vsaj pet vseh vrst za družinske zdravnike. Program specializacije iz poskusov sprejetja Nacionalne strategije razvoja osnovne družinske medicine je bil sprejet šele leta 1999. zdravstvene dejavnosti v Sloveniji, ki pa je vsakokrat obtičala v Preko Združenja je vsak zdravnik družinske medicine včlanjen predalih Ministrstva za zdravje Republike Slovenije. tudi v Svetovno združenje zdravnikov družinske medicine (WONCA). Leta 2003 je bila letna konferenca evropske WONCA 3. LITERATURA v Ljubljani. Slovenci sodelujemo v delovnih telesih WONCA, [1] The World Health Report 2008 - primary Health Care (Now prof. Igor Švab je bil od leta 2004 do 2010 predsednik WONCA More Than Ever). Dostop 15. maj 2018 na: evropske regije, od tedaj je še vedno tajništvo evropske WONCA http://www.euro.who.int/en/health-topics/Health- v Ljubljani. V Ljubljani sta tudi tajništvi EURACT (European systems/primary-health-care/publications/pre-2009/the- Academy of Teachers in General Practice/Family medicine) ter world-health-report-2008-primary-health-care-now-more- Europrev (European Network for Prevention and Health than-ever2 promotion in Family Medicine and General Practice). Predsednik [2] Macinko J, Starfield B, Shi L. 2003. The contribution of EURACT-a je bil od 2010 do 2015 prof. Dr. Janko Kersnik. primary care systems to health outcomes within Organization Europrev je od leta 2010 naprej vodila doc. dr. Mateja Bulc. for Economic Cooperation and Development (OECD) Pomembno je tudi članstvo v EGPRN (European General Practice countries, 1970-1998. Health Serv Res. (Jun.2003), Research Network) ter EURIPA (European and Rural Isolated 38(3):831-65. Practitioner Association). Katedra in Združenje sta in še vedno sodelujeta v veliko mednarodnih raziskovalnih projektih, v [3] Anand S. 2002. The concern for equity in health . J zadnjih letih objavijo člani raziskovalne skupine družinske Epidemiol Community Health. (Jul 2002), 56(7):485-7. medicine vsaj 50 člankov, ki so citirani v Medlinu. [4] Shi L et al. 2002. Primary Care, Self-Rated Health, and Sodelavci katedre so sodelovali v projektih Svetovne banke za Reductions in Social Disparities in Health. Health Services krepitev primarne ravni zdravstvene oskrbe v Črni gori, Research 37, no. 3 2002: 529-550. Makedoniji, Gruziji in Turčiji. Katedra je povezana z Imperial [5] Starfield B, Shi L, Macinko J. 2005. Contribution of primary college iz Londona in Maastrichtsko medicinsko fakulteto. Z care to health systems and health 83 ed., vol. 3: Blackwell veliko evropskimi univerzami je vzpostavljena izmenjava Publishing, 2005, 457-502. študentov in učiteljev prek Erasmusa. Leta 2012 je izšel nov učbenik Družinska medicina. Leta 2013 je bil ta učbenik z manjšimi dopolnitvami izdan v hrvaščini v 309 RAZŠIRJENI POVZETEK Usposabljanje menedžmenta za obvladovanje staranja zaposlenih JOŽE RAMOVŠ, TJAŠA GREBENŠEK Inštitut Antona Trstenjaka za gerontologijo in medgeneracijsko sožitje KLJUČNE BESEDE Demografska kriza, staranje zaposlenih, usposabljanje menedžmenta, sodelovanje med generacijami zaposlenih Razviti svet se prvič v zgodovini nahaja v krizi tukaj in zdaj je treba biti uspešen, sicer podjetje staranja prebivalstva, ko je starejših ljudi več kakor propada. mladih; v zadnjih desetletjih namreč z rojstvi Za reševanje tega kriznega stanja je na voljo nadomeščamo le dve tretjini za ohranitev istega števila zadostna, toda neizkoriščena možnost: sestaviti v domačih prebivalcev. Ena od treh bistvenih nalog za sinergijo nova znanja mladih in življenjske izkušnje obvladovanje demografske krize je ohranjanje starejših ter uporabiti doslej neizkoriščene človeške gospodarsko vzdržne družbe in socialno pravične razvojne zmožnosti. Edini vzvod, ki lahko vse to evropske družbe ob naglem staranju zaposlenih. poveže in ob staranju zaposlenih sproži nov razvoj, je Podatki o zaposlenih v Sloveniji in Evropi kažejo, vodstvo. Vodenje zaposlenih zahteva danes zaradi da se delež starejših naglo veča. V zadnjih desetih letih demografskega staranja tri nova znanja: se je delež zaposlenih, ki so starejši od 55 let, v 1. veščine za učinkovito vključevanje starejših nekaterih podjetjih in ustanovah celo potrojil: iz manj delavcev v delovni proces, kakor 10 % na blizu 30 %. V prihodnje bo v veliko 2. veščine za organiziranje dobrega sodelovanja med evropskih podjetjih več kakor polovica zaposlenih starejšimi in mlajšimi zaposlenimi, starih nad 50 let. Lastniki in vodstva podjetij, evropska 3. sodobna znanja o staranju na delovnem mestu, ki in državne politike, gerontološka stroka in drugi so pogoj za pridobivanje prvih dveh veščin. deležniki imamo zahtevno nalogo, da ustreznimi Ta možnost nam nudi dve odločilni prednosti. programi preprečimo škodljive poslovne posledice Prva: zadostuje, da lahko ohranimo konkurenčne tega demografskega razvoja med zaposlenimi. prednosti. In druga: če jo izkoristimo, nam rešuje V podjetjih doživljajo ob globalni konkurenci iz probleme slabega sodelovanja v delovnem in dežel z veliko mladega prebivalstva in poceni delovno vsakdanjem sožitju, ki so se nam nakopičili v dveh silo vsi zaposleni hud pritisk – mladi zaposleni stoletjih, ko je evropska kultura posvečala vse sile drugače, starejši drugače in vodje drugače. Starejši materialnemu razvoju. Menedžmentu na vseh ravneh zaposleni se neredko počutijo iztrošeni, težko sledijo vodenja se torej zelo splača osvajati veščine za vodenje razvojnim zahtevam ter se učijo uporabe novih medgeneracijskega sodelovanja ob staranju tehnologij in postopkov, z nezaupanjem gledajo v zaposlenih. mlade. Neredko naveličani čakajo upokojitev, pri tem pa računajo na posebne ugodnosti. Navade in Na Inštitutu Antona Trstenjaka za gerontologijo in mentaliteta mladih zaposlenih so zelo drugačni od medgeneracijsko sožitje smo za reševanje te naloge nekdanjih. Imajo veliko novih znanj, zlasti v razvili intenzivni seminar Menedžment sodelovanja ob nenadomestljivi elektronski tehnologiji, malo pa staranju zaposlenih. Z njim usposabljamo vse ravni življenjskih izkušenj, starejših ne razumejo. Mlajši in vodstva v podjetjih in ustanovah, vključno z lastniki. starejši težko navezujejo med seboj dober stik in Izvajamo ga v skupinah od 7 do 9 direktorjev in drugih učinkovito sodelujejo. V teh razmerah je vodenje v vodij na treh učnih srečanjih po štiri ure, praviloma v podjetju zelo zahtevno. Trg in dejstva so neizprosna: treh zaporednih tednih. Edina učinkovita metoda pri 310 tem je aktivno skupinsko socialno učenje po načelu vsi poznavanju med mlajšimi in starejšimi zaposlenimi ter smo učitelji in vsi učenci – razlike nas bogatijo. nasploh po boljšem sodelovanju. Udeleženci osvajajo spoznavanje in veščine za zgoraj navedena tri področja z učenjem:  iz lastnih dobrih izkušenj ter spoznavanjem orodij za učinkovito obdelavo slabih izkušenj,  s spoznavanjem celote človeških potreb, zmožnosti in motivacije v telesno-materialni, duševni, duhovni, socialni, razvojni in bivanjski razsežnosti,  osvajanjem veščin za uresničevanje sinergične komplementarnosti novega znanja mlajših in življenjskih izkušenj starejših delavcev,  namesto posredovanja spoznanj v obliki predavanj, dobijo dva priročnika, v katerih smo kratko in ciljno zbrali sodobna znanja z vidika potreb in možnosti menedžerjev,  vsak udeleženec s seminarsko nalogo reši en svoj konkretni vodstveni primer ob staranju zaposlenih. Ker je demografski problem velikega deleža starejših zaposlenih nov, je glavni učinkoviti vir potrebnega znanja in veščin za njegovo reševanje učenje iz izkušenj. Metoda skupinskega socialnega učenja omogoča ustrezno obdelovanje konkretnih dobrih in slabih izkušenj menedžerjev na tečaju. Na ta način postanejo njihove izkušnje učinkovit vir osebnega razvoja vodij in razvoja podjetja. Zaradi te sodobne metode je seminar kljub poglobljenim sodobnim strokovnim spoznanjem v celoti praktičen po meri konkretnih potreb udeležencev. Seminarje izvajamo interno za velika podjetja z več sto vodij ali pa v sestavljenih skupinah, kamor pridejo posamezni menedžerji iz malih podjetij, zavodov ali društev. V prvem primeru je določena prednost skupna identiteta in podobnost problemov staranja zaposlenih, v drugem pestrost različnih izkušenj. Evalvacija več deset seminarjev z nekaj sto udeleženci kaže, da so menedžerji in njihova podjetja zelo zadovoljni. Čeprav so učna srečanja raztegnjena na tri tedne in so menedžerji zelo zaposleni, so se tečaja udeleževali redno, vzdušje je bilo prijetno, sodelovanje zavzeto. Seminar Menedžment sodelovanja ob staranju zaposlenih in metodično podoben, vsebinsko pa zelo različen seminar za zaposlene po 50. letu starosti, sta naš odgovor na raziskovalna spoznanja o staranju zaposlenih in na čedalje bolj perečo potrebo po 311 312 Zbornik 21. mednarodne multikonference INFORMACIJSKA DRUŽBA – IS 2018 Zvezek G Proceedings of the 21st International Multiconference INFORMATION SOCIETY – IS 2018 Volume G Sodelovanje, programska oprema in storitve v informacijski družbi Collaboration, Software and Services in Information Society Uredil / Edited by Marjan Heričko http://is.ijs.si 9. oktober 2018 / 9 October 2018 Ljubljana, Slovenia 313 314 PREFACE This year, the Conference “Collaboration, Software and Services in Information Society” is being organised for the eighteenth time as a part of the “Information Society” multi-conference. As in previous years, the papers from this year's proceedings address actual challenges and best practices related to the development of advanced software and information solutions as well as collaboration in general. Information technologies and the field of Informatics have been the driving force of innovation in business, as well as in the everyday activities of individuals for several decades. Blockchain technology, Big Data, intelligent solution, reference models, open standards, interoperability and the increasing responsiveness of IS/IT experts are leading the way to the development of intelligent digital service platforms, innovative business models and new ecosystems where not only partners, but also competitors are connecting and working together. On the other hand, quality assurance remains a vital part of software and ICT-based service development and deployment. The papers in these proceedings provide a better insight and/or propose solutions to challenges related to: - Self-Assessment of Sustainability of ICT in SMEs; - Ontology-based knowledge sharing on BPMN graphical signs using semiotics; - Influence of notations used for conceptual design on knowledge perception; - Application of machine learning techniques to obtain new knowledge; - Establishment of domain specific reference models; - Introduction of Blockchain technology into real-life use cases; - Architectural design proposals for ensuring scalability of Blockchain platforms; - Application of usability questionnaires when evaluating gamification and serious games - Visualization, analysis and comprehension of complex software systems; - Continuous software development, integration and delivery; - Integration of source code repositories and QA tools. We hope that these proceedings will be beneficial for your reference and that the information in this volume will be useful for further advancements in both research and industry. Prof. Dr. Marjan Heričko CSS 2018 – Collaboration, Software and Services in Information Society Conference Chair 315 PREDGOVOR Konferenco “Sodelovanje, programska oprema in storitve v informacijski družbi” organiziramo v sklopu multikonference Informacijska družba že osemnajstič. Kot običajno, tudi letošnji prispevki naslavljajo aktualne teme in izzive, povezane z razvojem sodobnih programskih in informacijskih rešitev ter storitev kot tudi sodelovanja v splošnem. Informatika in informacijske tehnologije so že več desetletij gonilo inoviranja na vseh področjih poslovanja podjetij ter delovanja posameznikov. Tehnologija veriženja blokov, velepodatki, inteligentne storitve, referenčni modeli, odprti standardi in interoperabilnost ter vedno višja odzivnost informatikov vodijo k razvoju inteligentnih digitalnih storitvenih platform in inovativnih poslovnih modelov ter novih ekosistemov, kjer se povezujejo in sodelujejo ne le partnerji, temveč tudi konkurenti. Napredne informacijske tehnologije in sodobni pristopi k razvoju, vpeljavi in upravljanju omogočajo višjo stopnjo avtomatizacije in integracije doslej ločenih svetov, saj vzpostavljajo zaključeno zanko in zagotavljajo nenehne izboljšave, ki temeljijo na aktivnem sodelovanju in povratnih informacijah vseh vključenih akterjev. Ob vsem tem zagotavljanje kakovosti ostaja eden pomembnejših vidikov razvoja in vpeljave na informacijskih tehnologijah temelječih storitev. Prispevki, zbrani v tem zborniku, omogočajo vpogled v in rešitve za izzive na področjih kot so npr.: - samoocenitev kakovosti in zrelosti IKT podpore v malih in srednje velikih podjetjih; - deljenje znanja o grafičnih simbolih BPMN z uporabo semiotike; - vpliv notacije, uporabljene pri oblikovanju konceptualih modelov, na dojeti nivo pridobljenega znanja; - uporaba tehnik strojnega učenja za ekstrakcijo znanja; - vzpostavitev domenskih referenčnih modelov; - vpeljava tehnologije veriženja blokov v realne primere uporabe; - arhitekturni predlogi za rešitev razširljivosti platform tehnologije veriženja blokov; - uporaba standardnih vprašalnikov uporabnosti pri vrednotenju učinkov vpeljave igrifikacije in resnih iger; - vizualizacija, analiza in razumevanje kompleksnih programskih sistemov; - neprekinjen razvoj, integracija in dobava informacijskih rešitev; - integracija repozitorijev izvorne kode z orodji za zagotavljanje kakovosti. Upamo, da boste v zborniku prispevkov, ki povezujejo teoretična in praktična znanja, tudi letos našli koristne informacije za svoje nadaljnje delo tako pri temeljnem kot aplikativnem raziskovanju. prof. dr. Marjan Heričko predsednik konference CSS 2018 – Collaboration, Software and Services in Information Society Conference 316 PROGRAMSKI ODBOR / PROGRAM COMITTEE Dr. Marjan Heričko University of Maribor, Faculty of Electrical Engineering and Computer Science Dr. Gabriele Gianini University of Milano, Faculty of Mathematical, Physical and Natural Sciences Dr. Hannu Jaakkola Tampere University of Technology Information Technology (Pori) Dr. Mirjana Ivanović University of Novi Sad, Faculty of Science, Department of Mathematics and Informatics Dr. Zoltán Porkoláb Eötvös Loránd University, Faculty of Informatics Dr. Stephan Schlögl MCI Management Center Innsbruck, Department of Management, Communication & IT Dr. Zlatko Stapić University of Zagreb, Faculty of Organization and Informatics Dr. Vili Podgorelec University of Maribor, Faculty of Electrical Engineering and Computer Science Dr. Maja Pušnik University of Maribor, Faculty of Electrical Engineering and Computer Science Dr. Muhamed Turkanović University of Maribor, Faculty of Electrical Engineering and Computer Science Dr. Boštjan Šumak University of Maribor, Faculty of Electrical Engineering and Computer Science Dr. Aida Kamišalić Latifić University of Maribor, Faculty of Electrical Engineering and Computer Science Dr. Gregor Polančič University of Maribor, Faculty of Electrical Engineering and Computer Science Dr. Luka Pavlič University of Maribor, Faculty of Electrical Engineering and Computer Science 317 318 Self-Assessment Tool for Evaluating Sustainability of ICT in SMEs Jari Soini Jari Leppäniemi Pekka Sillberg Tampere University of Technology Tampere University of Technology Tampere University of Technology P.O. Box 300 P.O. Box 300 P.O. Box 300 FI-28101 Pori, Finland FI-28101 Pori, Finland FI-28101 Pori, Finland jari.o.soini@tut.fi jari.leppaniemi@tut.fi pekka.sillberg@tut.fi ABSTRACT effects (e.g., electricity used by database servers, cloud servers, The ever-increasing demand for ICT may compromise global and network routers) that may not be consciously recognized [1, objectives for emissions reduction if the aggregate effects of ICT 2, 3, 4]. Typically users are concerned only of the electricity sustainability are not considered in the business digitalization consumption of their own devices. The increasing demand for ICT processes. In this paper, we present a free self-assessment tool may, in fact, compromise the national objectives for emissions enabling small and medium sized companies to evaluate the reduction if the aggregate effects of ICT un-sustainability (Figure utilized ICT in terms of sustainability. The ICT4S is a free e- 1) are not considered in the business digitalization processes. service, in effect, a web-based self-assessment tool that was developed in co-operation with Swiss Green IT SIG. The assessment is currently divided into five categories of sustainability questions. The categories are strategy, procurement and recycling, practices, servers and network, and Green ICT. As the result, organizations will gain a general understanding about their state of sustainability, and practical suggestions for greater eco-friendliness and sustainability of their ICT operations. Categories and Subject Descriptors • Social and professional topics~Sustainability • Information systems~Web applications Figure 1. Environmental impacts of the ICT. [5] In 2017, it was estimated that ICT accounted for 12% of the General Terms overall electricity consumption around the globe, and the Measurement, Performance, Human Factors. percentage is expected to increase twice as rapidly in the future (by approximately 7% per year). Most of the energy is consumed by networks, server rooms, and computing centers, (Figure 2) the Keywords efficiency of which should urgently be improved. Sustainability, Assessment, ICT, Metrics, Web tools, E-services. 1. INTRODUCTION The study presented in this paper aims at contributing to the business activity digitalization of companies concerning the reduction of carbon footprint and improvement of sustainability. The paper introduces a self-assessment tool developed in the research project that allows companies to self-evaluate the sustainability of the ICT exploited in the organization. The objective is to provide companies with concrete tools and proposals for actions enabling more ecological procedures in the organization. Additionally, the knowledge gained by using the self-assessment tool allows companies to become generally more Figure 2. Electricity consumption in the ICT sector. [6] aware of the distribution of energy consumption in a modern ICT As most of the electricity is still being generated by using fossil infrastructure as well as the factors affecting sustainability of ICT. fuels (Figure 3), the current ICT, and its heavy usage of electrical energy, constitutes a global issue that is, unfortunately, little 2. BACKGROUND known outside the expert field [7, 8]. This is partly due to the There is a lot of evidence for significant benefits in terms of users not perceiving the energy consumption of data systems productivity and cost savings through the exploitation of ICT in operating invisibly or in the background, but rather only noticing the daily business activity of organizations. However, the the consumption of the terminal device, which, in reality, increasingly dependent use of ICT also brings about “invisible” comprises a fraction of the overall energy consumption (Figure 2). 319 In 2015 through 2017, the TUT Pori Department implemented a research project (AjaTar) with the aim of improving the digitalization of organizations and companies while promoting a low-carbon economy and sustainability. As part of the project, a technology enabling organizations to self-evaluate their ICT sustainability was developed, tested, and studied, aiming at increasing general awareness of the distribution of electricity consumption in a modern IT infrastructure in order for the organizations to be able to make ICT-related decisions more consciously than before. The most notable added value of the project comprise an increase in knowhow and knowledge promoting easy and lightweight Figure 3. Electricity generation by source of energy. [9] assessment of sustainability in terms of the organization’s The problem of energy consumption due to the constantly business activities and support processes, as well as a freely increasing utilization of ICT is expected to further worsen available tool for evaluating the sustainability of the ICT used in (Figures 4a and 4b) through the amount of IoT devices and the organization. By making the sustainability issues visible, the automatic steering systems [10]. If the majority of the predicted objective was to change attitudes and conventions related to the IoT devices and information systems supporting them are utilization of ICT in organizations: indeed, during the project, implemented by the current practices, a near-catastrophic peak several organizations distinctly declared their need to recognize demand in terms of electricity will ensue. This, in turn, will result practices promoting sustainable development as well as invest in in an increase in emissions rather than their reduction. an eco-friendly image. 3. ICT4S SELF-ASSESSMENT TOOL During the last six years, the SEIntS research group from TUT Pori Department has studied, developed, and piloted innovative ICT solutions in cooperation with local organizations. Additionally, SEIntS has collaborated with, for example Keio University in Japan as well as with various information society associations, for example, in Switzerland regarding the Green IT and assessment of datacenters. As a result of the AjaTar project, an open self-assessment website for organizations to quickly and easily evaluate the ecological aspects of their ICT-related operations was published at the end of 2017. The self-assessment tool, developed in collaboration with Green IT SIG, a Swiss Green IT information special interest group, is based on the assumption that most of the ICT equipment used in an organization is controllable, enabling the relatively easy (a) Estimated growth. adjustment of various functions. With the assessment tool developed in the project, it is possible to increase knowledge about the ecological aspects related to the use of ICT in organizations and, thus, affect their operations and practices. Based on the self-assessment, the organization is offered overall evaluation of the current state and propositions for practices for more sustainable ICT operations. The self-assessment tool is freely available on a dedicated website for sustainable ICT [11]. On the landing page of the tool (Figure 5) there is a welcoming message that explains the goals of the assessment. There is also information of the privacy solution that is used to guarantee all the information of the assessor’s company. The privacy solution is based on the HTML5 local storage concept. The assessment menu is currently divided into five categories of sustainability questions and the information of the organization to be evaluated. The categories are: strategy, procurement and recycling, practices, servers and network, and (b) Estimated standby energy consumption. Green ICT. Figures 4a and 4b. Estimated growth and impact of IoT devices. [10] Therefore, it is essential to establish instructions and an assessment procedure to support system planning to improve sustainability of ICT, and, thus, to promote methods for a low- carbon economy. 320 After assessing all categories, the assessment tool calculates and shows an evaluation of the given answers. The results are first shown in a short form as in Figure 8, but users can explore the results more carefully by selecting “Display detailed evaluation.” The percentage and the color of the beams give a fast response of the maturity of the different categories. In the case of 100% and a green beam, the user can be satisfied with the sustainability state of their company in that certain category. In the case of low percentages (0 - 70%) or yellow or even red beams, the evaluation shows that there is room for improvement. In such a case, the user may find the detailed evaluation useful when planning concrete actions for these improvements. Figure 5. Welcoming the assessors. Each of the categories comprises several questions and additional text that explains the current issue to the assessor. While trying to answer the questions, the assessor also receives background information on the current topic. In Figures 6 and 7, the assessor is facing questions concerning the strategy and practices at the office. Figure 8. Brief results of the assessment. The detailed evaluation can be shown by selecting the corresponding option in the user interface (see Figure 9). The user is also able to print the results – hopefully in a sustainable way, for example using an e-format such as Portable Document Format (PDF). Figure 6. Assessing the strategy. Figure 9. Detailed results of the assessment. The assessment tool has now been in use for several months. Unfortunately, we do not have the exact statistics concerning the usage of the tool. However, we piloted the tool with the assistance of local companies before launching it last December. Since the piloting groups were satisfied with the tool and because we wanted to keep our promises regarding the privacy of the assessments, we did not implement any logging system in it. We have planned to enhance the tool with a new capability – aiming to enable an easy way to estimate the carbon footprint of the ICT usage in a company. It will not be fully scientific life Figure 7. Assessing practices at the office category. cycle assessment (LCA) but a practical version of such targeted to 321 non-professionals in the field of sustainability. The reasoning for [3] Amsel, N., Ibrahim, Z., Malik, A. and Tomlinson, B. 2011. this new capability is that we anticipate that by introducing easy Toward sustainable software engineering: NIER track. assessment tools we will be able to raise the awareness of published in 33rd International Conference on Software companies in terms of sustainability issues and thus help them to Engineering (ICSE), 21-28 May 2011, Honolulu, USA. develop their business processes toward a sustainable state. [4] Baliga, J., Hinton, K., Ayre, R. and Tucker, R.S. 2009. 4. RESULTS AND FUTURE WORK Carbon footprint of the internet. Journal of Australia, vol. 59, no. 1, 5.1-5.14. This paper presented the ICT4S self-assessment tool enabling companies and other organizations to evaluate the utilized ICT in [5] Hilty, L. and Aebischer, B. (eds.). 2015. ICT Innovations for terms of a low-carbon economy and sustainability and thus Sustainability. Advances in Intelligent System and improve their image as well as resource efficiency. As the result, Computing 310, Springer International Publishing, organizations will gain a general understanding of the current Switzerland. sustainability state of their ICT and practical suggestions for more [6] Corcoran, A. and Andrae, A. 2013. Emerging Trends in eco-friendly and sustainable operations. Electricity Consumption for Consumer ICT. Retrieved The role of the TUT Pori Unit was to function as a producer and August 22, 2018 from https://www.researchgate.net/profile/ facilitator of new knowledge. The applied project aimed at Anders_Andrae/publication/255923829_Emerging_Trends_i contributing to the business development with TUT Pori Unit n_Electricity_Consumption_for_Consumer_ICT/ acting as a distributor of knowledge and knowhow as well as an [7] Pickavet, M., Vereecken, W., Demeyer, S., Audenaert, P., innovator. Within the project, the accumulation of diverse energy- Vermeulen, B., Develder, C., Colle, D., Dhoedt, B. and related knowhow and knowledge and exploitation of sustainable Demeester, P. 2008. Worldwide energy needs for ICT: The solutions of ICT in organizations were successfully implemented. rise of power-aware networking. In proceedings of 2nd Further development is planned to be realized in the ICT4LC International Symposium on Advanced Networks and project launched at the beginning of 2018. It focuses on Telecommunication Systems, 1-3. examining contemporary information processing that is based on [8] Lambert, S., and Van Heddeghem, W. 2012. Worldwide mobile and ‘thin clients’ as well as the increasing utilization rate electricity consumption of communication networks. Optics of information networks and cloud computing. The new project Express, vol. 20, no. 26, 513-524. explores tools for assessing the energy efficiency of business [9] OECD Factbook 2014: Economic, Environmental and Social activities and support processes as well as planning procedures of Statistics. Retrieved August 27, 2018 from business processes, promoting responsible and sustainable http://dx.doi.org/10.1787/888933025499 utilization of ICT in organizations. [10] International Energy Agency. 2016. Energy Efficiency of the 5. ACKNOWLEDGMENTS Internet of Things, Technology and Energy Assessment Our thanks to Niklaus Meyer and Beat Koch from Swiss Green IT Report. Prepared for IEA 4E EDNA. Retrieved August 27, SIG for collaboration. 2019 from https://www.iea-4e.org/document/384/energy- efficiency-of-the-internet-of-things-technology-and-energy- assessment-report 6. REFERENCES [11] Tampere University of Technology. 2017. ICT4S Self [1] Hilty, L., Arnfalk, P., Erdmann, L., Goodman, J., Lehmann, Assessment. Retrieved August 27, 2018 from https://green- M., Wager, A.P. 2006. The relevance of information and ict.fi/arviointi/?lang=en communication technologies for environmental sustainability – A prospective simulation study. Environmental Modelling & Software 2006, vol. 21, issue 11, 1618-1629. [2] Hilty, L. 2008. Information technology and sustainability: Essays on the relationship between ICT and sustainable development. Books on Demand, Norderstedt. 322 Reference Standard Process Model for Farming to Support the Development of Applications for Farming Rok Rupnik Faculty of Computer and Information Science University of Ljubljana Ljubljana, Slovenia +386 1 479 8266 rok.rupnik@fri.uni-lj.si ABSTRACT have managed to define so far: Domains, processes and elements The paper introduces the idea and the concepts of a Reference of process description. We also introduce the current list of Standard Process Model (RSPMF) which are based on the concepts processes and domains. of COBIT, an IT governance framework used worldwide. Our The structure of the paper is as follows. The second chapter research on RSPMF is focused in two directions. First, RSPMF is introduces the EU funded project AgroIT, during which the idea for aimed at becoming a support for Product Managers in software the Reference Standard Process Model arose. Only aspects of the companies developing software products or IoT systems. Namely, project relevant for the content of this paper are introduced. The each process in RSPMF is described through the following third chapter introduces key findings from the AgroIT project components: Process goals, process metrics, KPI’s (Key which led to the idea of RSPMF. To support the idea of RSPMF the Performance Indicators) and process activities, Second, RSPMF is COBIT framework for IT governance is also introduced, since aimed to help managers or owners of bigger farms in farm many concepts of RSPMF are taken from the COBIT framework. management. The paper introduces research in the progress state of The fourth chapter introduces the RSPMF, its concepts, draft list of our research. domains and their processes, and the methodology to facilitate the sustainability of RSPMF. The last chapter contains the conclusion Categories and Subject Descriptors and directions for future work on the RSPMF. D.2.2 [Requirements/Specifications]: Tools. 2. EXPERIENCE GAINED IN THE AgroIT General Terms PROJECT Farming, Standardization, Process model. AgroIT was an EU funded project covering various previously mentioned aspects and problems in today’s implementation of IT Keywords and IoT in farming [5], [6]. First, the project included the Standard Process Model, COBIT, Transformation of model. implementation of ERP systems for farming: A traditional ERP 1. INTRODUCTION system for small and medium enterprises which, additionally, also has modules for livestock, fruit growing, winery, etc. [7]. This area In recent years, farming has become an area with extensive need of farm management was covered, which was the subject of several for the use of information systems and IoT technologies [1]. The papers in recent years [8][1], [2], [6], [7], [9], [10]. Second, the experience gained in an EU funded project has revealed that project included the implementation of a decision support system software companies have diverse and unequal knowledge and based on advanced methods to support decision processes in understanding of farming processes, activities within processes and farming [8]. This way, the area of the use of decision support within metrics. This causes a problem when software products and IoT farm management was covered [1], [6]. Third, the project included systems need to be integrated. There are many software products the implementation of IoT systems where various sensors were and IoT systems on the market today, but each of them covers a used to collect data about several measurements [2], [11], [12]. quite narrow functional area and, for the treason the integration, is Having (a lot of) data available is the basis for farm management simply a necessity [2]. and operations of farms [13]. Fourth, the project also covered the The Reference Standard Process Model is one way to help Product implementation of the cloud integration platform. All applications Managers at software companies in removing the gap of diverse and IoT systems were integrated through the cloud integration and unequal knowledge and understanding of farming processes, platform to facilitate data exchange between them [6], [12], [14]. activities within processes and metrics. The reference model can Six software companies (they were called software partners during become a common denominator, a kind of Esperanto, as a the project) cooperated in the AgroIT project with their software knowledge base for the development of software products and IoT products: Applications, IoT systems and the cloud integration systems for farming. The reference model, on the other hand, will platform. Each software company “contributed” their product to the also help farm managers and owners in farm management. project and, during the project, software products were improved We built and designed a Reference Standard Process Model for significantly, i.e. upgraded and extended. They were also improved Farming (RSPMF) based on the idea and concepts of the COBIT implicitly through integrations between each other. framework, which is defined for the area of IT governance [3], [4]. For the pilot use of integrated software products and IoT systems This paper introduces the research in progress and the concepts we several pilot projects were organised in 5 EU countries by pilot 323 partners. Pilot partners did not do software implementation in the Figure 1. COBIT meta model [3] project, but supported pilot farms in the use of software products. A detailed explanation of the schema, i.e. a detailed explanation of For that reason, pilot partners were organisations with extensive the concepts and relations between them is beyond the scope of this knowledge in agriculture and experience in consulting for farming. paper. 3. KNOWLEDGE OF FARMING FOR 3.2 The idea of the Standard Process Model IMPLEMENTATION OF SOFTWARE for Farming PRODUCTS AND IoT SYSTEMS FOR The idea and concept of the previously introduced COBIT FARMING framework and the problems based on the diversity of knowledge of partners in the project initiated the idea of a Standard Process Improving software products and IoT systems was based on and Model for farming. COBIT is,, therefore based on various concepts, extending existing functionalities of software products and IoT and those concepts can be used and adapted in other areas as well, systems and upgrading them with new ones. The key goal of the not only in IT governance. The idea and concepts of COBIT were project was to design functionalities which base on integration already transferred and used in the governance of Flood between software products and IoT systems. This means that a Management [15] and Nursing [16]. software product also can use data from another software product or IoT system. The transfer of the idea and concepts of a particular Standard or framework to another area, in this case the transfer of COBIT to the During the analysis and design phase it has become apparent that area of farming, does not mean a one-to-one transfer. Some software partners have diverse and unequal knowledge and concepts of source area (in this case, IT governance), might not be understanding of farming processes, activities within processes and relevant or have any sense in the destination area (in this case metrics. The gap was even bigger when compared to the knowledge farming). For this reason, a successful and significant transfer with and understanding of the pilot partners. useful outcome can only be achieved through: The diversity mentioned, and having the expertise of COBIT, has,  Good understanding of the idea and concepts of the step-by-step, led to the idea of transferring the idea of COBIT to be framework of the source area (in this case COBIT), used for farming [3], [4].  Extensive knowledge and experience on the destination area: Processes and their activities, metrics, 3.1 COBIT framework for IT governance responsibilities, rules, etc. COBIT has, in recent years, become a de-facto Standard for IT governance in companies and organisations. COBIT defines a set 4. REFERENCE STANDARD PROCESS of generic processes (IT processes) for the management of IT. For MODEL FOR FARMING (RSPMF) each IT process the following is defined: Process inputs and As can be concluded based on the previous discussion, we designed outputs, goals of the process, key process activities, metrics of the RSMPF on the idea and concepts of COBIT 4.1 [3]. In the literature process (performance measures), and levels of process maturity we so far haven’t found any paper representing a Standard Process (maturity model) [3]. The development of COBIT has been Model for Farming. progressing since 1996, from version 1 to the current version 5. COBIT is the result of several working groups of highly 4.1 The concepts of RSMPF experienced experts as coordinated work owithin ISACA, which is Processes are divided on three hierarchical levels which are called an international professional association focused on IT governance. domains: Govern and Monitor (GM), Plan and Manage (PM) and COBIT is defined as a process model which divides IT into four Implement and Execute (IE). domains: Plan and Organise, Acquire and Implement, Deliver and Farming has several branches: Livestock, fruit growing, Support, and Monitor and Evaluate). Domains have altogether 34 agriculture, winery (viticulture), etc. RSMPF enables modular defined IT processes. definition of processes for every branch of farming. For the Govern The schema below shows a meta model of COBIT and all of its and Monitor domain only common processes are defined, for the concepts. The schema reveals the business orientation of COBIT: other two domains, a process module is also added for every branch The aim of defining the COBIT framework is to align IT and of farming. For now, only the process module for livestock is business where business goals dictate IT goals [3], [4]. defined for domains PM and IE. Each process is described through the following components: Process goals, process metrics, KPI’s ( Key Performance Indicators) and process activities. Each process has a unique code, which reveals the domain to which the process belongs and the process module. The code of Common Processes is CP and the code for LiveStock is LS. The aim of defining RSPMF is not to prevail over any existing Standard for farming. RSMPF is defined and structured to be opened and enables the reference to any existing Standard in the process description section. 4.2 Target groups and aimed benefits of RSPMF When designing a Standard Process Model, regardless of the area it is intended for, the group designing it must first decide which are 324 the target groups who will use the model, and what should be the  GM.09: Implement and monitor implementation of benefits of its use. For target groups this should become a strategy Reference Standard Process Model. Plan and Manage (PM) – Common Processes (CP):  We designed RSPMF for the following groups: PM.CM.01: Manage implementation of strategy and investments  Product Managers in software companies which  PM.CM.02: Manage budget and cost develop software products and IoT systems for farming.  PM.CM.03: Manage financials As can be revealed from our discussion, we noticed the  PM.CM.04: Manage risks need for a Standard Process Model,   PM.CM.05: Manage human resources Managers and owners of bigger farms: COBIT is the  PM.CM.06: Manage buildings and security first place aimed at bigger companies. Each Standard  PM.CM.07: Manage products sales Process Model should, in our opinion, be sized for bigger  institutions (organisations in general). Smaller PM.CM.08: Manage suppliers  institutions then use it to the extent for which they PM.CM.09: Manage sub-contractors believe is suitable for them. We followed this approach  PM.CM.10: Manage certifications in the designing of the RSPMF.  PM.CM.11: Manage environment and protection  PM.CM.12: Manage energy consumption The aimed benefits for Product Managers are as follows:  PM.CM.13: Manage energy production  Based on experience from the AgroIT project, we can  PM.CM.14: Manage farming machinery state that there is a diversity of farming knowledge of  PM.CM.15: Manage equipment Product Managers in software companies. RSPMF will  PM.CM.16: Manage IT become a common denominator, a kind of Esperanto as  PM.CM.17: Manage information system a knowledge base for the development of software  PM.CM.18: Manage innovations products and IoT systems for farming,   PM.CM.19: Manage investment projects We expect the integrations between various software  PM.CM.20: Manage needs and expectations products and IoT systems to be more straightforward and “softer” if  PM.CM.21: Manage knowledge and legislation Product Managers will base  functionalities on RSPMF. PM.CM.22: Manage changes based on legislation demands We are designing RSPMF to reach several aimed benefits for  PM.CM.25: Manage changes based on IT and managers and owners of bigger farms: innovation  Knowledge and experience of farming experts and  PM.CM.26: Manage assets academics will, step by step, be transferred to RSPMF.  PM.CM.27: Manage technical capacity We could say that RSPMF introduces the best practices  PM.CM.28: Manage internal control for farming, Plan and Manage (PM) – LiveStock (LS):  RSPMF provides the best practice guidelines for  PM.LS.01: Manage animal sales processes and their activities on farms. This helps  PM.LS.02: Manage animal purchases managers ensure that the processes perform according  PM.LS.03: Manage animals` health and veterinary to best practice, service  Metrics and KPI’s are defined for processes. This helps  PM.LS.04: Manage animal welfare managers to set goals and execute monitoring. This  PM.LS.05: Manage hygiene lowers various risks,  PM.LS.06: Manage animal feeding and grazing  Managers can identify gaps in process execution and  PM.LS.07: Manage animal reproduction monitoring. This helps them close the gaps identified  PM.LS.08: Manage animal breeding plan and improve processes, Implement and Execute (IE) – Common Processes (CP):  Managers can be better prepared for any auditing. If a  IE.CM.01: Perform internal control particular audited farm will be “RSPMF compliant”,  IE.CM.02: Perform farm accounting then this will increase the trust of auditors,   IE.CM.03: Perform maintenance of buildings Not only managers, but also other personnel working  IE.CM.04: Perform employments and other Human on farm can learn about processes, metrices and KPI’s. Resource issues 4.3 Draft list of domains and their processes  IE.CM.05: Perform product sales We already have defined a draft list of domains and their processes.  IE.CM.06: Perform purchases of equipment  IE.CM.07: Perform purchases of farming machinery Govern and Monitor (GM):   IE.CM.08: Perform purchases and implementation of GM.01: Define and maintain strategy  software products GM.02: Ensure profitability   IE.CM.09: Perform asset maintenance GM.03: Ensure risk governance   IE.CM.10: Perform purchases GM.04: Ensure machinery and equipment governance  Implement and Execute (IE) – LiveStock (LS): GM.05: Ensure IT and innovation governance   IE.LS.01: Perform animal feeding GM.06: Ensure compliance with legislation   IE.LS.02: Perform animal movements and grazing GM.07: Enable external and internal control   IE.LS.03: Preform animal health checking and health GM.08: Manage and monitor process definition and treatment change  IE.LS.04: Perform sales of animals 325  IE.LS.05: Perform purchasing of animals [5] L. Ruiz-Garcia and L. Lunadei, “The role of RFID in  IE.LS.06: Perform animal selection agriculture: Applications, limitations and challenges,”  IE.LS.07: Perform animal reproduction Comput. Electron. Agric. , vol. 79, no. 1, pp. 42–50, Oct. 2011. 4.4 Concepts of methodology to facilitate the [6] A. Kaloxylos et al. , “Farm management systems and the sustainability of RSPMF Future Internet era,” Comput. Electron. Agric. , vol. 89, no. COBIT was first issued in 1996, and this means that it has been null, pp. 130–144, Nov. 2012. going through evolution, where experts from the whole world participated. COBIT is now version 5, but had several versions [7] C. N. Verdouw, R. M. Robbemond, and J. Wolfert, “ERP in before that [3], [4]. agriculture: Lessons learned from Dutch horticulture,” Comput. Electron. Agric. , vol. 114, pp. 125–133, 2015. To facilitate the sustainability of RSPMF, we plan a similar approach. We have plan to issue the first version in a year or year [8] R. Rupnik, M. Kukar, P. Vračar, D. Košir, D. Pevec, and Z. and a half. The first version will cover only livestock. We will form Bosnić, “AgroDSS: A decision support system for an international panel of experts of various profiles: Consultants, agriculture and farming,” Comput. Electron. Agric. , no. academics, Product Managers, farmers and government officials. November 2017, 2018. 5. CONCLUSION AND FUTURE WORK [9] R. Nikkilä, I. Seilonen, and K. Koskinen, “Software architecture for farm management information systems in We have introduced the research in progress for the idea and precision agriculture,” Comput. Electron. Agric. , vol. 70, no. concepts of the Reference Standard Process Model for Farming. 2, pp. 328–336, Mar. 2010. Our aim of the design of reference model is to improve the support for managers and owners of bigger farms in farm management. [10] C. G. Sørensen et al. , “Conceptual model of a future farm Another aim is to facilitate Product Managers in development of management information system,” Comput. Electron. Agric. , software products and IoT systems. vol. 72, no. 1, pp. 37–47, Jun. 2010. In midterm, we also want RSPMF to be suitable for government [11] J. De Baerdemaeker, Precision Agriculture Technology and and EU officials who are responsible for farming. At the moment, Robotics for Good Agricultural Practices, vol. 46, no. 4. we plan to add the concept of maturity levels of a process. The IFAC, 2013. maturity level of a process will show or indicate the level of detail [12] J. Santa, M. A. Zamora-Izquierdo, A. J. Jara, and A. F. and expertise with which a farm executes a process. This way, the Gómez-Skarmeta, “Telematic platform for integral comparison of different farms will also be possible. management of agricultural/perishable goods in terrestrial We are aware that there are two phases of defining RSPMF: First, logistics,” Comput. Electron. Agric. , vol. 80, no. null, pp. 31– to define its concepts and structure; second, to put content in the 40, Jan. 2012. structure of processes` descriptions. Those two phases overlap, [13] J. W. Jones et al. , “Toward a new generation of agricultural because, while inserting the content, for sure some ideas to change system data, models, and knowledge products: State of structure will appear. The definition of concepts and the structure agricultural systems science,” Agric. Syst. , vol. 155, pp. 269– is our research mission for the next 12 months, that is how we plan 288, 2017. it. [14] J. W. Kruize, R. M. Robbemond, H. Scholten, J. Wolfert, and 6. REFERENCES a. J. M. Beulens, “Improving arable farm enterprise [1] A. Kaloxylos et al. , “A cloud-based Farm Management integration - Review of existing technologies and practices System: Architecture and implementation,” from a farmer’s perspective,” Comput. Comput. Electron. Agric. , vol. Electron. Agric. , vol. 100, pp. 168–179, Jan. 2014. 96, pp. 75–89, 2013. [2] S. Fountas et al. , “Farm management information systems: [15] M. Othman, M. Nazir Ahmad, A. Suliman, N. Habibah Current situation and future perspectives,” Arshad, and S. Maidin, “COBIT principles to govern flood Comput. management,” Electron. Agric. , vol. 115, pp. 40–50, 2015. Int. J. Disaster Risk Reduct. , vol. 9, 2014. [3] ISACA, COBIT 4.1. 2007. [16] M. Burnik, “The Approach for the Presentation of Nursing Processes,” University of Primorska, 2011. [4] ISACA, COBIT5: Enabling Processes. 2012. 326 Semiotics of graphical signs in BPMN Saša Kuhar Gregor Polančič Faculty of Electrical Engineering and Computer Science Faculty of Electrical Engineering and Computer Science University of Maribor University of Maribor Maribor, Slovenia Maribor, Slovenia sasa.kuhar@um.si gregor.polancic@um.si ABSTRACT RQ2: Can we categorize graphical signs from BPMN according The terminology of graphical signs (e.g. icons, symbols, to semiotic studies? pictograms, markers etc.) is ambiguous in academic articles. This We organized the remainder of the article as follows. The next is the same with articles focusing on graphics in business chapter presents the theoretical background. Chapters 3 and 4 notations, although concepts of graphical elements in notations represent the main objective of this paper – answering the are well defined. In semiotics, on the other hand, the concepts research questions. The conclusion is given in the last chapter. related to signs are defined in detail. In this paper, we examined linguistic terms that are used for describing graphical elements in 2. BACKGROUND BPMN specifications (BPMN being the de-facto Standard of business notations), and related them to the terminology specified 2.1 Semiotics in semiotics. We created a Sign ontology with BPMN graphical Semiotics is the study of signs and symbols (not only visual) and signs as ontology instances. The ontology can be used by their use or interpretation. For the purpose of the terminology researchers to share common knowledge about concepts of signs, definition, we will sum the book of Daniel Chandler Semiotics: symbols, icons, and indices, as well as the knowledge on BPMN The basics [3], which offers a comprehensive explanation of the graphical signs. field, including many views of modern theoreticians. There are two main traditions in contemporary semiotics: From Ferdinand Categories and Subject Descriptors de Saussure and Charles Sanders Peirce. H.1.m [Information Systems]: Models and Principles, Saussure's model of signs consists of two parts: Signifier (the Miscellaneous. form that the sign takes) and signified (the concept to which it refers). The sign is then the whole that results from the association General Terms of the signifier and the signified (Figure 1 on the left). For Management, Documentation, Design, Languages, Theory. Saussure, both signifier and signified take non-material form rather than substance. Nowadays, common adoption of his model Keywords takes a more materialistic form, where the signifier is commonly interpreted as the material that can be seen, heard, touched, Business Process Model and Notation, BPMN, Semiotics, smelled or tasted. Being concerned mostly with linguistics, Ontologies, Graphical signs, icons. Saussure stressed that the relationship between the signifier and the signified is relatively arbitrary: There is no inherent, essential, 1. INTRODUCTION transparent, self-evident or natural connection between the Business process diagrams provide a graphical notation for signifier and the signified – between the sound of a word and the specifying business processes. Among many business notations, concept to which it refers [3]. Business Process Model and Notation (BPMN) is known as the de-facto Standard [1]. BPMN consists of execution semantics and notation, the latter including graphical elements such as shapes, arrows, icons, and labels. Those elements are all signs where each has a defined meaning and represents a certain concept. However, terminology for graphical elements (e.g. icon, sign, or shape) is not used consistently among researchers in this domain. If one, for example, wants to perform a literature search on icons Figure 1: Saussure's model of signs on the left and Peircès in BPMN, the term icon does not incorporate all linguistic terms model of signs on the right that different authors use in their articles (other words for icon can Peirce, on the other hand, introduced a three-part model be pictogram, symbol, sign, marker etc). Even in BPMN consisting of: Representamen (the form which the sign takes, specifications [2], those terms are not used uniquely, but with also called “sign vehicle” or, in the Saussurean model, the loosely defined synonyms. signifier), interpretant (the sense made of the sign, or signified in With this situation in mind we formulated the following research Saussure's model). and object (something beyond the sign to questions: which it refers, also called the referent). In this model, the sign is the unity of what is represented (the object), how it is represented RQ1: What are the linguistic terms that are used in the BPMN (the representamen) and how it is interpreted (the interpretant) specification for graphical shapes, graphical icons, and other (Figure 1 on the right). The term sign is often used loosely and visual signs? 327 confused with signifier or representamen. However, the signifier and other visual signs?) we examined the BPMN specifications or representamen is the form in which the sign appears, whereas and mapped the specifications’ terms to semiotics’ terms. In the sign is the whole meaningful unity [3]. BPMN specifications the signs are denominated as follows: The term BPMN element represents the term signified, the terms 2.1.1 Symbol, Index, Icon shape, object, marker, indication, icon and depiction stand for In addition to his sign model, Peirce offered a classification of signifier. The answer to RQ1 and a detailed meaning of each signs, based on the relationship between representamen and its BPMN term is provided in Table 1. object or its interpretant, or, in Sausurres’ terms, the relationship Table 1: Linguistic terms used in BPMN specifications between signifier and signified. Dependent upon the relationship Semiotics’ BPMN Detailed meanings in BPMN being more arbitrary, directly connected, or more resembling, terms terms specification three types of signs are possible: Symbol, index, and icon Signified BPMN element Concepts in business notation respectively. Shape Graphical element SYMBOL represents a relationship where the signifier does not Basic shape (e.g. circle Object resemble the signified, but which is arbitrary or conventional. representing simple event) The relationship must be agreed upon and learned, such as in Signifier Marker, Graphical icon that can be language (letters, words, phrases, and sentences), numbers, Morse Indicator or included in an object (e.g. code, traffic lights or national flags. Icon message icon) Depiction Graphical example of the usage INDEX denotes a relationship where the signifier is not arbitrary, but connected directly (physically or causally) to the signified, As we can observe from the Table above, many linguistic terms which can be observed or inferred. An index indicates something are used for signifier, some of which are not used consistently (that is, necessarily, existent). Examples are natural signs (smoke, (e.g. marker, indicator, and icon). The only term from semiotics thunder, footprints), medical symptoms (pain, a rash, pulse-rate), that is used in BPMN specifications is the term icon, that is used measuring instruments (thermometer, clock), ‘signals’ (a knock on to denote a graphical icon and stands for the term signifier. a door, a phone ringing), recordings (a photograph, a film, video shot), personal ‘trademarks’ (handwriting, catchphrases). 4. ONTOLOGY CONSTRUCTION ICON represents a relationship where the signifier is perceived as For the purpose of Ontology construction and answering RQ2 resembling or imitating the signified – being similar in possessing (Can we categorize graphical signs from BPMN according to some of its qualities, like a portrait, a cartoon, a scale-model, semiotic studies?), we followed recommendations in Ontology onomatopoeia, metaphors, sound effects in radio drama, a dubbed Development 101: A Guide to Creating Your First Ontology [5]. film soundtrack and imitative gestures. [3] The authors suggest taking the following 7 steps for ontology creation: Step 1. Determine the domain and scope of the ontology, 2.1.2 Synonyms of terms Step 2. Consider reusing existing ontologies, Step 3. Enumerate The terminology from semiotics is used rarely in popular important terms in the ontology, Step 4. Define the classes and the language. The term sign in semiotics is frequently replaced by the class hierarchy, Step 5. Define the properties of classes, Step 6. term symbol in popular usage [3]. Also, several meanings of the Define the facets of the slots, and Step 7. Create instances. Steps 4 term icon can be found in everyday language: a) To be iconic and 5 are closely intertwined and can be executed simultaneously. means that something or someone is recognized as famous, b) In computing, an icon is a small image intended to signify a 4.1 Domain and scope of BPMN Sign particular function to the user (in semiotic terms these are signs ontology which may be iconic, symbolic or indexical), c) Religious icons For the domain definition, the authors [5] propose answering represent sacred, holy images [3]. If not stated otherwise, we will several questions. Our answers are provided below, after the continue to use terms as defined in semiotics throughout this paper. proposed questions. 2.2 Ontologies What is the domain that the BPMN Sign ontology will cover? Ontologies are explicit formal specifications of the terms in a Signs in BPMN domain and the relationships among them [4]. They define What are we are going to use the ontology for? common vocabulary and can, among other things, be used by To share a common understanding of knowledge about signs researchers, who need to understand and share the structure of among researchers, and to be able to reuse and analyze domain information in a domain [5]. Because of these reasons, we find knowledge. them appropriate for terminology clarification in the domain of For what types of questions should the information in the Graphical Signs in BPMN. Our research purpose is mainly ontology provide answers? definition of terms, so our ontology will, according to Obrst [6], Definitions of concepts in semiotics and relationships among be of the weak to moderately strong semantics, not intended to be them, categorization of BPMN graphical signs according to used for machine processing or machine interpretation (at least semiotics’ concepts, and the frequency of occurrence of sign types not at this stage of our research). in BPMN. 3. LINGUISTIC TERMS IN BPMN Who will use and maintain the ontology? The ontology will be maintained and used by us and will be SPECIFICATION available for other interested researchers. To answer the first RQ (What are the linguistic terms that are used To determine the scope of the ontology, a list of competency in the BPMN specification for graphical shapes, graphical icons, questions can be used that ontology will be able to answer [5]. 328 The competency questions we defined are listed next. Also, over time, a mode can change. Originally signs were in part  iconic, in part indexical (primitive writing), and symbols come What does the term icon mean?  into being by development out of other signs, particularly from How do icons, indices, and symbols correlate?  icons [3]. Which type of sign (icon, index or symbol) is used most in BPMN?  4.5 Sign Ontology construction Are symbols always arbitrary, or can they convey a certain degree of meaning? With the utilization of the Protégé 5.2.0 software tool and according to semiotic concepts and their relationships, we created 4.2 Reuse of existent ontology simple Sign Ontology as follows. We created a class Sign (with With a literature search we found no existing ontologies in the disjoint subclasses Icon, Index and Symbol), a class Relationship domain of signs or icons. However, we identified a Business (with subclasses PrimaryRelationship and SecondaryRelationship), Process Modelling Ontology (BPMO) that has been built and a class BPMNElement (with subclasses BasicShape, Activity, automatically, starting from the XML schemas contained in the Event, Gateway, and Data). We also created 2 object properties: BPMN 2.0 specifications from OMG [7]. It contains all the hasRelationshipType (with subproperties hasPrimaryRelationshipType BPMN elements and their relationships as defined BPMN and hasSecondaryRelationshipType), and its inverse property specifications. The class that is related most closely to our definesModeOf (with subproperties definesPrimaryModeOf and research domain (Graphical Signs) is DiagramElement and its definesSecondaryModeOf). The range of hasPrimaryRelationshipType subclasses (Figure 2). This class is, in BPMN specifications, is the class Sign, and the domain is the class PrimaryRelationship. defined under BPMN Diagram Interchange (BPMN DI) meta- We then defined 3 instances, Arbitrary, Indicative and Similar, model and schema for the purpose of the unambiguous rendering and included them in the classes PrimaryRelationship and of BPMN diagrams in different tools [2]. SecondaryRelationship. Next, we defined that, if a Sign has a hasPrimaryRelationshipType property of value Similar, it is included in the class Icon. Similarly, we defined classes Index (with hasPrimaryRelationshipType property value Indicative) and Symbol ( with hasPrimaryRelationshipType property value Arbitrary). 4.6 BPMN graphical shapes as Instances in Figure 2: DiagramElement class and its subclasses in BPMO, Sign Ontology visualized by the OntoGraf plugin for Protégé To decide whether graphical signs in BPMN are of the mode icon, As our focus in Sign Ontology is mainly on graphical signs that index or symbol, we invited 5 BPMN experts to evaluate BPMN are, as such, not contained in BPMO, we will start our own signs and define one sign mode for each. We chose BPMN ontology and, later, consider the options of merging both experts as they are fully familiar with the concepts (signifieds) in ontologies. BPMN. Before the evaluation, the experts were acquainted with concepts from semiotics. The results of the evaluation are given in Table 2. 4.3 Definition of concepts in Sign Ontology The next step in ontology creation is the enumeration of important On six shapes, the experts agreed on the sign mode, thus defining terms. We defined the concepts for BPMN sign ontology from the primary relationship between signifier and signified. For other semiotics (S ign, Icon, Index, and Symbol), and from BPMN shapes, where experts had different opinions, the mode was ( BasicShape, Activity, Event, Gateway, and Data). defined with the primary and the secondary relationship. The mode that was defined most often by experts was set for the 4.4 Relationships among concepts primary relationship, and the mode that ranked second in choices was set for the secondary relationship. For the definition of a hierarchy of classes and their properties, we will next define the relationships among three types of signs, As we can observe from Table 2, the majority of the signs were again from semiotics. specified as symbols (the primary relationship is arbitrary). 6 symbols were also the only signs where experts agreed fully on At first sight, the relationship among the signifier and the the sign mode. Furthermore, in all but one symbols, the secondary signified (and, consequently, the types of signs) seems mode was set as an index, and, similarly, the other way around; in unambiguous, but that is not always the case. We should keep in all indices, the secondary mode was set as a symbol. The mind that signs denote concepts (not material objects), and each consensus on the primary relationship was not possible for two person has their own understanding of a certain concept in his or signs (Script task and Data object), and on the secondary her mind. Concepts cannot be represented precisely [8] therefore relationship for one sign (Manual task). Thus, for the Script task icons, for example, cannot be denoted simply as similar. They are and the Data object, the primary relationship was not set, but two defined by perceived similarity [3]. Also, as stated in [9], the secondary relationships were set. For Manual task only the process of sign-making is the process of the constitution of primary relationship was set. metaphor, and, therefore, symbols are never only arbitrary. Within each type, signs vary in their degree of conventionality. After the modes of signs were defined we included the signs into Therefore, we must not speak of types of signs but of modes of Sign Ontology. The ontology, including the instances, is shown in relationships where the difference between signs lays in the Figure 3. The figure represents classes as circles and relationships hierarchy of their properties rather than in the properties as lines connecting the circles. The size of the circle corresponds themselves [3]. to the number of instances included in the class. 329 Table 2: Modes of BPMN signs Figure 3: Sign Ontology with BPMN shapes rendered in the NavigOwl plugin for ProtégéCONCLUSION * Signifier Signified Secondary relationship Primary relationship: Arbitrary (Symbol) In this paper, we mapped the linguistic terms from semiotics to linguistic terms regarding signs in BPMN specifications. We Activity found that, in BPMN specifications, many terms are used for the Gateway term signifier, some of which inconsistently. 5 Signal event To correlate concepts from semiotics to BPMN graphical signs, Multiple event we developed the BPMN Sign Ontology based on definitions Ad-hoc sub-process from semiotics. We categorized each BPMN graphical sign in a Complex gateway mode that represents the relationship between signifier and signified. The majority of the BPMN signs are of mode symbol, Event Indicative (index) following by mode index. As the meaning of symbols needs to be Parallel event Indicative (index) learned, this indicates a possible correlation with the principle of Escalation event Indicative (index) Semantic transparency from [10]. Addressing this issue, we will, 4 Link event Indicative (index) in future work, examine our results further with those from [11] Service task Indicative (index) and other related articles. Inclusive gateway Indicative (index) Since the current study included only 5 experts in BPMN, Parallel gateway Indicative (index) resulting in possible bias, empirical research with more users is Error event Indicative (index) planned, as well as a thorough literature search. At this point, the Send task Indicative (index) BPMN Sign Ontology can, in the BPMN domain, serve for unambiguous knowledge definition and sharing. Receive task Indicative (index) 3 Business rule task Indicative (index) 5. REFERENCES Sub-process Indicative (index) [1] M. Kocbek, G. Jošt, M. Heričko, and G. Polančič, Exclusive gateway Indicative (index) “Business process model and notation: The current state of Data object collection Similar (icon) affairs,” Comput. Sci. Inf. Syst. , vol. 12, no. 2, pp. 509–539, Primary relationship: Indicative (Index) 2015. Conditional event Arbitrary (symbol) [2] O.M.G., “Business Process Modeling Notation.” 2011. [3] D. Chandler and E. W. B. Hess-Lüttich, Semiotics the 4 Flow Arbitrary (symbol) Basics, Second Edi., vol. 35, no. 6. London: Routledge, Cancel event Arbitrary (symbol) 2007. Data store Arbitrary (symbol) 3 [4] T. R. Gruber, “A translation approach to portable ontology Compensation event Arbitrary (symbol) specifications,” Knowl. Acquis. , vol. 5, no. 2, pp. 199–220, Primary relationship: Similar (Icon) Jun. 1993. 4 Message event Indicative (index) [5] N. F. Noy and D. L. McGuinness, “Ontology Development Timer event Indicative (index) 101: A Guide to Creating Your First Ontology,” Standford 3 User task Arbitrary (symbol) Knowl. Syst. Lab. Tech. Rep. , pp. 1–25, 2001. Manual task Not set [6] L. Obrst, H. Liu, R. Wray, and L. Wilson, “Ontologies for Primary relationship: Not set semantically interoperable electronic commerce,” IFIP Adv. Inf. Commun. Technol. , vol. 108, pp. 325–333, 2003. Script task Similar/indicative (2*) [7] L. Cabral, B. Norton, J. Domingue, L. C. Kmi, B. Norton, Data object Similar/arbitrary (2*) and J. Domingue, “The business process modelling * - The number of experts who decided on this primary mode ontology,” Proc. 4th Int. Work. Semant. Bus. Process Manag. , pp. 9–16, 2009. [8] A. Fenk, “Symbols and icons in diagrammatic representation,” Pragmat. Cogn. , vol. 6, no. 1–2, pp. 301– 334, 1998. [9] G. R. Kress and T. van Leeuwen, Reading Images (The Grammar of Visual Design). London: Routledge, 1996. [10] D. Moody, “The physics of notations: Toward a scientific basis for constructing visual notations in software engineering,” IEEE Trans. Softw. Eng. , vol. 35, no. 6, pp. 756–779, 2009. [11] N. Genon, P. Heymans, and D. Amyot, “Analysing the Cognitive Effectiveness of the BPMN 2.0 Visual Notation,” in Journal of Visual Languages & Computing, vol. 22, no. 6, 2011, pp. 377–396. 330 Knowledge Perception influenced by Notation Used for Conceptual Database Design Aida Kamišalić Muhamed Turkanović Marjan Heričko Faculty of Electrical Faculty of Electrical Faculty of Electrical Engineering and Computer Engineering and Computer Engineering and Computer Science Science Science University of Maribor University of Maribor University of Maribor Maribor, Slovenia Maribor, Slovenia Maribor, Slovenia aida.kamisalic@um.si muhamed.turkanovic@um.si marjan.hericko@um.si Tatjana Welzer Faculty of Electrical Engineering and Computer Science University of Maribor Maribor, Slovenia tatjana.welzer@um.si ABSTRACT there are no researches that dealt with knowledge percep- The paper presents an experimental study which examined tion within the databases learning environment. In order the influence of the notation used for conceptual design on to examine the effectiveness of learning database fundamen- students’ knowledge perception at higher educational study tals, depending on the notation used for conceptual design, level. The results demonstrate that students’ knowledge per- we set-up a multi-level experimental study [7]. Different ception is higher than actual knowledge throughout the en- experimental instruments to evaluate the effectiveness of a tire learning process and is correlated with the used nota- teaching approach using Barker or Bachman notation for tion. conceptual database design were developed. In contrast to Barker notation, Bachman notation incorporates elements Categories and Subject Descriptors of logical design (i.e. foreign keys) in the conceptual design H.2.1 [Database Management]: Logical Design; K.3.2 level. Students’ achievements were examined with regard [Computers and Education]: Computer and Information to influencing factors throughout the learning process. Re- Science Education sults indicated that introducing the Bachman notation and a manual transformation from a conceptual into a logical data model increased students’ understanding of conceptual, General Terms logical and relational data model concepts (CLR concepts). Theory, Experimentation Here we present another aspect of this study. The influence of notation used for conceptual design on student knowl- Keywords edge perception is examined. Research questions that are entity relationship models, conceptual design, database de- addressed and answered in the paper are (RQ1) How does sign learning, Barker, Bachman, knowledge perception the notation used for conceptual design influence students’ knowledge perception? and (RQ2) Does the correlation be- 1. INTRODUCTION tween student’s knowledge perception and actual knowledge The relational databases are fundamental part of any infor- about CLR concepts change throughout the learning pro- mation system. Conceptual and logical design represent im- cess? portant segment of almost every application. Therefore, dif- The structure of the paper is as follows. In Section 2, a ferent issues related to teaching approaches of database fun- methodological framework and experimental setting are pro- damentals and design must be adequately addressed. The vided. The main contribution of the paper is presented in introduction to databases course is one of the fundamen- Section 3 where results and discussion are detailed. Finally, tals of computer science and/or informatics higher educa- the conclusions are presented in Section 4. tion programs. It is mostly a single semester course that covers data requirements elicitation, conceptual database 2. METHODOLOGY design, normalization, logical database design, and physical database design [3, 4, 5]. There is much research addressing 2.1 Experimental framework issues related to teaching computer science and informatics The study was carried out during the academic year 2016/2017 disciplines including various aspects of databases [3, 5, 9], at the Faculty of Electrical Engineering and Computer Sci- some research has dealt with the effectiveness of teaching ence at the University of Maribor. The experiment was per- approaches to database design (conceptual and logical mod- formed within the Database I course. It is a single semester eling) [1, 2, 7, 8]. However, to the best of our knowledge course that includes 45 hours of theory/practice lectures and 331 30 hours of laboratory work in the form of computer exer- participants have to model an entity (i.e. person) and give cises. it some attributes and possibly a primary key. For the sec- ond task (9b), the participants have to model an additional The focus of the experiment was on the evaluation of stu- entity (i.e. phones) and present an 1:N relationship between dents’ laboratory work. Students were randomly split into the previous entity and the newly added one. For the third two approximately equal size groups. Both groups worked task (9c), the participants had to add a third entity (i.e. ad- on the same database modeling tasks, using the Oracle SQL dress), and correctly use a form of M:N relationship between Developer Data Modeler design tool. One of the groups used the previous entities and the newly added one. In order to Bachman notation which explicitly includes the foreign key be able to analyze the results, five concepts are evaluated: in the E-R diagram, while another group used the Barker entity, relationship, attribute, PK and FK. The scoring is notation, which does not explicitly include the foreign key as follows: if they used any possible form of the concept in in the E-R diagram [6]. their solution and if the presented use of the concept was correct, participants got a point for the concept. Thus, five 2.2 Experimental instruments points could be scored in total. In this section, a detailed presentation of the experimental instruments used during the study is given. The question- 3. RESULTS AND DISCUSSION naire was conducted twice: Intro-Questionnaire and Final- In the next sections we report on the results achieved in the Questionnaire. The participation was optional in both oc- experiment. Statistical analyses were performed using IBM currences. The questionnaire used in the study is available SPSS Statistics version 23. on the web (http://bit.ly/2wMvrVQ). 3.1 Knowledge perception The questionnaire is split into three parts. The first part An analysis was performed on related samples of the per- consists of mainly closed-ended questions related to basic ception score and test score. It was based on data gathered demographic information and database design tools (Ques- from the Intro-Questionnaire and Final-Questionnaire. The tions 1 - 6). The second part consists of a Likert scale-like data for each questionnaire was analyzed separately. multi-level table (Question 8), where participants have to cross one of the multi-level options for five basic database In the analysis we excluded all those records where students terms and concepts: Entity, Relationship, Attribute, Pri- rated one of the concepts as undefined, thus the total num- mary Key (hereinafter PK) and Foreign Key (hereinafter ber of records taken into account were 116. Therefore, we FK). The values of the Likert scale are as follows: (1) - I am got four levels of knowledge and five different concepts. As not familiar with the term, (2) - I am familiar with the term, mentioned in the previous section, part of the questionnaires but not with the meaning, (3) - Undefined, (4) - I am famil- was a short test. We will refer to the total test score as iar with the meaning but I do not know how to use it and (5) the test score. In order to effectively compare the actual - I am familiar with the meaning and I know how to use it. knowledge with the perception, we normalized the results The third part included open-ended questions, given in the of knowledge perception so that the total score (max. 20 form of a short test (Question 9). The short test consists of points) was divided by five. We will refer to the normalized three consecutively simple tasks, whereby each is related to perception results as the perception score. Table 1 reports the previous and each presents an increase in difficulty. In on the results of the analysis which was performed using a order to solve the test correctly, the participants have to use Wilcoxon signed-rank test for related samples. a form of one-to-many (hereinafter 1:N) and many-to-many (hereinafter M:N) relationship. The participants should not be given any instructions on how to solve the test. They Table 1: Correlation of results for perception score should be left to use any means and techniques that seem and test score. appropriate. The foreseen time limit is 20 minutes. Asymp. Experimental Related Sig. N Decision instrument Samples The purpose of the questionnaire was to examine if there (2-tailed) was any correlation between the participant’s perception of Intro- Percep. score Reject the 0.000** 107 knowledge of CLR concepts (Question 8) and their actual Questionnaire - Test score null hypothesis knowledge (score on the test questions 9a, 9b, 9c). When Final- Percep. score Reject the 0.000** 116 the questionnaire was handed out the second time during the Questionnaire - Test score null hypothesis experiment, an additional closed-ended question was added **Significant at 1% to the first part (Question 7), whereby students were asked which notation they used during the laboratory work. The purpose of this particular question was to examine if there We used the Wilcoxon signed-rank test in order to compare was any correlation between the notation used during the two not normally distributed sets of scores, one actual score laboratory work and their knowledge (score on the test ques- and another normalized perception score, that came from tions 9a, 9b, 9c). the same participants, since each participant had to solve tasks and evaluate their knowledge on the aforementioned In order to evaluate the questionnaire a scoring structure CLR topics. The Shapiro-Wilk test of normality indicated for the third part is needed (Question 9). The test con- that data significantly deviates from a normal distribution sists of three consecutive tasks (9a, 9b, 9c), whereby each (p-value below 0.05). The Wilcoxon signed-rank test returns relates to the previous and each constitutes an increase in an asymptotic significance lower than 0.01, thus rejecting difficulty. In order to solve the first task (9a) correctly, the the null hypothesis for related samples. The null hypothesis 332 states that the median of difference between the perception participant reached the opposite result, while 15 assessed score and the test score will equal zero. There is a statisti- their knowledge correctly. A further indication of wrong cally significant difference between the perception score and knowledge perception can be deduced from the mean of the the test score, suggesting that students’ perception of their scored results. The mean of the perception score during knowledge is not in accordance with their actual knowledge the Intro-Questionnaire is 4.095, while the mean for the test on CLR concepts. Figures 1 and 2 depict the correlation be- score stood at 1.74. In addition, the means for the Final- tween students’ actual knowledge and their knowledge per- Questionnaire were 4.957 and 3.35, respectively. We con- ception, which indicates a higher knowledge perception than clude that students overestimated their knowledge of CLR the actual knowledge in both questionnaires. The results concepts throughout the entire course. indicate that the correlation between the knowledge percep- tion and actual knowledge is corrected by the end of the course (Final-Questionnaire), which is due to higher knowl- Table 2: Cases of knowledge perception scores ver- edge achieved by the end of the course. However, the knowl- sus actual knowledge scores. edge perception remains at a high level. Experimental Related Mean Sum of N instrument Samples Rank Ranks Negative 2 a 17.25 34.5 Ranks Intro- Percep. score Positive Questionnaire - Test score 100 b 52.19 5218.5 Ranks Ties 5 c Total 107 Negative 1 a 1 1 Ranks Final- Percep. score Positive Questionnaire - Test score 100 b 51.5 5150 Ranks Ties 15 c Total 116 a Perception score Test score c Perception score = Test score Figure 1: Correlation between students’ actual Conclusions regarding RQ2: Students overestimated their knowledge and their knowledge perception. Intro- knowledge of CLR concepts throughout the entire course. Questionnaire (course start). The correlation between the students’ knowledge perception and actual knowledge is corrected by the end of the course, due to higher knowledge reached by the end of the course. However, the knowledge perception remains at a high level. 3.2 Knowledge perception and notation Additionally, we analyzed the results of the students’ knowl- edge perception and actual knowledge considering the nota- tion used in the learning process. Normalized results of stu- dents’ self-assessment of their knowledge and results of our assessment of their knowledge was summarized and used to assess the students’ perception of knowledge in terms of the dependence of the notation. The range of the summed score is thus 1 - 10. As the summed score approaches the extremes, the students were better able to assess their knowledge. It means that their perception of their knowledge and their ac- tual knowledge were very close. On the contrary, the closer Figure 2: Correlation between students’ actual the results were to the middle, the more students incorrectly knowledge and their knowledge perception. Final- assessed their knowledge. It means that they either overes- Questionnaire (course end). timated or underestimated it. For example students could assess their knowledge as high and reach five points for the perception and also score all five points on the test, thus Table 2 reports on the ranks of the performed Wilcoxon collecting ten points. On the contrary, students could assess signed-rank test. There were 100 out of 107 participants at their knowledge as high, but reach a minimum or even none the Intro-Questionnaire who assessed their knowledge higher points on the test, thus scoring five points in total. The than their actual knowledge was. On the contrary, only two analysis of the impact of the notation was based on the data participants reached the opposite results and only five as- gathered from the Final-Questionnaire only, because the im- sessed their knowledge correctly. The Final-Questionnaire pact of the notation can only be seen after the notation was results showed a slight increase in correctly assessed knowl- used in the learning process. Table 3 reports on the re- edge. There were 100 out of 116 participants at the Final- sults of the Mann-Whitney U test for independent samples. Questionnaire who assessed their knowledge as being higher We used the Mann-Whitney U test in order to compare dif- than their actual knowledge was. On the contrary, only one ferences between two independent groups (students using 333 4. CONCLUSIONS Table 3: Correlation of summed perception and test The paper reported on the results of an experimental study score and influencing factor (notation used). aimed at analyzing the influence of notation used for the Exper. Independ. Depend. Asymp. N Decision conceptual design on students’ knowledge perception. The instr. variable variable Sig. study continues on the work already presented in [7], while Summed reporting on students’ knowledge perception being higher Final- perc. Notation 116 0.008** Reject the Quest. and test null hypothesis than the actual knowledge. score *Significant at 5%; **Significant at 1% We examined whether students’ perception of knowledge is in accordance with their actual knowledge of CLR concepts. The results confirm that their perception is higher than the Bachman or Barker notation) and the dependent variable actual knowledge throughout the entire learning process. By (students’ summarized test score and normalized perception the end, their knowledge increases and perception remains score), while the groups are not normally distributed. The at a similar level as at the beginning. Additionally, the re- Shapiro-Wilk test of normality indicated that data signifi- sults prove that students who used the Bachman notation cantly deviates from a normal distribution (p-value below during the learning process were able to better estimate their 0.05). knowledge. In the future we plan to analyze the correlation between students’ educational background and their success The Mann-Whitney U test returns an asymptotic signifi- rate while learning the CLR concepts on the higher educa- cance lower than 0.01 for the notation variable, therefore tion degree level. rejecting the related null hypothesis. The null hypothesis states that the distribution of the summed score is the same 5. ACKNOWLEDGMENTS across categories of both Bachman and Barker notations. Considering the results, there is a statistically significant dif- The authors acknowledge the financial support from the ference between the summed results scored by notation used Slovenian Research Agency (Research Core Funding No. P2- in the learning process. There were 68 out of 116 students 0057). who used the Barker notation during the learning process, and their summed mean score stood at 8.1. The Bachman 6. REFERENCES notation was used by 48 students, whereby their summed [1] A. Al-Shamailh. An Experimental Comparison of ER mean score was 8.608. According to Figure 3, it is evident and UML Class Diagrams. International Journal of that there were more students who used the Bachman nota- Hybrid Information Technology, 8(2):279–288, 2015. tion and better assessed their knowledge. [2] H. C. Chan, K. K. Wei, and K. L. Siau. Conceptual level versus logical level user-database interaction. In Figure 3 depicts the correlation between the summed per- ICIS 45. Proceedings, pages 29–40, 1991. ception score and test score, and the notation used during [3] T. M. Connolly and C. E. Begg. A the learning process. Constructivist-Based Approach to Teaching Database Analysis and Design. Journal of Information Systems Education, pages 43–53, 2005. [4] R. Dargie and A. Steele. Teaching Database Concepts using Spatial Data Types. In In Proceedings of the 4th annual conference of Computing and Information Technology Research and Education New Zealand, pages 17–21, 2013. [5] C. Dom´ınguez and A. Jaime. Database design learning: A project-based approach organized through a course management system. Computers & Education, 55(3):1312–1320, 2010. Figure 3: Summed perception score and test score in [6] D. C. Hay. A comparison of data modeling techniques. correlation with the notation used during the learn- Essential Strategies, Inc, pages 1–52, 1999. ing process. [7] A. Kamišalić, M. Heričko, T. Welzer, and M. Turkanović. Experimental Study on the Effectiveness of a Teaching Approach Using Barker or On the contrary, there were students who used the Barker Bachman Notation for Conceptual Database Design. notation with a summed score of five which indicates the Computer Science and Information Systems, worst assessment of knowledge. We conclude that students 15(2):421–448, 2018. who used the Bachman notation in the learning process bet- [8] H. C. Purchase, R. Welland, M. McGill, and ter evaluated their knowledge than students who used the L. Colpoys. Comprehension of diagram syntax: an Barker notation. empirical study of entity relationship notations. International Journal of Human-Computer Studies, Conclusions regarding RQ1: Bachman notation posi- 61(2):187–203, 2004. tively influences students’ ability of knowledge self-assessment. [9] S. D. Urban and S. W. Dietrich. Integrating the By the course’s end, the difference between knowledge per- Practical Use of a Database Product into a Theoretical ception and actual knowledge lowers. Curriculum. SIGCSE Bull., 29(1):121–125, 1997. 334 The Use of Standard Questionnaires for Evaluating the Usability of Gamification Alen Rajšp Katja Kous Tina Beranič Faculty of Electrical Faculty of Electrical Faculty of Electrical Engineering and Computer Engineering and Computer Engineering and Computer Science Science Science University of Maribor University of Maribor University of Maribor Maribor, Slovenia Maribor, Slovenia Maribor, Slovenia alen.rajsp@um.si katja.kous@um.si tina.beranic@um.si ABSTRACT the standardised definitions of usability. Fernandez et al. [7] Usability has a significant impact on the satisfaction and found that 59% of the reviewed papers reported end-user- frequency of use of a designed system. Nowadays, gami- based usability testing, while 35% of the reviewed papers fication and serious game approaches are implemented in used the inquiry methods (such as focus group, interviews, software solutions to increase their usability. We present a questionnaires and surveys). Based on these facts, this re- literature review of 32 identified studies measuring usability, search focuses on inquiry methods, more specifically on tech- with established questionnaires in gamified systems and se- nique questionnaires, and investigates which standard ques- rious games. We identified 18 different questionnaires used tionnaires are used most commonly for usability evaluation for measuring usability, and found System Usability Scale in The Gamification domain. Within the presented paper, to be the most widely used. An immense issue exists in the we focus on the research question: Which standard question- field, with only 22% of studies measuring usability actually naires are used for evaluating the usability of gamification? describing or defining what usability is. Using a literature review, we study the use and popularity of established usability questionnaires in the Gamification Categories and Subject Descriptors domain. H.5.2 [User Interfaces]: User-centered design A similar study, made by Yᘠnez-Gómez et al. [18], presents General Terms the review of academic methods for usability evaluation of serious games. The scope of the study is broader, aiming at Measurement, Experimentation, Standardization, Theory, finding the preferred approach for evaluating the usability Verification of games. As the results show, standard questionnaires are the second most used technique applied in post-use analysis Keywords [18]. They mention three questionnaires in use, but detailed Usability Evaluation Method, Formal Questionnaires, Gam- analysis is not provided. Also, in comparison to the pre- ification, Serious Games sented study, our search string differs. Another review is presented by Calderón and Ruiz [5], also covering the do- 1. INTRODUCTION main of Serious Games’ Evaluation. One of the research In recent years, gamification has become an essential part questions concerned evaluation techniques, and discovered of varieties of domains, from Education to Medicine. It is that questionnaires are the most commonly used, but the used for facilitating the use of developed products. They categorization or detailed analysis of the used questionnaires cannot achieve its purpose if the usability of the product is was not provided. inadequate. Therefore usability evaluation should present a crucial step of development. The paper is structured as follows. We start by presenting the research background covering usability evaluation and Solutions utilising (1) gamification or (2) serious game ap- gamification, we continue by presenting and discussing the proach should be evaluated separately, due to them being results of the literature review. We close our paper by pre- inspired by games which have very specific (and different) senting the conclusions reached by our review. natures. The primary function of games is to entertain through experience whereas serious games and gamification have some intended useful purpose [10]. Because gamifi- 2. USABILITY EVALUATION cation and serious games’ approaches utilise elements from The term usability represents a combination of several prop- games, this leads to solutions where even other needs of so- erties and attributes [13]. Regardless of the variety of def- lutions intended for an audience are being met to varying initions by different authors [1, 3, 9, 13, 15, 17], Jeng [12] degrees. In solutions this causes an increase of user satisfac- states that Nielsen and ISO 9241-11 definitions are the most tion. widely cited. ISO 9241-11 defines usability as “the extent to which a product can be used by a specified user to achieve In the web area of expertise, only 18% of the reviewed pa- specified goals with effectiveness, efficiency and satisfaction pers in [7] present usability evaluation methods relying on in a specified context of use” [11], while Nielsen [15] defines 335 usability as an aggregation of five attributes: Learnability, efficiency, memorability, errors and satisfaction. The usability evaluation method is defined as “a procedure, composed of a set of activities for collecting usage data re- lated to end user interaction with a software product, and/or how the specific properties of this software product con- tribute to achieving a certain degree of usability” [7]. Ac- cording to Battleson et al. [2], the usability evaluation meth- ods are classified into three categories: (1) Inquiry methods (such as focus group, interviews, questionnaires and sur- veys), (2) Formal usability testing (such as interactions with a website by performing tasks) and (3) Inspection methods (such as heuristic evaluation, cognitive walk-through, plu- ralistic walk-through and formal inspection). The first two categories involve real-users, while inspection methods are Figure 1: Primary studies by years based on reviewing the usability aspects of web artifacts, which have to comply with established guidelines, and are performed by expert evaluators or designers [7]. from selected primary studies showed that only seven pri- mary studies (22%) defined and described the term of us- 3. GAMIFICATION ability. Two of them indicated usability as a concept (S5, S10), while five researches treated usability as construct, Gamification is the use of design elements characteristic for namely two studies (S11, S21) used Nielsen’s definition, one games in non-game contexts [6]. Gamification should not be research (S4) used the ISO definition, one research (S18) de- confused with serious games. Whereas the goal of introduc- scribed usability as “ease of use of the game”, while study ing gamification is influencing learning related behaviuors S25 defined usability similar to ISO, but expanded the defi- and attitudes without providing knowledge, the use of seri- nition with two new concepts (“simple” and “operating with ous games should influence learning and provide knowledge ease”). The remaining studies (78%) used the term usability by the experience itself [14]. Another way to compare gam- without providing the meaning of usability. ification and serious games is that gamification represents using only parts (game elements) from games, while serious Studies are classified by domain in Table 1. Over half (56%) games represent the whole immense gaming experience [6]. of all studies were from the field of Health and Medicine. Most of the studies from the domain addressed (1) Training 4. EVALUATING THE USABILITY OF of health care personnel (S8, S17, S18), (2) Rehabilitation GAMIFICATION and exercise for patients (S3, S6, S7, S16) and (3) Assessing patients (S1). The second most popular domain (37%) was 4.1 Research Education and Learning. All other identified domains had Our research aims to find available standard questionnaires only 1 study per domain. used for evaluating the usability of gamification. Using the following search string ”usability” AND (”gamification” OR Domain Primary studies ”serious games” OR ”educational games”) we conducted a Agriculture S27 search in the following digital libraries: ScienceDirect, IEEE Xplore, ACM Digital Library and Sage journals. Deter- Business Intelligence S5 mined inclusion and exclusion criteria guided the study se- Computer Science S5 lection process. We considered the papers evaluating us- Education & Learning S2, S8, S10, S13, S14, S16, S17, ability with the help of established and well-known ques- S18, S23, S28, S29, S31 tionnaires. Therefore, we excluded primary studies using Entertainment S4 ad-hoc questionnaires. Health & Medicine S1, S3, S6, S7, S8, S11, S12, S16, S17, S18, S19, S20, S21, After the review process, we selected 33 primary studies. S22, S24, S29, S30, S32 The list of primary studies we used as input into the data ex- Social Science S25 traction and data synthesis step is available at: https://tiny Task Management S9 url.com/CSS2018-IJS. 26 out of 33 primary studies are con- Travel S15 ference papers, whereas seven papers are journal articles. Figure 1 shows the number of primary studies by year of Table 1: Domain publishing. We selected 23 primary studies from the IEEE Xplore digital library, six from the ACM Digital Library, We continued the data extraction by identifying standard three from ScienceDirect and one from Sage journals. questionnaires used for usability evaluation. We followed the explanation provided by Yᘠnez-Gómez et al. [18], which 4.2 Results states that standard questionnaires are the ones that are Within data extraction, we focused on two main areas. First, validated statistically. Table 2 presents used questionnaires we searched for used definitions of usability, since the lat- in connection with primary studies. The majority of stud- ter was evaluated in the analysed studies. Extracted data ies evaluated usability by using the System Usability Scale 336 (SUS). It was used in 78% of primary studies. Although most established questionnaire SUS for measuring usability, Technology Acceptance Model (TAM) is used in the model- but did not define the measured attribute in their research. driven analysis for measurement of users’ acceptance and us- Table 3 presents the connection between the used question- age of technology and it is not classified as a standard ques- naires and measured attributes that were measured at least tionnaire for usability evaluation, it was used for assessment in two primary studies. The most frequently measured at- of gamification in four primary studies. On the other hand, tributes were ”ease of use” and ”usability” and both were Game Experience Questionnaire (GEQ), Task Load index used in six primary studies. In all cases, the attribute ”us- (TLX), Game Engagement Questionnaire (GEQ), Post-Study ability”, was measured with SUS, while the attribute ”ease System Usability Questionnaire (PSSUQ) and Net Promoter of use” was measured with three different questionnaires: Score (NPS) are each used in two primary studies. We ex- SUMI (S7), USE (S9) and TAM (S2, S3, S11, S20). The tracted other questionnaires that are used only in one pri- second most frequent measured attribute was attribute ”use- mary study, such as Presence Questionnaire (PQ) and Soft- fulness”. In three primary studies (S2, S11, S20), it was ware Usability Measurement Inventory (SUMI). To achieve treated and determined as one of the two factors defined in TAM, while, in one case, it was measured with USE (S9) and Questionnaire Primary studies PSSUQ (S15). The attribute ”satisfaction” was the third most commonly used attribute measured by two different System Usability Scale S1, S3, S6, S7, S10, S11, (SUS) S12, S14, S16, S17, S18, questionnaires: SUS (S21, S22, S31) and USE (S9). S19, S20, S21, S22, S23, S24, S25, S26, S27, S28, Measured attribute Questionnaires S29, S30, S31, S32 Ease of use SUMI (S7), USE (S9), Technology Acceptance S2, S3, S11, S20 TAM (S2, S3, S11, S20) Model (TAM) Usability SUS (S10, S11, S16, S23, Game Experience Ques- S1, S4, S30 tionnaire (GEQ) S24, S31) Task Load index (TLX) S1, S22 Usefulness TAM (S2,S11,S20), USE Game Engagement S11, S18 (S9), PSSUQ (S15) Questionnaire (GEQ) Satisfaction USE (S9), SUS Post-Study System Us- S8, S15 (S21, S22, S31) ability Questionnaire Flow GEQ (S1, S4, S11) (PSSUQ) Learnability SUMI (S7), USE (S9) Net Promoter Score S31-S32 Competence GEQ (S1, S4) (NPS) Overall CSUQ (S13), SUMI (S7) User Engagement Scale S5 (UES) Quality of Information CSUQ (S13), PSSUQ (S15) Computer System Us- S13 Quality of interface CSUQ (S13), PSSUQ (S15) ability Questionnaire (CSUQ) Table 3: Connection between the measured at- Software Usability Mea- S7 tributes and used questionnaires surement Inventory (SUMI) The most popular devices on which developed/proposed so- Intrinsic Motivation In- S16 lutions were run were computers (62%), virtual reality equip- ventory (IMI) ment (22%) and mobile devices (16%) as seen in Table 4. User Interaction Satis- S18 faction (QUIS) Device Primary studies Presence Questionnaire S10 (PQ) Computer S1, S2, S4, S5, S6, S7, S8, S10, Usefulness, Satisfaction, S9 S11, S13, S14, S16, S21, S22, and Ease of use (USE) S23, S25, S26, S28, S29, S31 Questionnaire Customised system S19 Pick-A-Mood (PAM) S10 Mobile device S9, S12, S15, S20, S27 Technology Affinity S20 Smart TV S3 - Electronic Devices Virtual reality S10, S15, S17, S18, S24, S30, S32 (TA-ED) Questionnaire Game User Experience S19 and Satisfaction Scale Table 4: Devices on which the studied system runs (GUESS) Differential Emotions S10 4.3 Discussion Scale (DES) An extensive collection of standard questionnaires were found for evaluating the usability of gamification, with System Us- Table 2: Standard questionnaires in use ability Scale (SUS) as the prevailing choice (84% of all stud- ies). Since SUS is a well-known questionnaire, which is easy a comprehensive usability evaluation, it is crucial that mea- to perform and analyse, this is not a surprise. As SUS was surement instruments used are utilised appropriately accord- developed for providing a subjective assessment of usability ing to the attribute they are measuring. 41% (13/32) of [4], its extensive use is even more understandable. The ma- primary studies (S6, S12, S17-S19, S25-S30, S32) used the jority of researchers that used SUS in their studies did not 337 quote explicitly which attribute of usability was measured; Kaufmann, San Francisco, 2002. the remaining studies, where the SUS were used, defined two [4] J. Brooke. Sus: A quick and dirty usability scale, 1996. different attributes that can be measured with SUS. The [5] A. Calderón and M. Ruiz. A systematic literature first attribute was ”usability” and it is in accordance with review on serious games evaluation: An application to description of SUS usage purpose [4], while the second one software project management. Computers & was ”satisfaction”, which is recommended by the ISO/TS Education, 87:396–422, 2015. 20282-2:2013 [8] Standard, where the SUS is defined as a [6] S. Deterding, D. Dixon, R. Khaled, and L. Nacke. questionnaire for measuring satisfaction. From game design elements to gamefulness: Defining gamification. Proceedings of the 15th International Another aspect is also if standard usability questionnaires Academic MindTrek Conference on Envisioning Future can evaluate the usability of gamification adequately. She- Media Environments - MindTrek ’11, pages 9–11, gawa et al. [16] claims that the SUS questionnaire is a veri- 2011. fied instrument for measuring usability in the Serious Games [7] A. Fernandez, E. Insfran, and S. Abrah˜ ao. Usability domain. Technology Acceptance Model (TAM) is widely evaluation methods for the web: A systematic used in the Information System domain to investigate how mapping study. Inf. Softw. Technol., 53(8):789–817, accepted the use of technology is among their target users. Aug 2011. Although it is not classified as a standard questionnaire for [8] I. O. for Standardization. ISO/TS 20282-2:2013 usability evaluation, but rather as a model combining con- Usability of consumer products and products for structs ease of use and usefulness, it was the second most public use - Part 2: Summative test method, 2013. used measuring instrument for usability evaluation in re- [9] E. Furtado, J. J. V. Furtado, F. Lincoln Mattos, and viewed literature. On the other hand, it is also seen that J. Vanderdonckt. Improving usability of an online questionnaires, like Game Experience Questionnaire (GEQ) learning system by means of multimedia, and Game Engagement Questionnaire (GEQ), that originate collaboration, and adaptation resources. In Usability from Gaming domain, are nowadays used to evaluate the us- Eval. Online Learn. Programs, pages 69–86, October ability of gamification. Therefore, the fusion of two fields is 2003. perceived. [10] C. Girard, J. Ecalle, and A. Magnan. Serious games as new educational tools: how effective are they? A 5. CONCLUSION meta-analysis of recent studies. Journal of Computer The paper presents conducted literature review which was Assisted Learning, 29(3):207–219, 2013. aimed at finding standard questionnaires used for usabil- [11] ISO. Standard 9241: Ergonomic Requirements for ity evaluation of gamification and serious games. We found Office Work with Visual Display Terminals (VDT)s, that the majority (84%) of studies evaluate usability using Part 11. Guidance on Usability. 1998. a System Usability Scale (SUS), though some other ques- [12] J. Jeng. What is usability in the context of the digiral tionnaires were also detected and used independently, or in library and how can it be measured? Information combination with SUS. We, as prospective researchers, can Technology and Libraries, 24(2):47–56, Nov 2005. determine only in a minority of cases what primary studies [13] Z. Kılıç Delice, Elif Güngör. The usability analysis were measuring, because only 22% of primary studies mea- with heuristic evaluation and analytic hierarchy suring usability defined or described what usability is. That process. Int. J. Ind. Ergon., 39(6):934–939, Nov 2009. is an immense issue on validity of their measurements of us- [14] R. N. Landers. Developing a Theory of Gamified ability, since multiple definitions of it exist. We propose that Learning: Linking Serious Games and Gamification of methods for measuring usability in the field of Gamification Learning. Simulation & Gaming, 45(6):752–768, 2014. and Serious Games should be formalised in the future. Al- though researchers are already using standardised methods [15] J. Nielsen. Usability Engineering. Academic Press, San for measuring usability, research should also present what Diego, 1993. usability means for them, what they are measuring. [16] R. Shewaga, A. Uribe-Quevedo, B. Kapralos, K. Lee, and F. Alam. A Serious Game for Anesthesia-Based Crisis Resource Management Training. Entertainment 6. ACKNOWLEDGMENTS Computing, 16(2):6:1–6:16, apr 2018. The authors acknowledge the financial support from the [17] G. Tsakonas and C. Papatheodorou. Exploring Slovenian Research Agency (Research Core Funding No. P2- usefulness and usability in the evaluation of open 0057). access digital libraries. Information Processing & Management, 44(3):1234–1250, May 2008. 7. REFERENCES [18] R. Yᘠnez-Gómez, D. Cascado-Caballero, J.-L. J.-L. [1] A. Abran, A. Khelifi, W. Suryn, and A. Seffah. Sevillano, R. Yanez-Gomez, D. Cascado-Caballero, Usability meanings and interpretations in iso and J.-L. J.-L. Sevillano. Academic methods for standards. Information and Software Technology., usability evaluation of serious games: a systematic 11(4):325–338, Aug 2003. review. Multimedia Tools and Applications, [2] B. Battleson, A. Booth, and J. Weintrop. Usability 76(4):5755–5784, Feb 2017. testing of an academic library web site: A case study use of academic library web. J. Acad. Librariansh., 27(3):325–338, 2001. [3] T. Brinck, D. Gergle, and S. D. Wood. Designing Web Sites that Work: Usability for the Web. Morgan 338 Analyzing Short Text Jokes from Online sources with Machine Learning Approaches Samo Šimenko Vili Podgorelec Sašo Karakatič Faculty of Electrical Engineering and Faculty of Electrical Engineering and Faculty of Electrical Engineering and Computer Science Computer Science Computer Science University of Maribor University of Maribor University of Maribor Maribor, Slovenia Maribor, Slovenia Maribor, Slovenia samo.simenko@student.um.si vili.podgorelec@um.si saso.karakatic@um.si ABSTRACT jokes from the online sources. Third section presents the individual This paper presents the whole data mining process of analyzing steps of data- and text- mining in details. It consists of machine jokes in Slovenian language gathered from various online sources. learning method description, applications and techniques used in The gathering was done with the help of web scrapping system and the process and the results itself. We finish up with the conclusion the analysis was carried out on the gathered jokes to determine the and the discussion on the topic of joke analysis with various data properties of various types of jokes. In addition, with the help of mining methods. various text-mining methods, we analyzed different types of jokes 2. GATHERING AND PARSING OF THE and built a machine learning model for classifying jokes into categories. These results are supplemented with the visualization of JOKES FROM THE ONLINE SOURCES different categories and the interpretation of constructed machine In order to fulfill the set goals of analyzing jokes, we obtained these learning classification models. from various sources. Three different sources were used: – Categories and Subject Descriptors From the first source, a web site called VERZIVICI [2], joker already classified into categories; H.4 [Information Systems Applications]: Miscellaneous; – Jokes from the second source NAJVICI [3]; I.2.m [Artificial Intelligence]: Miscellaneous; – Jokes from a third source MLADINSKI [4]. General Terms For the data acquisition we developed a program in the Visual Machine Learning, Data Mining, MDS, SVC Studio IDE, using the C# programming langauge, which acquired jokes from the selected sources and saved them in a suitable text Keywords format. Due to the unstructured data of selected web resources, we Data mining, Machine Learning, Joke analysis, Short text analysis, used HAP (HTMLagilityPack) for processing. HAP is a HTML Text mining parser written in C# for reading/writing the DOM (Document Object Model) and supports plain XPATH or XSLT [1]. Using the 1. INTRODUCTION HAP library and XPATH, we could easily access individual Due to the ever-advancing technology, opportunities are sections, which contained content known as a “joke”. opening for analyzing all types of data, so we can make the most of this and use it for our benefit. By studying and examining various Jokes from VERZIVICI, which were categorized when gathered, types of texts, scientists were already involved in the initial phases were manually entered, since the program for collecting jokes from of textual analysis [8, 9, 10], but studying the meaning and different categories used the category name in the creation of a URL, which is used for scrolling between categories. For connection of texts presents a rather new direction of research, where there is still a lot of room for improvement. While there has NAJVICI, we manually created a URL for gathering jokes so we been a lot of work done on various short text types, i.e. tweets [12], can easily access all jokes on the site. reviews [11], recipes [13] and others, there is a lack of research On the website MLADINSKI, jokes were already grouped and the published on the topic of jokes analysis. jokes were sequentially recorded on one side of the web page. For the purpose of processing and subsequent manipulation, a simple In our paper, we present a process of gathering, parsing and VIC class was created, which contains two textual attributes of Text pre-processing jokes and applying various data- and text-mining and Category. Both attributes can store values in string format, Text techniques to extract patterns and new knowledge from jokes data. attribute is for raw text of a joke, Category is for type of category By semantic text processing, we identify more than just a sequence in which joke is categorized. When we were capturing blank of symbols, we can assign them meaning, which can influence the spaces, we encountered redundant badges before text and between classification of jokes. In our case, we undertook the processing of texts. Also, unreadable machine records were created instead of various jokes that we analyzed in order to determine how the symbols due to coding. All badges with associated symbols and categories of such texts are interconnected by their content and find non-nominal groups of words, which were created instead of out which categories of jokes share the most similar content. Based symbols, were manually entered into the program and then on the texts, we created a classification model for the classification programmatically removed. of jokes into predefined categories. As a result of obtaining and processing the data from the selected The rest of the paper is structured in the following way. The sources, we received the data, which are used as the basis below: following section presents the method for gathering and parsing – VERZIVICI [2] – 13 categories, a total of 1729 jokes, 339 – NAJVICI [3] – a total of 297 jokes, and classification is a supervised machine learning method, which – means that machine learns to classify jokes from the already solved MLADINSKI [4] – a total of 145 jokes. (classified) examples [15]. We have saved the acquired data in the CSV format. Due to the characteristics of the CSV format, the comma symbol "," was There are numerous different classification algorithms [18], but for our case we used the Support Vector Machine (SVM) classifier, changed to the XX symbol, addressed below, because comma in CSV represents a separator between lines, in jokes commas can developed by Vapnik in 2000 [16]. This method learns the have different meaning. All of the jokes were in Slovenian boundaries that separate instances (jokes in our case) from one language, so this had to be taken into consideration during the text category to another, by finding a linear separation border called hyper-plane that has a maximum distance from the entire instance analysis. set, which is called the maximum margin. The instances that are 3. DATA ANALYSIS closest on the hyper-plane (on the hyper-plane itself) are called In this section, we will present the methods and techniques for support vectors. This SVM method also uses a kernel trick [19], analyzing the jokes and the results of these analysis. The whole which maps the attribute space of the classification instance to a process of cleaning, preprocessing, and the analysis itself was done higher dimensional space. In our case, we used a linear kernel, with the Python programming language, and its libraries. which uses a liner function to transform the attributes in such a way, that the margin of the hyper-plane is maximized. 3.1 Cleaning and preprocessing the data We used the implementation of SVM from the library liblinear [20], As mentioned, we use the Python programming language to process which has high flexibility in the choice of penalties and loss data in which you can simply import information in a CSV format functions and should scale to large numbers of samples. This library using the Pandas library [5]. Pandas is an open source, BSD- supports both dense and sparse input and the multiclass support is licensed library providing high-performance, easy-to-use data handled according to a one-vs-the-rest scheme [6]. structures and data analysis tools for the Python programming language [5, 14]. The imported data is then appropriately structured Upon preliminary data preparation, the whole joke dataset is using the DataFrame class with the following columns (attributes): divided into train and test sets, where the training set is used to build – the SVM classification model, and the test set is used to test the Index, quality of the model – the ability to correctly classify yet unseen – Category, and jokes. In our experiment, we applied stratified sampling to split the – RawText. data and used 60% of data for training test and the rest 40% for the test set. The results of the experiment show, that the resulting The XX symbols are also removed and replaced with the comma classification model classifies test jokes with 61% accuracy. The symbol ",". From the text, we also removed stop-words, which is a classifier has correctly classified more than half of jokes into their list of common words that do not carry any semantic meaning and proper category out of 13 possible categories. information. Stop words occurred in texts in high frequency but are of little significance and consequently uninteresting. A sample of The default classification of instances in one of 13 categories would stop words in Slovenian language are the following: result in only 0.08 accuracy, so our resulting classifier improves the default classifier significantly. This represents a high percentage of “in” (En. and), “ali” (En. or), precision as was not foreseen at first glance. Additionally, we also ”je”(En. is), ”za” (En. for), manually examined some of the jokes that were misclassified. ”to” (En. this), ”na” (En. on), Interestingly, although the predicted categories were not correct, ”to” (En. this), ”ti” (En. you), several of the examined jokes would fit well into the predicted ”ko” (En. when), ”bi” (En. would), category as well, as the semantics of a joke is not always ”ne” (En. no), ”da” (En. yes), monolithic. ”že” (En. already), ”le” (En. only). In addition, the punctuations were removed, so the resulting text 3.3 Word frequency analysis and was in the form of one sentence without most common stop words. visualization From the resulting text, we built a representation of every joke in From the dataset of jokes, with attributes of individual word’s tf- the format appropriate for the analysis. We used the method of idf scores, we built word cloud diagrams for every category of the counting the frequency of individual words called word frequency. joke. The word clouds were made with the help of libraries This number was normalized by the word frequency of the word in matplotlib [21] and wordcloud for the Python programming all categories, so the more common words got the lower score and language. In the word cloud, the most common words (or rather the less common and maybe unique words got higher score. This those with higher tf-idf scores) are written in larger font, while process is called tf-idf (term frequency-inverse document those with lower frequency (lower tf-idf scores) are written in frequency) and is a common word scoring method in text mining smaller font. The color of the words only serves to make words [17]. The new dataset was built in such way, that all of the identified more differentiable and thus improves the readability of word words represented one attribute of the joke, and the corresponding clouds. value of that attribute is the tf-idf score of that word in that joke. Also, these word cloud show which highly informative words (non- stop words) are common for each category and can be used for 3.2 Classification of jokes in the manual classification, this way we can check whether a joke, which predetermined categories reads: “pride nekega dne k janezkovemu očetu domov nek njegov We used the classification machine learning technique in order to nadležen prijatelj tone potrka vpraša dober dan oče doma janezek construct a model of classification that would learn how to classify tone ja kje janezek vem grem vprašat” was appropriately classified yet unseen jokes to one of the predetermined categories. This can into a category (the original category is called “janezek”, “Solski” be useful if one would want to automate joke categorization on an was predicted). As we can see in Figure 1, our model correctly online joke portal without any need for human intervention. The decided to classify the joke in the category “Solski”, because the 340 word “janezek” prevails in this category and is the dominant word expressions, which are more commonly used in foreign jokes as in the content of the joke. well as older jokes. Figure 1: Hierarchical clustering of joke categories. 3.5 Multidimensional scaling Multidimensional scaling (MDS) enables the visualization of the level of similarity of individual cases of a dataset by lowering the number of different attributes to only two. It refers to a set of related ordination techniques used in information visualization, in particular to display the information contained in a distance matrix [7]. By using the MDS in the sklearn.manifold[23] library and the mpl_toolkits.mplot3d[24] library, we can observe relations between categories even more efficiently, as shown in a 2D graph in the Figure 3. This plot shows which categories are closer together and which categories differ the most. Contrary to the dendrogram, we can see that “Mujo in Haso” are not so close to “Ciganski” and “Stari vici”, but these three categories differ the most from the rest. Figure 1: Word-clouds for ten joke categories. 3.4 Hierarchy of the categories With the help of the scipy [22] Python library, we also built a dendrogram of relations between the categories using a hierarchical clustering method, which is shown in the Figure 2. Here we also included the category from sources NAJVICI and MLADINSKI, so that we can visually display the content linkage between different categories. The dendrogram is a hierarchical diagram, which shows which terms (in our case joke categories) are closer together by putting the more similar categories closer together on the Y-axis. The more similar are the categories, shorter are the lines connecting these categories, and vice versa. From the dendrogram we can see that the categories MLADINSKI (En. young ones) and SOLSKI (En. School ones) are most similar, since the school is usually visited by young people. Based on the names of the categories NAJVICI and Mesane sale (En. Random jokes), it can also be assumed that these categories are very similar. Figure 2: 2D Multidimensional scaling plot, which shows the From the dendrogram we can also see that groups of categories similarity of different joke categories marked by red and green connections are very different. We can conclude that this division can be attributed primarily to slang 341 This shows the seclusion of three categories (a group of categories [3] http://www.naj-vici.com, Last visited: 5.8.2018 marked in a dendrogram with red color, which includes Stari Vici, [4] http://www.mladinska.com/, Last visited: 5.8.2018 Mujo in Haso and Ciganski) in relation to other categories. These make up a kind of circle around the categories “NAJVICI” and [5] https://pandas.pydata.org, Last visited: 13.8.2018 “Mesana Sale”. Categories “NAJVICI” and “Mesana Sale” are the [6] http://scikit-learn.org/stable/modules/generated/sklearn.svm.Li closest neighbors, which also suggests an exceptional similarity nearSVC.html, Last visited: 13.8.2018 between the categories.With the help of Figure 3, we can see the [7] https://en.wikipedia.org/wiki/Multidimensional_scaling, relationship between categories even better; in the case of the categories “Moski” and “Zenske”, we can see that according to Last visited: 20.8.2018 their content, these two are very similar categories. [8] Song, G., Ye, Y., Du, X., Huang, X. and Bie, S., 2014. Short As depicted in the Figure 4 is a 3D graph of relations for use in text classification: A survey. Journal of Multimedia, 9(5), p.635. further discussions for the show. By using the 3D graph (Graph 4), [9] Chen, M., Jin, X. and Shen, D., 2011, July. Short text we can even more accurately determine the differences between the classification improved by learning multi-granularity topics. In categories of texts. This display mode can turn out to be even more IJCAI (pp. 1776-1781). useful in a larger number of data and when looking for interesting patterns in these texts. [10] Sriram, B., Fuhry, D., Demir, E., Ferhatosmanoglu, H. and Demirbas, M., 2010, July. Short text classification in twitter to improve information filtering. In Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval (pp. 841-842). ACM. [11] Dave, K., Lawrence, S. and Pennock, D.M., 2003, May. Mining the peanut gallery: Opinion extraction and semantic classification of product reviews. In Proceedings of the 12th international conference on World Wide Web (pp. 519-528). ACM. [12] Wang, Y., Liu, J., Qu, J., Huang, Y., Chen, J. and Feng, X., 2014, December. Hashtag graph based topic model for tweet mining. In Data Mining (ICDM), 2014 IEEE International Conference on (pp. 1025-1030). IEEE. [13] Badra, F., Bendaoud, R., Bentebibel, R., Champin, P.A., Cojan, J., Cordier, A., Després, S., Jean-Daubias, S., Lieber, J., Meilender, T. and Mille, A., 2008, September. Taaable: Text mining, ontology engineering, and hierarchical classification for Figure 3: 3D Multidimensional scaling plot, which shows the textual case-based cooking. In 9th European Conference on Case- similarity of different joke categories Based Reasoning-ECCBR 2008, Workshop Proceedings (pp. 219- 4. CONCLUSION 228). This paper presents a use case of machine learning methods in the [14] McKinney, W., 2012. Python for data analysis: Data analysis of short texts in a form of jokes. We presented the process wrangling with Pandas, NumPy, and IPython. " O'Reilly Media, of gathering, cleaning and preprocessing the jokes, which was Inc.". followed by the description of the analysis done with machine [15] Friedman, J., Hastie, T. and Tibshirani, R., 2001. The elements learning methods and various visualization techniques. We of statistical learning (Vol. 1, No. 10). New York, NY, USA:: demonstrated how jokes could be automatically categorized in the Springer series in statistics. predefined categories using the Support Vector Machine classification method. With two different visualizations: the [16] Vapnik, V. and Mukherjee, S., 2000. Support vector method dendrogram and the multidimensional scaling plot, we showed how for multivariate density estimation. In Advances in neural different joke categories are similar one to another. With these information processing systems (pp. 659-665). methods, we demonstrated, how we could perform different [17] Aizawa, A., 2003. An information-theoretic perspective of tf– comparisons, which can serve us in the further processing of data, idf measures. Information Processing & Management, 39(1), and the connection of data between us is visualized in a useful and pp.45-65. interesting way. [18]http://en.wikipedia.org/wiki/Category:Classification_algorith In this paper, we only analyzed the jokes in Slovenian language. ms, Last visited 13.9.2018 For future work, we could compare jokes in different languages to find similarities and differences of jokes and their popularity across [19] https://en.wikipedia.org/wiki/Support_vector_machine Last different languages and cultures. visited 13.9.2018 [20] https://www.csie.ntu.edu.tw/~cjlin/liblinear/, Last visited ACKNOWLEDGMENTS 13.9.2018 The authors acknowledge the financial support from the Slovenian [21] https://matplotlib.org/, Last visited 13.9.2018 Research Agency (research core funding No. P2-0057). [22] https://www.scipy.org/, Last visited 13.9.2018 REFERENCES [23] http://scikit-learn.org/stable/modules/generated/sklearn.manif [1] http://html-agility-pack.net, Last visited: 20.8.2018 old.MDS.html, Last visited 13.9.2018 [2] http://www.verzi-vici.com, Last visited: 5.8.2018 [24] https://matplotlib.org/2.0.2/mpl_toolkits/mplot3d/api.html 342 A Data Science Approach to the Analysis of Food Recipes Tjaša Heričko Sašo Karakatič Vili Podgorelec Faculty of Electrical Engineering and Faculty of Electrical Engineering and Faculty of Electrical Engineering and Computer Science Computer Science, Computer Science, University of Maribor, FERI University of Maribor University of Maribor Maribor, Slovenia Maribor, Slovenia Maribor, Slovenia tjasa.hericko@student.um.si saso.karakatic@um.si vili.podgorelec@um.si ABSTRACT from recipes needed to perform well at cuisine prediction. (3) In this paper, we explore the correlation between cuisine and text- Enhancing cuisine prediction. based information in recipes. The experiments are conducted on a This paper is organized as follows. Section 2 gives a brief real dataset consisting of 9,080 recipes with data science overview of related work. Section 3 presents the dataset used in approaches focusing on enhancing cuisine prediction and our research. Section 4 describes the applied methodologies. providing a detailed insight on the characterization of food Section 5 provides results of our research. Section 6 concludes the cultures. The findings suggest that information about ingredients paper by summarizing the main results of our work. is the most relevant predictor of cuisines, however, despite being less efficient, recipe name, preparation instructions, preparation 2. RELATED WORK time, skill level and nutritional facts can be considered as well. The correlation between recipes and their cuisines has been the subject of several recipe analysis related research. Mostly, there Categories and Subject Descriptors have been previous studies conducted on classifying recipes into I.2.m [Artificial Intelligence]: Miscellaneous. respective cuisines based on ingredients. H. Su et. al. [1] I.5.m [Pattern Recognition]: Miscellaneous. evaluated data collected from Food2 and used the techniques of associative classification and support vector machine to classify General Terms 226,025 recipes to one of six cuisines, using ingredients as inputs, Algorithms, Measurement, Experimentation. with a precision and recall of about 75 %. The researchers in [2– 8] further studied cuisine-ingredient connection, using 39,774 Keywords recipes from twenty cuisines provided by Yummly3. Similar Data Science, Machine Learning, Text mining, Classification, studies were conducted on data from Epicurious [9], Epicurious Food Recipes, Cuisines. and Menupan4 [10] and Food, Epicurious5 and Yummly [11]. A variety of machine learning algorithms, including k-means [2, 9], 1. INTRODUCTION random forest classifier [2, 5, 6, 8, 9, 10], support vector machine In response to technological advancements and social changes in [3, 5, 6, 7, 10, 11], logistic regression [4, 5, 6, 10, 11] and naive the last decades, the tendency to collect and store recipes only in Bayes [5, 6, 7, 9, 10, 11], were used in these studies. From several cookbooks has changed. Numerous online recipe portals started to tested algorithms, linear support vector machine, reaching up to rapidly accumulate food-related content, with more and more 80,9 % accuracy in [7], was found to be the most efficient for this recipes being published online daily. The growth in the amount of cuisine prediction task based on ingredients. user-generated recipe data available on the Internet has raised Other studies focused on the importance of other information several issues that researchers have been trying to address in extracted from recipes for cuisine prediction. H. Kicherer et. al. recent years. The objective of this paper is to explore the [12] evaluated the use of ingredients and preparation instructions correlation between cuisine and text-based information in recipes, for cuisine prediction, conducted on recipes from German website including recipe name, list of ingredients, preparation Chefkoch6. The study revealed that ingredients alone are as good instructions, preparation time, skill level, calories and nutritional an indicator as the recipe instructions. Whereas a combination of information. The results of this study address the issue of information from both – nouns from the instructions and the list automatic recipe cuisine categorization, making it easier to submit of ingredients – performs better. T. Ozaki et. al. [13] also a new recipe and preventing possible additional noise in recipe demonstrated that, based on Japanese recipes from Cookpad database – this can be helpful for both the contributors as well as Data7, certain sets of ingredients and preparation actions deeply for the culinary website curators. correspond to cuisine types. We conducted a series of experiments on a real dataset retrieved Previous studies have already noted that ingredients reveal from BBC Good Food1 consisting of 9,080 recipes from various important information about cuisines and that predicting cuisines cuisines with data science approaches focusing on the following: (1) Providing a detailed insight on the characterization of various 2 https://www.food.com/ food cultures. (2) Identifying necessary text-based information 3 https://www.yummly.com/ 4 https://www.menupan.com/ 5 https://www.epicurious.com/ 6 https://www.chefkoch.de/ 1 https://www.bbcgoodfood.com/ 7 https://cookpad.com/ 343 based on the ingredients is possible. Though, to our knowledge, 4.1 Data Preprocessing few researchers have considered using additional text-based For the dataset to be feasible for the analysis, preprocessing was information from recipes, for instance, preparation instructions, performed on the raw scraped data. preparation time and nutrition facts, as possible attributes in cuisine prediction. Therefore, there is little understanding of how During the data cleaning step, missing values and duplicates were they are related to cuisine types. In contrast to the work presented resolved by removing these recipes from the original dataset, above, we performed a richer analysis of recipes with a wider leaving a subset of 9,080 recipes. range of attributes extracted from recipes, whereas the dominant The original dataset included 45 cuisine categories, many of them approach appears to deal only with ingredients as attributes. only consisted of few recipes. In the next step of data preparation, 3. DATASET based on the findings of previous researches of cuisines being location-dependent [14], we combined smaller cuisines into Our research was conducted on the crawled data collected from an bigger regional cuisine categories (e.g. Balinese, Thai, online food recipe portal BBC Good Food. A dataset of 9,429 Vietnamese and Indonesian into Southeast Asian cuisine) and recipes was scraped with Python8, using Scrapy framework9 and therefore reduced cuisine categories to the following 13: African, CSS selectors, in June 2018. Middle Eastern, South Asian, Southeast Asian, East Asian, For each recipe, the following information was provided: recipe Oceanic, American, Latin American, Western European, Northern name, cuisine, list of ingredients, preparation instructions, European, Central European, Eastern European, Mediterranean. preparation time, skill level and nutrition facts, including the As highlighted in Table 1, preparation time and nutrition facts are amount of calories, total fat, saturated fat, total carbohydrate, numerical, cuisine and skill level are categorical, whereas recipe sugars, protein, fiber and salt per serving. More details are name, list of ingredients and preparation instructions are presented in Table 1. described in natural language. For all of them, additional Table 1. Characteristics of text-based information in recipe preprocessing was needed prior to conducting analyses. Numerical attributes were standardized, considering certain Information Data Type Description algorithms used in our research are sensitive to varied number Arbitrary string described in Recipe name Unstructured scales and intervals used [15]. As scikit-learn algorithms only natural language. work on numerical data, categorical data needed to be encoded as Cuisine Categorical One of 45 cuisine types. numerical. This was done by converting categorical data into Arbitrary string depicting dummy variables [16]. For unstructured data to be used for needed ingredients for classification, several more text preprocessing methods were List of preparation, each ingredient needed: tokenization, stop word removal, stemming and tf–idf Unstructured ingredients normally consisting of an term weighting. Tokenization is the process of segmenting a text ingredient type, an amount and into identifiable basic linguistic units called tokens, such as words a unit. and punctuation [17]. For better processing, all tokens were Step-by-step instructions for Preparation converted to lowercase. Stop words are frequently used common Unstructured preparation using ingredients instructions words, such as ‘and’, ‘the’ and ‘this’. Because their presence in a described in natural language. text fails to distinguish it from other texts and are therefore not A number representing time Preparation Numerical measured in minutes needed useful in classifications, they were removed before further time for preparation. processing [18]. We also made a custom list of stop words, where One of 3 difficulties: easy, we included numbers that represent amounts and words that Skill level Categorical more effort or a challenge. represent units, e.g. ‘2’ and ‘tbs’, that would not be of value in the A number representing analysis. The same applies to punctuation, therefore they were nutrition per serving measured filtered out as well. Next, stemming using the Porter stemming in kcal for calories intake or in Nutrition facts Numerical algorithm, the process of removing morphological affixes from grams for fat, saturated fat, words, which conflate variant forms of a word into a unified carbohydrate, sugars, protein, representation [19], was performed. Lastly, for words counts fiber and salt. being suitable for usage by a classifier, tf–idf transform was 4. METHODOLOGY conducted. Tf–idf, short for term-frequency times inverse The methodology in this paper was implemented in Jupyter document-frequency, is used to re-weight a words importance notebook environment10 running Python code and using a based on a frequency of a world in a document compared to the combination of Python libraries comprising pandas11, scikit- appearance in other documents [20]. learn12, NLTK13, seaborn14, matplotlib15 and wordcloud16. 4.2 Exploratory Data Analysis To get an overall view of the data, exploratory data analysis was made on preprocessed data using graphs, word clouds and tables. 8 https://www.python.org/ Visualization was especially used to provide clarity on the 9 https://scrapy.org/ characterization of various cuisines. 10 http://jupyter.org/ 11 https://pandas.pydata.org/ 4.3 Classification 12 http://scikit-learn.org/ Various classification algorithms were used to perform the cuisine 13 https://www.nltk.org/ prediction based on the information from the recipes. The recipe 14 https://seaborn.pydata.org/ 15 dataset was randomly divided into training (75 %) and testing set https://matplotlib.org/ 16 http://amueller.github.io/word_cloud/ 344 (25 %). The training set was used to train, while the test set was To give an idea of the ingredients that form an integral part of used to assess models. each cuisine, we extracted the most common ingredients in every cuisine and visualized unigrams from the ingredient list in word 4.3.1 Naive Bayes clouds. As detailed in Table 2, many ingredients are frequent in Naive Bayes is based on applying Bayes’ theorem with the naïve all the cuisines, e.g. oil and onion, hence, these will not be useful independence assumption between every pair of features. for prediction. While others are typically used only in certain Gaussian naive Bayes assumes the probability of features is cuisines, e.g. soya sauce and clove. Gaussian. Multinomial naive Bayes implements the algorithm to Figure 1 represents word clouds consisted of the most common the usage for text classification [21]. unigrams extracted from the ingredient list for East Asian cuisine. Although most common ingredients did not give us much insight, 4.3.2 Support Vector Machine these word clouds do show some typical ingredients, based on A linear support vector machine constructs a hyper-plane or set of which they can be distinguished from other cuisines, e.g. sugar, hyper-planes in a high or infinite dimensional space using linear flour, milk, cream, chocolate, egg, mayonnaise, butter in algebra [22]. American cuisine and soy sauce, rice, ginger, soy, chili in East Asian cuisine. 4.4 Evaluation Metrics To measure classification performance the following metrics were used: accuracy and F-score. Accuracy is the percentage of correct predictions. F-score is a weighted average of the precision and recall, where precision represents the ability of the classifier not to label as positive a sample that is negative and recall the ability of the classifier to find all the positive samples [23]. 5. RESULTS As an initial step, we carried out an exploratory data analysis to get a better understanding of cuisines and their characteristics. Figure 1. Word cloud for East Asian cuisine Table 2 lists average preparation time and calories per serving for each cuisine. Given the analysis, recipes from Northern Europe, Middle East and Western Europe take the longest to prepare, Cuisines also differ on nutrition facts. In Figure 2, for every whereas recipes from East Asia, Latin America and Southeast cuisine an average value of each nutrition per serving is presented. Asia are generally the quickest to prepare. Furthermore, on average, Mediterranean, Oceanic and American cuisines are high in energy, on the contrary, Southeast Asian, East Asian and South Asian have recipes with lower energy values. Table 2. Overview of the cuisines Average Average Cuisine Common ingredients preparation calories time [min] [kcal] Oil, onion, lemon, African 51,73 399,68 clove, coriander. Oil, onion, tomato, Middle Eastern 76,67 409,11 garlic, clove. Onion, oil, coriander, South Asian 53,74 367,50 chili, clove. Sauce, lime, chili, oil, Southeast Asian 45,00 350,78 sugar. Sauce, oil, onion, chili, East Asian 40,49 363,18 rice. Oceanic Sugar, oil, egg. 60,36 430,70 Sugar, butter, oil, flour, American 57,68 422,37 egg. Onion, oil, chili, Latin American 43,17 399,30 coriander, lime. Western Sugar, oil, butter, egg, 66,59 394,85 European flour. Northern Oil, sugar, onion, egg, 119,59 374,61 European cream. Central Sugar, butter, egg, 62,73 402,85 European flour, oil. Eastern Oil, butter, egg, flour, Figure 2. Nutrition facts for cuisines 57,96 390,04 European garlic. In the next step, classification algorithms were applied to identify Oil, garlic, clove, Mediterranean 48,68 433,36 which text-based information from recipes is needed to perform tomato, onion. 345 well at cuisine prediction. A classification with multinomial naive Accessed on: August 16, 2018.R. Ghewari, and S. Raiyani, Bayes, based on the list of ingredients, proved to be the most “Predicting Cuisine from Ingredients.” [Online]. Available: efficient. This model yielded an accuracy of 73,8 %. Less than 1 http://cseweb.ucsd.edu/~jmcauley/cse255/reports/fa15/029.pdf. % lower was the accuracy obtained with classification based on Accessed on: August 16, 2018. [6] S. Kalajdziski, G. Radevski, I. Ivanoska, K. Trivodaliev, and B. R. recipe name and more than 2 % based on preparation instructions. Stojkoska, “Cuisine classification using recipes ingredients,” 2018 Classifications based on skill level, preparation time, calories and 41st International Convention on Information and Communication nutritional information all performed with an accuracy of about 56 Technology, Electronics a nd Microelectronics (MIPRO), 2018. %. Classification performance based on accuracy and F-score are [7] R. M. R. V. Kumar, M. A. Kumar, and K. P. Soman, “Cuisine summarized in Table 3. Prediction based on Ingredients using Tree Boosting Algorithms,” Indian Journal of Science and Technology, vol. 9, no. Table 3. Results of classification 45, Aug. 2016. Information Classifier Accuracy F-score [8] T. Arffa, R. Lim, and J. Rachleff, “Learning to cook: An exploration of recipe data.” [Online]. Available: Multinomial naive Recipe name 72,73 % 72,73 % https://pdfs.semanticscholar.org/3f63/269aa7910774e9386b1ffb340 Bayes a9e8638c02d.pdf. Accessed on: August 16, 2018. List of Multinomial naive 73,83 % 73,83 % [9] J. Naik, and V. Polamreddi, “Cuisine Classification and Recipe ingredients Bayes Generation,” 2015. [Online]. Available: Preparation Multinomial naive 70,97 % 70,97 % https://pdfs.semanticscholar.org/aaa9/67ce597961bad308ec137a616 instructions Bayes 9e1aba1fe35.pdf. Accessed on: August 16, 2018. Preparation Gaussian naive Bayes 55,29 % 55,29 % [10] S. Jayaraman, T. Choudhury, and P. Kumar, “Analysis of time Linear SVM 55,68 % 55,68 % classification models based on cuisine prediction using machine learning,” 2017 International Conference On Smart Technologies Skill level Linear SVM 56,12 % 56,12 % For Smart Nation (SmartTechCon), pp. 1485–1490, 2017. [11] H. Kicherer, M. Dittrich, L. Grebe, C. Scheible, and R. Klinger, Gaussian naive Bayes 55,68 % 55,68 % Calories “What you use, not what you do: Automatic classification and Linear SVM 55,68 % 55,68 % similarity detection of recipes,” Data & Knowledge Engineering, Nutritional Gaussian naive Bayes 53,48 % 53,48 % 2018. information Linear SVM 57,00 % 57,00 % [12] T. Ozaki, X. Gao, and M. Mizutani, “Extraction of Characteristic Sets of Ingredients and Cooking Actions on Cuisine Type,” 2017 31st International Conference on Advanced Information 6. CONCLUSION Networking and Applications Workshops (WAINA), pp. 509–513, Thousands of recipes from various cuisines were analyzed with 2017. [13] K. J. Kim, and C. H. Chung, “Tell Me What You Eat, and I Will Tell data science approaches with the objective of providing a deeper You Where You Come From: A Data Science Approach for Global understanding of culinary cultures and cuisine prediction. While Recipe Data on the Web,” IEEE Access, vol. 4, pp. 8199–8211, previous research efforts have mostly used only ingredients for 2016. cuisine prediction, our findings demonstrate that other text-based [14] Scikit-learn, “sklearn.preprocessing.StandardScaler.” [Online]. information extracted from recipes can be used as well. While Available: http://scikit- ingredients with an obtained accuracy of almost 74 % remain to learn.org/stable/modules/generated/sklearn.preprocessing.StandardS be the most efficient, cuisine prediction from recipe name and caler.html. Accessed on: August 21, 2018. preparation instructions also performs well. Whereas prediction [15] Pandas, “pandas.get_dummies.” [Online]. Available: based on preparation time, skill level and nutrition facts were https://pandas.pydata.org/pandas- discovered to be less effective, with about 56 % accuracy. docs/stable/generated/pandas.get_dummies.html. Accessed on: August 21, 2018. [16] NLTK, “NLP with Python – Processing Raw Text.” [Online]. 7. REFERENCES Available: http://www.nltk.org/book/ch03.html. Accessed on: [1] H. Su, M. K. Shan, T. W. Lin, J. Chang, and C. T. Li, “Automatic August 21, 2018. recipe cuisine classification by ingredients,” Proceedings of the [17] NLTK, “NLP with Python – Accessing Text Corpora and Lexical 2014 ACM International Joint Conference on Pervasive and Resources.” [Online]. Available: Ubiquitous Computing Adjunct Publication - UbiComp 14 Adjunct, https://www.nltk.org/book/ch02.html. Accessed on: August 21, pp. 565–570, 2014. 2018. [2] S. Srinivasasubramanian, B. Kushwaha, and V. Parekh, “Identifying [18] NLTK, “NLTK HOWTOs – Stemmers.” [Online]. Available: Cuisines From Ingredients,” 2015. [Online]. Available: http://www.nltk.org/howto/stem.html. Accessed on: August 21, https://pdfs.semanticscholar.org/3daa/3c535a3c2580e69984203137 2018. db3ee6422601.pdf. Accessed on: August 16, 2018. [19] Scikit-learn, “Feature extraction.” [Online]. Available: http://scikit- [3] P. Bhat, S. Gupta, and T. Nabar, “Bon Appetite: Prediction of learn.org/stable/modules/feature_extraction.html. Accessed on: cuisine based on Ingredients.” [Online]. Available: August 21, 2018. http://cseweb.ucsd.edu/~jmcauley/cse255/reports/fa15/020.pdf. [20] Scikit-learn, “Naive Bayes.” [Online]. Available: http://scikit- Accessed on: August 16, 2018. learn.org/stable/modules/naive_bayes.html. Accessed on: August 21, [4] H. H. Holste, M. Nyayapati, and E. Wong, “What Cuisine? - A 2018. Machine Learning Strategy for Multi-label Classification of Food [21] Scikit-learn, “Support Vector Machines.” [Online]. Available: Recipes,” 2015. [Online]. Available: http://scikit-learn.org/stable/modules/svm.html. Accessed on: http://jmcauley.ucsd.edu/cse190/projects/fa15/022.pdf. Accessed on: August 21, 2018. August 16, 2018. [22] Scikit-learn, “Classification metrics.” [Online]. Available: [5] R. S. Verma, and H. Arora, “Cuisine Prediction/Classification based http://scikit- on ingredients.” [Online]. Available: learn.org/stable/modules/model_evaluation.html#classification- http://cseweb.ucsd.edu/~jmcauley/cse255/reports/fa15/028.pdf. metrics. Accessed on: August 21, 201 346 Introducing Blockchain Technology into a Real-Life Insurance Use Case Aljaž Vodeb Aljaž Tišler Martin Chuchurski Faculty of Electrical Engineering and Faculty of Economics and Business Faculty of Electrical Engineering and Computer Science University of Maribor Computer Science University of Maribor Maribor, Slovenia University of Maribor Maribor, Slovenia aljaz.tisler@student.u Maribor, Slovenia aljaz.vodeb@student.um. martin.chuchurski@student. si m.si um.si Mojca Orgulan Tadej Rola Tea Unger Faculty of Electrical Engineering and Faculty of Electrical Engineering and Faculty of Law Computer Science Computer Science University of Maribor University of Maribor University of Maribor Maribor, Slovenia Maribor, Slovenia Maribor, Slovenia tea.unger@student.um. mojca.orgulan@student. tadej.rola@student.u um. si si m.si Žan Žnidar Muhamed Turkanović Faculty of Electrical Engineering and Faculty of Electrical Engineering and Computer Science Computer Science University of Maribor University of Maribor Maribor, Slovenia Maribor, Slovenia zan.znidar@student muhamed.turkanovic@ um.si um.si ABSTRACT The outcome of such approaches is various: (1) proposal and The paper presents an analysis of a possible introduction of the prototypes of blockchain-based use cases, which unnecessarily use blockchain technology into an insurance business use case. The this technology, (2) prototypes which are consistent with the technology’s purpose but are unpractical and not user analysis is focused on the implications such an attempt can have -friendly, and from various standpoints and the technical workaround needed for (3) failed attempts to produce a practical prototype or a production a prototype to be implemented. system. In this article, we explore the possibility of introducing the blockchain technology in an insurance-based use case. The aim was to explore the possible reasonableness of such a use case, its possible Categories and Subject Descriptors restrictions, limitations, advantages and disadvantages. The focus of H.3.4 [Information Storage and Retrieval]: Systems and the paper is on thus the implications of such a use case on all related Software processes and the overall picture of a possible implementation. General Terms 2. BLOCKCHAIN Performance, Economics, Reliability, Experimentation, Security, A blockchain is an invention that can be seen as a distributed ledger Legal Aspects, Verification. of all transactions or events that have been executed and shared among distributed participants. All transactions are verified with Keywords distributed consensus inside the system. Considering basic blockchain platforms, once a transaction is recorded, it cannot be Blockchain; Smart contracts; Ethereum; Insurance removed [2]. Group of verified transactions are stored in a block. Each block contains a cryptographic hash of the previous block and 1. INTRODUCTION a timestamp. New linked block strengthens the integrity of the Blockchain technology nowadays is considered as the new IT previous one, making the chain extremely tamper resistant and revolution and even as the messiah for all IT-based problems. secure. With a public blockchain, a copy of the entire transaction Nevertheless, as with other innovative technologies, public’s hype database (ledger) is distributed to the network. Every person can about the technology is fading. Experts now know that the view transactions and even participate in a consensus process. technology is useful only for specific domains and use cases as public, virtual and untrusted environment or cryptocurrency-based scenarios. Nonetheless, media is full of articles and news about corporations and companies using blockchain technology for some specific use case, which may or may not be fully meaningful. The result of such news is rising prices of cryptocurrencies and more 347 importantly, rising stock prices of organisations [1]. Blockchain enables a more effective way to solve the virtual 3. USE CASE currency problem. It solves it in a distributed manner, without the To test the concept of introducing the blockchain technology in a need for a central authority [3]. Central authority represents costs real-life business use case, we chose the insurance domain, which is and must be trusted to act honestly. also one of the promising domains for the blockchain technology Public blockchain is not the only type of possible blockchain [5]. platforms. There are also private and consortium blockchains [4]. A preliminary result of a market analysis has shown that a possibly Private blockchains have write permission kept centralized to one meaningful, but not yet implemented use case would be the lost organization. That can be useful for a single company for database baggage insurance. This specific real-life use case nowadays still management, auditing, etc. In a consortium blockchain partner represents long-term problems for passengers and airlines. To make companies are joined together in a trusted and adaptable network. it as user-friendly and meaningful as possible, an app was envisaged. The right to read in such blockchain types may be public or restricted The key functionalities of such an app, as presented in Figure 1, to the participants. would be: (1) user scans QR code of the flight ticket, (2) confirms read data, (3) scans barcode of baggage, (4) acknowledges terms of 2.1 Smart contracts the smart contract, (5) info about the possible payout is provided. The concept of smart contract has been known since 1994, when With help of RFID trackers at the airports the system would be able Nick Szabo defined it as a "computerized transaction protocol that to surveillance the position of passenger's baggage based on the executes the terms of a contract". Inside the blockchain context, newly confirmed IATA resolution 753. In case of a lost or delayed smart contracts are stored on the blockchain. They can be presented baggage, an activation of a blockchain-based smart contract is as stored procedures in relational databases. Given that smart executed. A compensation could be given in crypto or fiat currencies contracts are deployed on the blockchain, they have their own (ex. ETH, EURO), within 4 levels of payout. unique addresses. A smart contract is invoked by executing a transaction to the unique address of the contract. It is then executed independent and automatically on each node in the network [8]. The contract has its own state and can manage assets on the ledger. It allows expressing the business logic within a programming code. A well-written smart contract should describe all the possible outcomes of the contract. This means that a function would refuse to execute in case of incorrect (inconsistent with business logic) parameters [8]. Smart contracts are deterministic - this means that the same input will always produce the same output. Implementation of smart contracts on known platforms (e.g., Ethereum), written for example in the Solidity programming language, the developer is prevented from writing non-deterministic contracts, since the programming language does not contain non-deterministic Figure 1: Poster for a possible lost baggage insurance. constructs. All communication with a smart contract is done through cryptographically signed transactions. This means that all blockchain stakeholders will receive a cryptographically verified 4. IMPLICATIONS trace of a contract operation. This section provides the implications of a possible implementation of a blockchain-based solution as presented in section 3 on three 2.2 Oracles domains, legal, economic and organizational. Smart contracts on the Ethereum blockchain platform run within the Ethereum ecosystem, where they communicate with each other. 4.1 Legal implications External data can only enter the blockchain (i.e. smart contracts) Blockchain technology as presented in section 3 raised up some through external interaction using a transaction. This is also a legal issues. The main legal question is the General Data Protection shortcoming of the platform, because the majority of business logic Regulation (GDPR). GDPR is a legal framework for personal data is based on external data, which is thus not part of the blockchain privacy, it has been written by the European Union (EU) and became ledger (e.g., weather, currency price) [9]. To overcome such a effective on May 25th. This framework is drastically changing shortcoming an oracle can be used. Oracle is a trusted data source business of any digital venture. The Regulation granted EU citizens that sends external data to a smart contract in form of a transaction. new rights, e.g., the right to be forgotten and right to request all data By doing so, it relieves the smart contract of the need to directly storage and acquisition links. The latter allows an individual to ask access the desired data outside of the network. Oracles are usually an organization to delete all their personal data they store. This offered as a third-party solution [8]. specific right is also the main problem in the blockchain technology. The oracle service behaves like a data courier where communication Blockchain technology relies on the principles of decentralization between the service and smart contract is asynchronous. First, the and immutability, which means that data stored on the ledger transaction performs the function within a smart contract in which cannot be deleted. When this data includes personal data, we have the instructions for service are sent. The Oracle service will then a problem in the GDPR area. This is the main implication of this obtain a result based on the parameters that will be returned to the domain, since the use case worked on required the processing of smart contract via a special function (callback) implemented in the personal data. The main question is thus how to process personal main smart contract in which we want data (result) from the service data with the blockchain, but still being able to delete it if needed [9]. or to process it outside the blockchain. Research shows that many experts are trying to find a solution [7]. Majority of the solutions are focused on the off/on chain paradigm, whereby personal data is never dealt with on the blockchain. Nonetheless, new problems arise as how to link off/on chain data and if the link itself is a GDPR violation. 348 4.2 Economic Figure 2 presents the architecture of the possible solution. Users The main goal of the solution is to enable air passengers to sign an connect to the service through a dApp with the option to pay with ad hoc luggage insurance, which is tied to an airline ticket. The crypto or fiat currencies. For clarity, the former option will be blockchain technology will be used for the insurance coverage and marked with the letter (a), and the latter with (b). There are two blockchains used, the Ethereum’s MainNet to process paymen the payout of an insurance premium. The solution should allow the t payment of the insurance coverage through cryptocurrencies to get transactions and our InsurNet for business logic (private Ethereum the biggest customer coverage. It is a new business model, where the network). Crypto transactions are first processed on the MainNet target group are all airline users. (2a), where an oracle is triggered to convert the value into fiat (2.1a), before sending it to the InsurNet (2.2a), whereas fiat requests are The biggest negative factor associated with the possible solution is processed directly through the API and if successful, forwarded the volatility of cryptocurrencies. In practice, this represents the towards the InsurNet (2b) to create the insurance (smart) contract. possibility that we lose some of our assets as a customer or as airlines. The InsurNet smart contract uses an oracle deployed at an airline to In addition to volatility, problems can occur in certain processing retrieve the status of the baggage (3.1 and 3.2) before processing the delays. The application itself is also linked to airline and airport data. business logic to determine the validity of the claim. If the user is If the system fails, automatic payment is not made possible, nor can entitled to a payout, the payout oracle is called (4) to determine the the insurance be concluded. From an economic point of view, the correct payment method and convert currency if needed. In case the application also brings many positive aspects. It is about introducing user paid in cryptocurrency (5a), the payout is processed on the the possibility of speeding up the rigid process of current luggage MainNet (6a). Otherwise the FIAT payout is handled off-chain (5b). insurance and redress. The cost of maintaining a blockchain network and smart contracts is not negligible. These can be covered through the annual contribution of airlines for their usage of such a possible solution. At the same time a certain percentage can be collected from each insurance. The economic advantages of such a solution are many: (1) introduction of new technology, (2) the possibility of ad hoc insurance, and (3) a new business model. 4.3 Organizational One of the main problems of a possible solution are of organizational structure. For it to make sense, a platform should be implemented, where all willing airlines could register and provide baggage insurance to all possible consumers. Each airline can and should have a partnership with an insurance company. Thus, to Figure 2: Architectural model of the proposed solution. complete the registration, the airlines must provide their insurance price and max payout in case of a lost baggage. Furthermore, the 6. DISCUSSION solutions must be automatic and enable easy baggage check and Due to the Ethereum Protocol, where every transaction must be insurance claim. A simplification of such a request comes with the validated by miners and added to the block, these can be slowly IATA Resolution 753, which states that by June 2018, airline processed. When a user pays insurance with the cryptocurrency members must be able to, among others, demonstrate delivery of Ether into the smart contract on the MainNet and the transaction is baggage when custody changes [6]. This furthermore implies that confirmed, the function in our smart contract will trigger an event, the ecosystem must include airports which will provide the data which we can listen from outside of our dApp. We will detect the mentioned about the status of the baggage. Technically, a link to a event only when the transaction is confirmed. Once our server web service is required, where data about the baggage is accessible. detects the "Paid" event from the MainNet, it will create a new smart contract on our private blockchain InsurNet. This is reflected 5. PROTOTYPING in some latency for the user. With the aforementioned oracle, we It should be emphasized that blockchain technology is a rather have two more. One is to verify the location of the luggage, while unexplored thing. In most cases there are no examples of good the other one is to process the payment when the event is triggered practice on process of how the introduction of the blockchain on InsurNet. should start. We can consider the following example where the user pays After analyzing the possible use-case and its implications we insurance for one luggage in the cryptocurrency. We will assume propose a prototype in a form of a decentralized application (dApp), the average time to validate the transaction on MainNet is 25 based on the Ethereum smart contracts. The front end of the seconds. The user transfers the cryptocurrency to our smart solution could be a simple Angular 2 web application with an contract, where the validation of this transaction takes 25 seconds. intuitive, user-friendly interface, accessible on multiple devices. Then, on a triggered event, oracle performs a new transaction on The main advantage of using a web application as opposed to our network, where the transaction validation time is defined for 10 device-specific applications, is the support of various operating seconds. Because the user does not have the luggage yet, after three systems and models. If a user selects to pay with cryptocurrency, hours of landing, he performs a payout using the dApp. he/she can use the plugin MetaMask to connect to the Web3 part of Transactions are done within 10 seconds. An oracle then performs the application and send a signed transaction to a smart contract on a new transaction to write the current location information in the the blockchain. According to GDPR laws, personal information smart contract (+ 10 seconds). Since baggage is not yet available, needs to be delible, therefore it should be stored in a separate the user is entitled to a payout, which is reflected in a new event database off-chain, accessible through an API. Such an architecture where an oracle performs a transaction on the MainNet. The can be given by storing airline information off-chain and non- validation of this transaction takes 25 seconds. Thus, it takes at least identifying user insurance data on the blockchain. 80 seconds for all transaction validations to complete. 349 8. REFERENCES [1] CB Insights. Companies 'pivoting to blockchain' see huge stock spikes - but does the hype hold up? CB Insights - Research 7. CONCLUSION Brief. [Available] 2018. By proposing the concept of a fully workable prototype, we www.cbinsights.com/research/blockchain-hype- stock-trends. demonstrate that a solution is possible. Nevertheless, after considering all the implications, we conclude that such a solution [2] BlockChain Technology: Beyond Bitcoin. M. Crosby, would be unpractical and not user friendly, due to all workaround Nachiappan, P. Pattanayak, S. Verma and V. Kalyanaraman. needed in order to prepare a fully working technical solution. 2016, Applied Innovation Review . Considering the current evolutional stage of the blockchain [3] Mattila, Juri. The Blockchain Phenomenon – The Disruptive technology, we conclude that a fully crypto-based solution can be Potential of Distributed Consensus Architectures. [Available] met with approval, thus advocating the idea of the blockchain researchgate.net/publication/313477689_The_Blockchain_P technology being seen as business disruptor in the sense of digital henomenon_-_The_Disruptive_Potential_of_Distribute. money. [4] EduCTX: A Blockchain-Based Higher Education Credit Platform. Muhamed Turkanović, Marko Hölbl, Kristjan Košič, ACKNOWLEDGMENTS Marjan Heričko, Aida Kamišalić. 2018, IEEE Access , str. 5112 Our thanks to the public scholarship, development, disability and - 5127. maintenance fund of the Republic of Slovenia and the project [5] Bruno Teboul, Frédéric Maserati, Maxime Leroux. Following the Creative Path to Knowledge 2017 – 2020 (Po BLOCKCHAIN: CONCEPT AND APPLICATION kreativni poti do znanja 2017 – 2020) - SmartInsTech. DOMAINS. Keyrus. [Available] http://keyrus- prod.s3.amazonaws.com/Avis%20d%27expert/Blockchain/Avis % 20d%27Expert_BLOCKCHAIN-EN%20COM.pdf. [6] IATA. Baggage Reference Manual. 2018. [Available] https://www.iata.org/publications/Documents/brm03-toc- 20180523.pdf. [7] Mercer, Rebekah. Privacy on the Blockchain: Unique Ring Signatures. arXiv. [Available] 2016. https://arxiv.org/pdf/1612.01188.pdf. [8] Podgorelec, Blaž. Arhitektura za nadgradljivost in zamenljivost pametnih pogodb na platformi Ethereum. s.l. : DKUM, 2018. [9] Zdun, Maximilian Wöhrer and Uwe. Design Patterns for Smart Contracts in the Ethereum Ecosystem. univie.ac.at. [Available] 8 2018. http://eprints.cs.univie.ac.at/5665/1/bare_conf.pd 350 A Brief Overview of Proposed Solutions to Achieve Ethereum Scalability Blaž Podgorelec Patrik Rek Tadej Rola Faculty of Electrical Engineering and Faculty of Electrical Engineering and Faculty of Electrical Engineering and Computer Science Computer Science Computer Science University of Maribor University of Maribor University of Maribor Maribor, Slovenia Maribor, Slovenia Maribor, Slovenia blaz.podgorelec@um.si patrik.rek@um.si tadej.rola@student.um.si Muhamed Turkanović Faculty of Electrical Engineering and Computer Science University of Maribor Maribor, Slovenia muhamed.turkanovic@um.si ABSTRACT conclude that it is becoming increasingly popular. The increase in Blockchain technology is part of Gartner’s top technological popularity consequently affects the increased number of trends in the following five years, whereby already moving away transactions performed within the Ethereum blockchain network from the peak of the inflated expectations on its hype cycle, [2], whereby we can assume that the number of business towards the slope of enlightenment. With the development of the processes that are implemented with the help of blockchain blockchain technology, the emergence of completely new business technology and Ethereum is also increasing. processes is anticipated, as well as changes to existing business All transactions transmitted on the blockchain network are processes, which will include the use of blockchain technology in irreversibly recorded in a shared ledger among all network nodes its implementation, partially or completely, thereby taking [3, 4]. Nodes in the blockchain network perform a protocol, advantage of the benefits that the technology itself offers. defining the ability to create new blocks with associated Nevertheless, the technology has several drawbacks, whereby the transactions in an approximate 15 seconds time frame. This allows most vivid is the scalability problem. With the introduction of the frequency of transactions executed in the network to be Blockchain 2.0 and the Ethereum platform, the scalability approximately 7 - 15 transactions per second (tp/s) [5]. The open problem seemed settled out for a moment, which proved source Ethereum platform is based on a permisionless and otherwise with first generations of non-fungible tokens and high publicly accessible blockchain network, which is at the same time traffic. Although Ethereum is in its infancy, progress is on high a distributed and decentralized operating system for running smart tracks, with this year’s focus on the infrastructure. A lot of contracts via its Ethereum Virtual Machine (EVM). Because of research and work is being done on the Ethereum’s layer 2 scaling the platform indigenous crypto currency called Ether, generated solution such as the state channels, plasma and sharding. This by the blockchain network and defined by the protocol, the paper presents a brief overview of the current state of the platform is often used as a payment system, like the Bitcoin. mentioned proposed solutions and some ongoing projects, which Therefore it is often compared to existing non-crypto payment are focused on their implementation. solutions, such as Visa, which, unlike the Ethereum platform, is capable of processing a much larger number of transactions Categories and Subject Descriptors (56,000 tp/s) [6]. H.3.4 [Information Storage and Retrieval]: Systems and In the paper, we will present the problem of scaling the Software Ethereum network and the proposed solutions. These solutions could increase the number of transactions carried out on the General Terms Ethereum platform, thus getting closer or exceeding the Performance, Design, Reliability, Experimentation, Security processing capacity of existing non-crypto payment systems. This would enable the development and implementation of new business processes with the blockchain technology. Keywords Blockchain, scalability, Ethereum, channels, plasma. 2. ETHEREUM SCALING PROBLEM The current implementation of the Ethereum protocol requires the 1. INTRODUCTION processing of all, within the network transmitted transactions, as In recent years, on the basis of an increase in the market well as the storage of all states, from each node in the network, capitalization [1] of the Ethereum platform, the performance of that acts as a validator [7]. To confirm a change of the network which is based entirely on the blockchain technology, we can state with a transaction, the transaction must be included in a 351 block created by a node, which must solve the calculation puzzle network due to the need for processing transactions of defined by the distributed consensus protocol, which is in the blockchain networks [12]. current Ethereum version the Proof of Work (PoW). The The described "simple" solutions directly relate to the so-called processing speed of the transactions is limited by the capacity of trilemma of blockchain technology, which says that the each individual node participating in the network as the blockchain network can contain only two of the three features, transaction validator. Such an implementation of the protocol such as: provides increased safety in terms of secure processing of transactions within the network, which is one of the key properties - Decentralization of such systems. At the same time, the way in which an increased security is achieved, is a major obstacle achieving a greater - Scalability number of transactions carried out within the blockchain network, - Security due to its need for heavy computation [8]. In the case of the use of different altcoins, this would mean The number of transactions one block can include is limited by increasing the efficiency (scalability) of transaction processed the number of gas (fee for processing the operations within the within the blockchain network, while in contrary a reduction of transaction), that can be consumed by all transactions in the block. security of the network itself. The increase in the limit of number In the future, it is possible to expect a change in the way of of transactions in a single block and the aggregation of reaching consensus between the individual nodes in the Ethereum computational power or the share between different blockchain network. Namely, the transition to the use of the Proof of Stake networks would theoretically increase the efficiency (scalability), (PoS) protocol is planned, which would mean that the time of which would require greater use of computational power for the block generation within the Ethereum network with associated processing of all requirements within the blockchain network transactions could be reduced to an average of four seconds [5]. from the network nodes. This reduces the possibility of equal The transition to a new protocol for reaching consensus among participation in the network by nodes with less computational the nodes in the blockchain network will thus reduce the current power, which can lead to a reduction in the decentralization of the scaling problems. In addition, the switch to PoS distributed blockchain network by nodes who have greater computing power consensus will decrease the required computational power and [8]. thus energy consumption of the network. In the following chapters, we will present some solutions that Changing the network consensus protocol between nodes will could solve the described problem of efficiency, whereby not to have a positive effect on the transaction processing frequency affecting one of the described properties of the trilemma of the within the blockchain, but it is expected that the number of blockchain technology. processed transactions will still be significantly smaller compared to the existing payment systems. Described problems in the terms of achieving greater efficiency of blockchain, assuming 3. PROPOSED SOLUTIONS knowledge of its structure and understanding of the concepts of The main concern of blockchain technology is the security and a the blockchain technology, offer so-called "simple" theoretical distributed consensus in a decentralized network. The processing solutions, such as: of every transaction by all nodes of the network is a process that provides these characteristics but does not provide enough 1. It envisages the use of different "altcoins" within a measure for increasing efficiency and scalability. Below we variety of separate blockchain networks, which results describe some already proposed solutions, which can help in a strong increase in the flow rate of the performance increasing the efficiency and scalability of the Ethereum of individual transactions within the separate blockchain blockchain network without undermining the security and networks. As a result, due to the increased number of decentralization of the network as such. different blockchain networks, a reduced number of nodes within different blockchain networks are 3.1 State channels expected, which would mean that separate blockchain One of the proposed solutions, which is currently considered to be networks will be more susceptible to attacks by the most mature and used, is based on the transaction processing malicious nodes than if all network nodes are merged approach outside the blockchain network (i.e. off-chain) through within a single common blockchain network [9, 10]. the establishment of state channels [13]. The proposal of the solution derives from the so-called payment channels, the purpose 2. Increasing the limit of the number of transactions per of which was to allow multiple micro-transactions between two block or increasing the ceiling of fuel consumption in users of the system without the need of transmitting each the case of the Ethereum protocol, theoretically implies transaction through the blockchain network [14]. a large number of processed transactions. Nevertheless, this requires significantly more computational power While payment channels focus on off-chain processing of (for using the PoW protocol, or the percentage (stake) payment transactions, the purpose of the "state channels" is to when using the PoS protocol) to validate a block with establish a channel, through which the state can be changed an increased number of transactions of an individual outside the blockchain network, between predefined participants node in the network [9, 11]. [15]. This is because Ethereum blockchain holds the state of each defined variable of every deployed smart contract. The need to 3. Combining computational power (when using the PoW process a transaction within a blockchain network occurs only in protocol) or stake (when using the PoS protocol) case of disagreement about the state changed by a transaction between the different blockchain networks, can within the established channel by any participant or in the case of theoretically increase the flow of transaction processing, a closed communication within the channel. In case that there is but this could burden each individual node in the 352 no disagreement about the changed state during the 3.3 Sharding communication within the established channel, this solution With the current implementation of the protocol, each node that is significantly increases the number of transactions, since it part of the Ethereum network must validate every transaction, aggregates micro transactions and issues them as one in a which ensures a high level of network security. One solution is predefined time [16]. sharding, where the protocol would separate the network state into State channels are implemented with the help of dedicated smart smaller partitions, called shards. Each shard would store its contracts. The establishment of communication through such a separate state and transaction history. By implementing such a channel is carried out with a special “channel smart contract”, protocol, certain nodes would process only the transactions of aimed at ensuring fair communication between participants that certain shards. Transactions on different shards at the same time perform operations and record the final state into the blockchain would increase the permeability of these [20]. network, after the communication has ended. In case of a conflict Sharding is a general technique used in distributed computing, the between participants in communication outside the blockchain implementation of which can be expected in Ethereum by 2020 (within the channel), the smart contract has the task of selecting [21]. Implementation of sharding is the only one of the described the most relevant last state that the users still agreed on when scaling solutions that will practically have no impact on end users, communicating within the channel [17]. The security of such an as well as not on smart contract developers on the Ethereum off-chain communication approach is based on the fact that each platform. The system for storing states will remain the same. The message sent through the status channel is cryptographically change will be at layer 1 of the Ethereum Protocol. Solutions signed, with the aforementioned channel smart contract having an mentioned in 3.2. and 3.1. will work on layer 2 [22]. Sharding implementation for verifying these messages. Each participant can eliminates the need for the entire network (each node) to process cancel the communication at any time, and the final state that is all transactions. The result is increased number of processed recorded in the blockchain is that which is recognized by all transactions per second [21]. participants in the off-chain communication [15]. Prior to implementing sharding in the protocol, various challenges This type of communication allows the implementation of more must be addressed. The main challenge is a single-shard take over complex operations defined within smart contracts, completely attack. With such an attack, an attacker could possibly take independent of the blockchain network. Consequently this means control of the entire shard, which may result in the avoidance of almost instantaneous execution of operations with very low total sufficient validations, or even worse, to validate the blocks that costs of execution of all implemented channel transactions, since are incorrect. These attacks are usually prevented by random all transactions carried out within the established off-chain sampling schemes. The next challenge is the availability of states channel are aggregated into a single transaction [17, 13]. between different shards. The most appropriate approach for addressing this challenge is that the effect of a transaction 3.2 Plasma depends on the events that happened before in the second shard. The scalability of the Ethereum network with theoretically trillion A simple example is the transfer of money where the user A (e.g. transactions per second should be achieved by the introduction of in shard 2) transfers money to user B (e.g. in shard 7). First, a a strategy called Plasma. Similarly, as in the solution described in debit transaction is executed that destroys the tokens at user A (in Chapter 3.1, the purpose of Plasma is to implement transactions shard 2), after which a "credit" transaction is created that creates without the need for individual confirmation of each of them by the tokens of user B (in shard 7). This transaction has an account the blockchain network. The solution envisages the introduction indicator on a "debit" transaction, which proves that the "credit" of several side chains, whereby the last state of the newly created transaction is legitimate [8]. chain being recorded in i.e. the main blockchain network. This could be implemented without any need to change the current 4. CONCLUSION protocol and Ethereum network. The most important factor in In the paper, we presented several different solutions, the common terms of achieving security in the Plasma solution, relates to the purpose of which is to achieve greater efficiency of scalable privilege of every user to perform transactions within any side transaction processing in the Ethereum blockchain network. State chain (with the exception of the main Ethereum chain) and to channels move state modifications outside of the main blockchain leave the side-chain and write the final state in the main Ethereum network. The Plasma solution envisages the introduction of chain - where the final valid state is defined. To prevent the several blockchains, whereby each chain is used for a specific recording of a false state into the main chain, the Plasma solution purpose. Both solutions allow users to record the final state in the suggests a "Challenge mechanism", which assumes that the state main Ethereum blockchain network. We also descried the that a user wants to record in the main chain is frozen for a certain sharding solution, the introduction of which, in contrast to the period. During this period, other users can prove that the above-mentioned solutions, requires the change of the lowest proposed state is not relevant. Because of the above mechanism, layer of the Ethereum protocol. All the described solutions pursue the user must provide a sum of the Ether cryptocurrency into such the goal of not reducing the current level of transaction processing a transaction that writes the state into the main Ethereum chain, security, as well as maintaining the decentralization of the which if another user proves that such a transaction contains an blockchain itself in order to achieve scalability. In the future, due invalid state, loses and is acquired by that user, who proved the to the increase in the number of transactions transmitted within invalid state. This mechanism could trigger a lot of false evidence the Ethereum network, it is reasonable to expect several concrete of invalid transactions; therefore, a user wishing to prove an implementations (Loom Network, OmiseGO, Raiden,...) of the invalid transaction must pledge a sum of the Ether cryptocurrency, described solutions, as well as an increased use of these in which in the case of false evidence of invalidity, is acquired by the practice, since it is the increase in the efficiency of the transaction user of the original transaction [18, 19]. processing which is one of the key factors in achieving the 353 optimization of existing and new business processes, supported by [12] A. Judmayer, A. Zamyatin, N. Stifter, A. G. Voyiatzis, and E. the blockchain technology. Weippl, “Merged mining: Curse or cure?,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. 5. ACKNOWLEDGMENTS Lect. Notes Bioinformatics), vol. 10436 LNCS, pp. 316–333, The authors acknowledge the financial support from the 2017. Slovenian Research Agency (research core funding No. P2-0057). [13] P. Mccorry, S. Meiklejohn, and A. Miller, “Pisa : Arbitration Outsourcing for State Channels.” 6. REFERENCES [14] “Lightning Network.” [Online]. Available: [1] “Total Market Capitalization,” coinmarketcap.com, 2018. https://lightning.network/. [Accessed: 01-Aug-2018]. [Online]. Available: https://coinmarketcap.com/charts/. [15] J. Coleman, L. Horne, and L. X. L4, “Counterfactual: [Accessed: 06-Jul-2018]. Generalized State Channels,” 2018. [2] “Ethereum Transaction Chart,” etherscan.io, 2018. [Online]. [16] S. Dziembowski, L. Eckey, and S. Faust, “Perun : Virtual Available: https://etherscan.io/chart/tx. [Accessed: 06-Jul- Payment Hubs over Cryptocurrencies.” 2018]. [17] S. Dziembowski, S. Faust, and K. Hostáková, “Foundations [3] B. Podgorelec, “Arhitektura za nadgradljivost in of State Channel Networks,” pp. 1–56, 2018. zamenljivost pametnih pogodb na platformi Ethereum,” University of Maribor, 2018. [18] J. Poon and V. Buterin, “Plasma : Scalable Autonomous Smart Contracts Scalable Multi-Party Computation,” [4] M. Pustisek, A. Kos, and U. Sedlar, “Blockchain Based Whitepaper, pp. 1–47, 2017. Autonomous Selection of Electric Vehicle Charging Station,” 2016 Int. Conf. Identification, Inf. Knowl. Internet [19] “Explained: Ethereum Plasma – Argon Group – Medium.” Things, pp. 217–222, 2016. [Online]. Available: https://medium.com/@argongroup/ethereum-plasma- [5] F. M. Benčić and I. P. Žarko, “Distributed Ledger explained-608720d3c60e. [Accessed: 02-Aug-2018]. Technology: Blockchain Compared to Directed Acyclic Graph,” 2018. [20] R. Jordan, “How to Scale Ethereum: Sharding Explained,” 2018. [Online]. Available: https://medium.com/prysmatic- [6] Visa, “Visa Inc. at a Glance,” no. August, p. 1, 2015. labs/how-to-scale-ethereum-sharding-explained- [7] V. Buterin, “A next-generation smart contract and ba2e283b7fce. [Accessed: 01-Aug-2018]. decentralized application platform,” Etherum, no. January, [21] J. Kim, “Vitalik Buterin: Sharding and Plasma to Help pp. 1–36, 2014. Ethereum Reach 1 Million Transactions Per Second,” 2018. [8] J. Ray, “On sharding blockchains,” github.com/ethereum, [Online]. Available: https://cryptoslate.com/vitalik-buterin- 2018. [Online]. Available: sharding-and-plasma-to-help-ethereum-reach-1-million- https://github.com/ethereum/wiki/wiki/Sharding-FAQs. transactions-per-second/. [Accessed: 01-Aug-2018]. [Accessed: 04-Jul-2018]. [22] A. Rathod, “We Should See Sharding in 2020 as Part of [9] “The State of Scaling Ethereum – ConsenSys Media,” 2018. ‘Ethereum 2.0,’” 2018. [Online]. Available: [Online]. Available: https://media.consensys.net/the-state-of- https://toshitimes.com/we-should-see-sharding-in-2020-as- scaling-ethereum-b4d095dbafae. [Accessed: 04-Jul-2018]. part-of-ethereum-2-0-eth-foundation-researcher/. [Accessed: 01-Aug-2018]. [10] A. Back, M. Corallo, and L. Dashjr, “Enabling blockchain innovations with pegged sidechains,” URL http//www., pp. 1–25, 2014. [11] GoChain, “GoChain : Blockchain at Scale,” pp. 0–5, 2018. 354 Integration Heaven of Nanoservices Ádám Révész Norbert Pataki EPAM Hungary Department of Programming Languages Budapest, Hungary and Compilers, Faculty of Informatics, Adam_Revesz@epam.com Eötvös Loránd University Budapest, Hungary patakino@elte.hu ABSTRACT benefits, improved scalability, separate responsibilities, bet- ter maintainability to name but a few [17]. On the other Microservices have become an essential software architec- hand, having a software architecture utilizing more than 70 ture in the last few years. Nanoservices as a generalization own built nanoservices in active development requires spe- of microservice architecture are getting more and more pop- cial care for build processes. ular recently. However, this means that every component In terms of continuous integration (CI) and continuous de- has more and more public interfaces, and the number of livery (CD) – modern software development process frame- components is increasing, as well. works pipelines are defined as composable parts of the pro- Integration hell had been appeared when the number of cess describing how the product is created, transformed and developers was increased. The developers work parallelly, delivered from making source code and configurations on so it is necessary to merge their work. Collaboration re- developers workstations to serving them to end users [16]. quires software support, such as version controll tools and The pipelines mentioned in this paper are executed by continuous integration servers. automation systems following deterministic scripts referred However, modern software development tools such as build as “pipeline scripts”. systems, testing frameworks and continuous integration servers This paper discusses the topic of bulk management of uni- become sensitive regarding the version of source code to deal fied pipeline scripts in aspects of reproducibility, replayabil- with. This can result in exponential explosion in many ways ity, compactness and overhead of change management. when nanoservices are in the focus. This paper is organized as follows. We present the prob- In this paper, we argue for workflow that can handle this lem of integration hell in section 2. We describe the problem exponential explosion. This workflow can be included into in section 3. Our proposed workflow is presented in section continuous integration servers as jobs in order to execute test 4. Finally, this paper concludes in section 5. cases in a reproducible way even if the test cases deal with special environment specifications. Moreover, the workflow is able to deal with building and artifact publishing pro- 2. INTEGRATION HELL cesses, as well. 2.1 Case Categories and Subject Descriptors The subject of the study is a software running on top of a container orchestration system operating over multiple D.2.7 [Software Engineering]: Distribution, Maintenance, nodes. Using event sourcing with Command Query Respon- and Enhancement; K.6.3 [Computing Milieux]: Software sibility Segregation (CQRS), the software utilizes over 70 Management services. Every own built service is stored in its own version con- Keywords troller system (VCS) repository [14]. Most of them are iden- tical in the aspect of programming language, project struc- Nanoservices, Integration, version control ture, packaging system, types of artifacts, testing frame- works, static analysis system (e.g. [11]). The discussion 1. INTRODUCTION continues about this kind of services. Microservices and nanoservices are essential software ar- 2.2 Orchestration chitectures recently. These software architectures have many A container orchestration tool manages resource alloca- tions, configurations, credentials of containers. Provides common internal network with service discovery, domain Permission to make digital or hard copies of all or part of this work for services, serving well defined endpoints for outer network personal or classroom use is granted without fee provided that copies are communications. not made or distributed for profit or commercial advantage and that copies In terms of scalable services, operating with nanoservices bear this notice and the full citation on the first page. To copy otherwise, to an orchestration tool must provide load balancer service republish, to post on servers or to redistribute to lists, requires prior specific over multiple nodes ensuring high availability. Also provides permission and/or a fee. declarative configuration and deployment management with CSS ’18 Ljubljana, Slovenia Copyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$15.00. the ability of rolling updates and rollbacks between config- 355 uration and deployment versions also. 2.4.5 Common pipeline Currently the industry standard for a real battle tested, The subject project uses mostly Java Spring Boot nanoser- serious production-grade orchestration tool is Kubernetes, vices, which kind of services have a common pipeline script developed by Google [9]. actively developed. The common pipeline script contains the following stages: 2.3 Build tools Modern programming laguage ecosystems have their own • VCS checkout (sometimes multiple) package manager for dependency han- • Build source code using package manager (like npm, dling and easy build, test, install and deploy management Gradle, Cabal, etc.) [12]. The common pipeline script utilizes those package managers, reaching higher level of abstraction [10]. For ex- • Run tests on the artifact using package manager ample: • Sending the source code to the static analysis system • Java, Scala: Gradle[3], Maven[5], Ant[1] • Building Docker image artifact • JavaScript - NodeJS: NPM[6], Yarn[8] • Uploading artifacts • C++: Conan [13] • Announcing build status on channels (email, instant • Python: Pip[7] messaging) • Haskell: Cabal[2] Since these are nanoservices, their Docker images differ only on the built artifact. The configurations, including en- • Docker (images): Docker (registry) [15] vironment variables, configuration and secret files, are han- Closed source software projects as the subject utilize arti- dled by the orchestration tool and building them into an fact repository systems which can serve repositories for mul- image is an anti-pattern in this use case. tiple type of packages for own artifacts and serve as cache 2.5 Integration hell definition for public domain packages (in case of outage and lowering network traffic). For example: Nexus, JFrog Artifactory. Integration hell is a place where developers have to main- tain all the pipeline scripts manually for each service or use 2.4 Pipelines a common pipeline script and update all the source codes and configurations on each service repository to be compat- The services are built automatically on VCS commit on ible with the pipeline script. Also called one pipeline script marked branches. Build pipeline scripts of actively devel- over all. oped services have to be in sync in order to guarantee the same level of quality and compatibility with environment (following its changes). 3. PROBLEM STATEMENT 2.4.1 Pipeline script 3.1 Build job generation A pipeline script is interpreted by a CI tool, a build system The jobs are generated depending on the VCS repository (e.g. Jenkins [4]), is a sequence of commands optionally path structure. The generator job accepts the list of the separated into stages. service names to make build job for. The build jobs are generated from template, the only difference is in the source 2.4.2 Pipeline script stage code repository URL and the project name. A pipeline script stage is a named sequence of commands. Used for visualizing the main parts of the script, leverag- 3.2 Single pipeline script repository approach ing process status display during execution, variable scope Having dozens of services with identical pipeline scripts, it segregation. would come in hand to use the exactly same pipeline script file checked out from one build script repository. 2.4.3 Pipeline command Each pipeline command can be variable declaration and 3.2.1 Limitations of updates definition (including functions), function invocation, shell The single pipeline script repository approach has mul- invocation. tiple pitfalls. Since the the job configuration has only the Ideally, a build system has its own pipeline script domain- repository, the branch name and the path of the pipeline specific language (DSL) with an application-programming script, any change on the pipeline script would affect all the interface (API) library for common operations like VCS check- build jobs at once. In this case either the ability to create out, packaging operations, status notifications, common con- experimental changes on the build scripts is lost or the abil- figuration and secret storage operations. ity to recreate all the build jobs without breaking any of them. 2.4.4 Build job In common CI tools, each pipeline script invoked by a 3.2.2 Lack of replayability corresponding build job. These jobs contain metadata for Other problem regarding the single repository approach running the pipeline script, like the location of the pipeline is the lack of replayability. Having a case when recreat- script itself. Storing and passing variables like job name, ing an artifact based on an older state of the service source parameters (given on job invocation via API call or web code repository is needed, there is no guarantee the cur- UI). rent state of the pipeline script in its repository is backward 356 Figure 1: Sequence diagram of the proposed work- flow compatible, so there is the risk of broken or unstable build (in worse case it turns out in production). The correct build script should be searched in the history of the pipeline script repository (see Figure 1). Figure 2: Sequence diagram of the single source of 3.2.3 Growing overhead truth approach The mentioned problems are getting harder to resolve as the size of the software project (the number of services) is This solution does not introduce the problem of difficult growing. The maintenance cost of those pipeline scripts is generator job but still carries the synchronization problem. high. Onboarding a new developer-, handing out the de- Pipeline scripts are being modified in multiple cases. There velopment of such project could be extremely difficult due are cases which are not strictly drived by source code changes. to the multiple tools and sytems, scripts and their difficult Having the case of enriching the log of the pipeline script in dependency graph. order to leverage traceability of the process. This change is made only in the pipeline script and the side effects are 4. PROPOSED WORKFLOW present only on the pipeline script log. Has no side effect on the artifacts or test results. There are multiple open ques- Addressing these problems a reasonable solution could be tions about which service VCS repository has to be updated a property file in each service source code repository. This first, which should be the subject of experimental changes approach makes the generator job more difficult since every and how to update all the other service pipeline script? invocation it should parse the property file of every repos- itory and generating the job according to that. An other 4.4 Automatized script updating problem is the synchronization of those property files. Addressing these questions, there is a pipeline script in the 4.1 Single source of truth VCS repository but unlike the single pipeline script reposi- tory approach (see 3.2), the service build jobs are not refer- There is an other, more compact, more robust and more ring to the script repository. There is a synchronization job redundant way to address the problems. The single source introduced instead. The pipeline script synchronization job of truth for service artifact build workflows should be the takes service name list as its arguments as the service build repository of their source code. This approach leverages the job generator job does. The pipeline script updater job has compactness of each service. The service VCS repository permission to update the service VCS repositories. To en- should contain the source code of the service, package de- force traceability an issue id referencing an issue describing scriptor (build scripts included) and the pipeline script. This the change and its cause is recommended to be present in the approach can be seen on Figure 2. commit message in all affected VCS repository. The figure 4.2 Utilization of VCS 3 presents this workflow. Since the VCS repository handles the pipeline script along with the source code, any arbitrary snapshot (commit) of 5. CONCLUSION the repository in any time of its history should contain the Microservices and nanoservices are popular software archi- pipeline script which executes exactly the same pipeline with tectures. On the other, dealing with complex software devel- exactly the same result any time. opment processes and many different development software tools, the maintenance can be a critical problem because of 4.3 Keeping job generator simple the combinatorical explosion. 357 [12] M. P. Martinez, T. László, N. Pataki, C. Rotter, and C. Szalai. Multivendor deployment integration for future mobile networks. In A. M. Tjoa, L. Bellatreche, S. Biffl, J. van Leeuwen, and J. Wiedermann, editors, SOFSEM 2018: Theory and Practice of Computer Science: 44th International Conference on Current Trends in Theory and Practice of Computer Science, Krems, Austria, January 29 - February 2, 2018, Proceedings, pages 351–364, Cham, 2018. Springer International Publishing. [13] A. Miranda and J. a. Pimentel. On the use of package managers by the C++ open-source community. In Proceedings of the 33rd Annual ACM Symposium on Applied Computing, pages 1483–1491, New York, NY, USA, 2018. ACM. Figure 3: Sequence diagram of the proposed work- [14] S. Phillips, J. Sillito, and R. Walker. Branching and flow merging: An investigation into current version control practices. In Proceedings of the 4th International Workshop on Cooperative and Human Aspects of This solution holds some security concerns like the up- Software Engineering, CHASE ’11, pages 9–15, New dater pipeline execute right has to be available for restricted York, NY, USA, 2011. ACM. group of users since the VCS enables Jenkins to commit to [15] Á. Révész and N. Pataki. Containerized A/B testing. the master (trunk) branch. In Z. Budimac, editor, Proceedings of the Sixth The current prototype version is restricted to only one Workshop on Software Quality Analysis, Monitoring, kind of services to upgrade their build pipeline. Enabling Improvement, and Applications, pages 14:1–14:8. modular build scripts and their modular upgrade could be a CEUR-WS.org, 2017. next iteration. The bulk update problem could be derivated to a version controll system problem, updating common files [16] S. Stolberg. Enabling agile testing through continuous in two or more repositories. In context of build systems like integration. In Agile Conference, 2009. AGILE ’09., Jenkins (git) submodules could not be an optimal solution pages 369–374, New York, Aug 2009. IEEE. increasing complexity. [17] E. Wolff. Microservices: Flexible Software The proposed solution grants the robust script handling Architectures. CreateSpace Independent Publishing workflow allowing bulk pipeline script updates and replaya- Platform, 2016. bility. It introduces some additional difficulty with the up- date process but it has been automatized. The approach reached a single source of truth state for each service artifact creation process and the refered source is the VCS repository which is a great tool to manage and observe the whole devel- opment of its content through time. The approach reduces the cost of maintaining pipeline scripts. 6. REFERENCES [1] Ant. https://ant.apache.org/. [2] Cabal. https://www.haskell.org/cabal/. [3] Gradle. https://gradle.org/. [4] Jenkins. https://jenkins.io/. [5] Maven. https://maven.apache.org/. [6] Npm. https://npmjs.com/. [7] Pip. https://pypi.org/project/pip/. [8] Yarn. https://yarnpkg.com/. [9] D. Bernstein. Containers and cloud: From LXC to Docker to Kubernetes. IEEE Cloud Computing, 1(3):81–84, Sept. 2014. [10] C. Ebert, G. Gallardo, J. Hernantes, and N. Serrano. Devops. IEEE Software, 33(3):94–100, May 2016. [11] G. Horváth and N. Pataki. Source language representation of function summaries in static analysis. In Proceedings of the 11th Workshop on Implementation, Compilation, Optimization of Object-Oriented Languages, Programs and Systems, ICOOOLPS ’16, pages 6:1–6:9, New York, NY, USA, 2016. ACM. 358 Service Monitoring Agents for DevOps Dashboard Tool Márk Török Norbert Pataki Department of Programming Languages Department of Programming Languages and Compilers, Faculty of Informatics, and Compilers, Faculty of Informatics, Eötvös Loránd University Eötvös Loránd University Budapest, Hungary Budapest, Hungary tmark@caesar.elte.hu patakino@elte.hu ABSTRACT sends report to the developers regarding the changes and their effects [6]. Deployment of the compiled application DevOps is an emerging approach that aims at the symbiosis and its necessary dependencies can be launched in various of development, quality assurance and operations. Develop- infrastructures [4]. Virtual machines in cloud, Docker con- ers need feedback from the test executions that Continuous tainers on a host take part in the deployment frequently [5]. Integration servers support. On the other hand, developers Configuration management tools (e.g. Ansible) can execute need feedback from deployed application that is in produc- specific code snippets for the deployment. Monitoring and tion. logging of the started application is useful to detect every Recently, we are working on the dashboard tool which vi- kind of runtime phenomenon and orchestrate the application sualizes the runtime circumstances for the developers and seamlessly [3]. architects. The tool requires runtime circumstances from However, tools landscape is missing good tools which are the production environment. In this paper, we introduce able to present the runtime performance of applications in our background mechanism which uses agents to retrieve staging or production environment regarding the changes of runtime information and send it to our tool. We present the source code. We are working on a dashboard tool to many specific agents that we have developed for this soft- visualize how the deployed application behaves in specific ware. Our approach deals with many useful services and environment. Many typical use-cases can be mentioned. tools, such as Docker and Tomcat. Does the memory consumption decrease when a feature’s new implementation is deployed? Which commit may cause Categories and Subject Descriptors a memory leak, if it is suspicious. Does the introduction of D.2.5 [Software Engineering]: Testing and Debugging; a new feature or API cause increase in the number of end- D.2.8 [Software Engineering]: Metrics users? How can one compare the performance of the system if the webserver or a database server is replaced? Keywords For our dashboard tool, we have developed many tool- specific agents to report runtime perception. Our tool vi- Agents, Monitoring, DevOps sualizes the reports come from agents. We have developed agents that deal with Docker, Tomcat webserver, etc. In this 1. INTRODUCTION paper, we present our agent-based approach and illustrate DevOps is an emerging approach in modern software en- some agents’ internal high-level functions. gineering. The key achievements of DevOps are compre- This paper is organized as follows. In section 2, we briefly hensive processes from building source to deployment, con- present the main concept of our tool. After, we present our tinuous synchronization of development and operations in agent-based approach in a detailed way with some examples order to make every new feature delivered to the end users. in section 3. Finally, this paper is concluded in section 4. DevOps emphasizes the feedback from every phase. DevOps-culture uses a wide range of software tools. Au- 2. DASHBOARD TOOL tomation of build processes is essential solution for many years. Continuous Integration (CI) servers track the version A safe software development requires control over the en- control system if a change of the source has been commited tire software development lifecycle (SDLC). During the de- [7]. In this case, the CI server (e.g. Jenkins [1]) starts the velopment, it is essential to avoid memory leakage, or overuse compilation process and executes the test cases and finally, of the CPUs. To get a good overview of the resource uti- lization engineers, DevOps engineers have to keep their eyes on these units that means they have to monitor their envi- ronments by using tools that can reflect the status of the Permission to make digital or hard copies of all or part of this work for different services, databases, network I/Os, or the amount personal or classroom use is granted without fee provided that copies are of written/read blocks. not made or distributed for profit or commercial advantage and that copies In this chapter, we would like to give a brief introduction bear this notice and the full citation on the first page. To copy otherwise, to about our Dashboard tool which can help developers to get republish, to post on servers or to redistribute to lists, requires prior specific metrics about their environments. Developers can declare permission and/or a fee. new environments on the board and assign charts to them. CSS ’18 Ljubljana, Slovenia Copyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$15.00. A chart represents a single observable unit from the real en- 359 vironment. Metrics are provided by agents which run on the period. Beside these steps, an agent also has minor charac- machine where the application is deployed. A continuously teristics, like running agents send the gathered information back to the • Listener. This way software and DevOps engineers can get Runs as a daemon an accurate picture immediately. A screenshot can be seen • Validates the configuration file to have proper keys in Figure 1 about how a chart looks like. • Validates the values in the configuration yml file • Checks whether the related OS-level dependencies ex- ist • Transfers the collected metric to JSON Beside these steps, an agent also has minor characteristics. It runs as a daemon. It checks whether the related OS- level dependencies exist. It transfers the collected metric in JSON. All agents require a file that contains specific information for the observed unit, as well as, parameter for the connec- tion to the Listener. One file can be used by many agents, and one file can contain configurations for multiple observed units. Here we detail some of the agents mechanism, how they Figure 1: Memory consumption of a Tomcat in- work and what information we can get from the unit. stance 3.1 Tomcat Tomcat is one of the most popular and widely-used ap- 3. AGENTS IN OUR APPLICATION plication server among Java developers. It provides a sim- In this section, we give a detailed view of how our agents ple dashboard-like landing page where software and DevOps work and what the main steps are that we kept in focus dur- engineers can manage the deployed packages. Via this page ing the implementation. Before we go through the agents those users, who are dedicated to enter the server, can check listed below, we would like to introduce system require- the state of their applications. This can be a simple health- ments. The target hosts are always based on Debian images, check, the number of threads or how much memory is avail- or any of its derivatives, like Ubuntu or Linux Mint. As we able for Tomcat to allocate more space for the applications. present later, we have strived to use as less dependencies as The tomcat agent monitors both the inside status page possible, like OS-related functionalities or commands. Most and the process itself as well. In the configuration file (see of the commands come with the basic OS, like Listing 2), DevOps engineer has to declare specific parame- ps, but some of the switches can be different on other OS, like ters. -eo is Unix syntax, but using axo is acceptable on both Unix and BSD uri : ’ l o c a l h o s t ’ OS, as well. po rt : 8 080 The architecture consists of a server, the Listener and u s e r n a m e : ’ admin ’ nodes which serve as hosts for the agents. In our solution, p a s s w o r d : ’ admin ’ an agent is responsible for the following steps: pid : 2 4 5 6 7 • After start, it runs endlessly Listing 2: Agent configuration file example • Collects the information about the observed unit If pid is not available, agent monitors the inside status • Transforms and if necessary aggregates the collected page only. An example metric that the agent is intended to data send towards the Listener can be seen in Listing 3. • Transfers the data towards the Listener server in JSON { format " s t a t u s ": { " jvm ": { At first, we have to start the agent with an agent-specific " m e m o r y ": { sub-command and a configuration file which contains all the " fr ee ": 233 564 5 , information that are necessary to observe the chosen unit " t o t a l ": 8 8 2 3 4 1 2 3 , (e.g. see Listing 1). " max ": 2 4 5 3 4 2 2 } $ tomcat - a g e n t s t a r t -- fil e c o n f i g . yml } , " c o n n e c t o r ": { Listing 1: Launching the agent " r e q u e s t I n f o ": { When it starts running, it validates the arguments and " m a x T i m e ": 12 , then parse and validates the file against the expected con- ... figuration settings that are required to the unit. Then it } , starts monitoring and collecting metrics in a specified time " t h r e a d I n f o ": { 360 " m a x T h r e a d s ": 1 , pa th : ’/ logs ’ " c u r r e n t T h r e a d C o u n t ": 1 , fi le : ’ obs erv ed ’ " c u r r e n t T h r e a d B u s y ": 0 f o r m a t : ’ S E V E R I T Y || ’ } n u m b e r _ o f _ l i n e s : 10 } } Listing 5: Example configuration for the log agent } The path tag is responsible for the path of the folder which is considered as a log folder and Listing 3: Example metric sent by the tomcat agent file is the observed unit. To distinguish an ERROR leveled message from other messages that contains the word error, engineers have to declare the 3.2 Docker format of the log. The last key is responsible for the number Containerization is new directive in virtualization: this fetched and forwarded messages. A sent message example lightweight approach supports operating system-based iso- sent can be seen in Listing 6. lation among different pieces of the application. Container- { ization is on the crest of a wave since Docker has been de- " l i n e s ": [ ".. ."] , veloped. Docker provides a systematic way to automate the " s e v e r i t y ": { fast deployment of Linux applications inside portable con- " in fo ": 655 , tainers [2]. " w a r n i n g ": 848 , The name of docker is basically almost equivalent of con- " e r r o r ": 2 , tainer for most of the engineers. Docker, just like Tomcat, " f a t a l ": 0 provides a calculation on how much memory it consumes or } what the total bytes of the received and transmitted data } is over the network for each container. These are the stats. Without declaring any specific container name in the config Listing 6: JSON message example sent by the log file, the agent sends information about all the containers at collection agent the same time that are shown up in the stats. An example message can be seen in Listing 4. Since an agent is run on a machine by an arbitrary user, the software, DevOps and test engineers have to take care { that the observed log can be any file depends on the privi- " c o n t a i n e r s " : [ leges of the user. { " pid ": 38 , 3.4 Host Machine " na me " : ’ j i n g l e _ b e l l ’ , The host machine which the agent is executed on, can be " cpu " : 1.86 , a real machine, a virtual machine or a container whether it " mem ": { is on local or on remote. Whichever the host machine is, " u s a g e ": " 1 6 8 . 2 M " , from the agent perspective they are the same. From inside " l i m i t ": " 1 5 . 4 3 G " , out it seems that machine has memory, CPU (or GPU), " p e r c e n t a g e ": 1 .06 hard disk and other resources. These resources are reachable } , for the agents that means agents can use them. Having a ... picture about the usage and consumption of these resources } are essential. ] With this agent, we can monitor the above-mentioned re- } sources and gather their metrics. These metrics are cumu- lated, agent takes, for example the total memory, the total Listing 4: JSON message example sent by agent swap memory or the size of the available space on the hard disk, regardless which processes use them. 3.3 Log Here we would like to give a view which metrics are taken One of the most important mirror of the status of an ap- during the agent’s execution. We arranged the resources plication is its logs. It could contain all the steps that an into three groups. All the metrics belong to the memory, or execution takes and provide those steps in different granu- CPU, or disk storage (volume). larity. The two main approaches in case of this agent are, first, 3.4.1 Memory get the last n messages from the log and forward it to the Memory has multiple parts from total to used to swap. Listener, and second, get the number of the different severity To get an accurate picture about the consumption we use, levels. The earlier can provide a view of the latest messages, multiple commands that can help calculating the usage of which is a talkative information based on the error or excep- the different parts. The agent uses free (see Listing 7), tion messages raised in the code. The latter one can show /proc/meminfo and the vmstat commands to get metrics the ratio of the different levels giving a clear overview how about the memory (see Listing 8). All of them provide in- much warnings or errors get hit during the execution. To formation about how much total memory is in that host, get these two metrics we mentioned above, engineers have what the size of the cached swap or how much memory is to use such a configuration seen in Listing 5. free or how much is available for allocating new processes. ... $ f ree - m 361 t o t a l u sed fre e ... $ df - t e x t 4 Mem : 1 5 8 0 2 5 485 570 7 ... F i l e s y s t e m 1 K - b l o c k s U s e d A v a i l a b l e Use % M o u n t e d on / dev / n v m e 0 n 1 p 5 1 2 0 4 6 2 0 6 4 7 7 2 5 9 4 9 2 3 7 0 4 0 3 9 6 68% / Sw ap : 204 7 0 204 7 Listing 11: Using the df command Listing 7: Using the free command { { " f i l e s y s t e m ": "/ dev / n v m e 0 n 1 p 5 " , " Mem ": { "1 k _ b l o c k s ": 1 2 0 4 6 2 0 6 4 , " t o t a l ": 15802 , " us ed ": 7 7 2 5 9 4 9 2 , " us ed ": 5485 , " a v a i l a b l e ": 3 7 0 4 0 3 9 6 , " fr ee ": 5707 , " use ": 68 , " s h a r e d ": 2088 , " m o u n t e d _ o n ": "/" " bu ff / c a c h e ": 4609 , } " a v a i l a b l e ": 789 4 } , Listing 12: Sent JSON message about volume usage " Sw ap ": { " t o t a l ": 2047 , " us ed ": 0 , 4. CONCLUSION " fr ee ": 204 7 DevOps is an emerging approach that aims at the symbio- } sis of development, quality assurance and operations. Devel- } opers need feedback from the test executions that CI servers support. On the other hand, no tools have been created that Listing 8: Sent message about memory consumption support feedback from the production enviroment to the de- velopers to follow up the code changes and its effect on the 3.4.2 CPU end-users and the production or the staging environment. There are plenty of tools that provide the opportunity to In this paper, we argue for a new tools into the DevOps monitor the usage of the CPU. Some of them are part of toolset. The aim of this tool is retriving and visualizing the default OS, then the rest come as a third-party tool and the runtime circumstances of deployed application because require installation with privileges. We took the focus on this information can be essential for the developers and ar- those tools that are part of the OS, or used in wide range, like chitects. For this tool, we have developed many agents to vmstat, or iostat (see Listing 9). Both tools can provide a collect the runtime performance information from specific picture of the CPU utilization in percentage. services. In this paper, we presented the mechanism of some $ i o s t a t - c specific agents in Linux environment. L i n u x 4.15.0 -32 - g e n e r i c 2018 -08 -25 _ x 8 6 _ 6 4 _ (8 CPU ) avg - cpu : % u s e r % n i c e % s y s t e m % i o w a i t % s t e a l % i d l e 5. REFERENCES 24 ,97 0 ,03 6 ,07 0 ,03 0 ,00 68 ,90 [1] Jenkins. https://jenkins.io/. [2] D. Bernstein. Containers and cloud: From LXC to Listing 9: Using the iostat command Docker to Kubernetes. IEEE Cloud Computing, The agent sends the above information towards the Lis- 1(3):81–84, Sept. 2014. tener as it seen in Listing 10. [3] P. P. I. Langi, Widyawan, W. Najib, and T. B. Aji. An { evaluation of twitter river and logstash performances as " us er ": 24.97 , elasticsearch inputs for social media analysis of twitter. " ni ce ": 0.03 , In Information Communication Technology and " s y s t e m ": 6.07 , Systems (ICTS), 2015 International Conference on, " i o w a i t ": 0.03 , pages 181–186, New York, Sept 2015. IEEE. " s t e a l ": 0.00 , [4] M. Leppänen, S. Mäkinen, M. Pagels, V. P. Eloranta, " id le ": 68. 9 J. Itkonen, M. V. Mäntylä, and T. Männistö. The } highways and country roads to continuous deployment. IEEE Software, 32(2):64–72, Mar 2015. Listing 10: Sent JSON message about CPU usage [5] Á. Révész and N. Pataki. Containerized A/B testing. In Z. Budimac, editor, Proceedings of the Sixth Workshop 3.4.3 Volume on Software Quality Analysis, Monitoring, Volume usage does not belong to the major metrics of Improvement, and Applications, pages 14:1–14:8. the previously mentioned three units. Though it can tell CEUR-WS.org, 2017. useful information about a running application. To get a [6] J. Roche. Adopting DevOps practices in quality metric about the volume agent uses df (see Listing 11) and assurance. Commun. ACM, 56(11):38–43, Nov. 2013. du commands. Both of them are responsible for giving a [7] S. Stolberg. Enabling agile testing through continuous view of how much space is taken by a folder or how the integration. In Agile Conference, 2009. AGILE ’09., size of the local storage changes. Moreover, agent can be pages 369–374, New York, Aug 2009. IEEE. parameterized. It takes the path to the observed folder or partition of the storage of type of the disk. The agent sends aggregated information as it seen in Listing 12. 362 Incremental Parsing of Large Legacy C/C++ Software Anett Fekete, Máté Cserép Eötvös Loránd University Faculty of Informatics Budapest, Hungary {hutche, mcserep}@inf.elte.hu ABSTRACT incremental parsing [14] and the lazy analysis [10] have been CodeCompass is an open source project intended to sup- studied. A great overview of pratical algorithms and the port code comprehension by providing textual information, exsiting methodology is given by Tim A. Wagner in [13]. source code metrics, version control information and visu- C/C++ language-specific compilation tools [12, 4] and pro- alization views of the file and directory level relations for gramming environments [7] supporting incremental parsing the analyzed project. Regarding the typical software de- have also emerged as an advancement. velopment methodologies (especially the agile ones), only a smaller portion of the code base is affected by any change CodeCompass [9] is an open source, scalable code compre- during a shorter amount of time (e.g. between nightly hension tool developed by Ericsson Ltd. and the Eötvös builds), therefore parsing the entire project each time is un- Loránd University, Budapest to help understanding large necessary and expensive. A newly introduced feature, in- legacy software systems. Its web user interface provides rich cremental parsing is intended to solve this problem by only textual search and navigation functionalities and also a wide processing files that have been recently changed and leaving range of rule-based visualization features [5, 6]. The code the rest alone. This is achieved by the maintenance of the comprehension capabilities of CodeCompass is not restricted project workspace database followed by the partial parsing to the existing code base, but important architectural infor- of the project. The feature has been tested both on medium mation are also gained from the build system by processing and large scale projects and proved to be an effective tool the compilation database of the project [11]. The C/C++ in CodeCompass. static analyzer component is based on the LLVM/Clang parser [1] and stores the position and type information of Categories and Subject Descriptors specific AST nodes in the project workspace database to- D.2.3 [Software Engineering]: Coding Tools and Tech- gether with further information collected during the parsing niques; D.3.4 [Programming Languages]: Processors process (e.g. the relations between files). By introducing the concept of incremental parsing into CodeCompass we General Terms can detect the added, deleted or modified files in the pro- gram and carry out maintenance operations for the database Management, Languages of the code comprehension tool in only the required cases. Thus the required time of the reanalysis can be reduced by Keywords multiple magnitudes. code comprehension, software maintenance, static analysis, incremental parsing, C/C++ programming language In this paper first we present our research in Section 2 on how we extended the static analytical capabilities of the 1. INTRODUCTION CodeCompass code comprehension tool with incremental One of the main tasks of a code comprehension software parsing. Then Section 3 demonstrates the usability of the tool is to provide exact textual information and visualiza- concept by showcasing incremental parsing and measuring tion views regarding the analyzed codebase to support the its performance on a medium and a large size C/C++ soft- (newcomer) developers in understanding the source code. ware. Finally, Section 4 concludes the results and discusses For an enterprise software under development this requires further research opportunities. the frequent static reanalysis of the program, which could take several hours for a large legacy software. Performing a complete static analysis each time is a signif- 2. METHODOLOGY icant waste of computational resources, since in most cases A major consideration of the introduced incremental pars- (e.g. between nightly builds) only a few percent of the file ing feature was to integrate it seamlessly into the existing set has been affected by any change. In order to boost the parsing process by not differentiating in how an initial or a parsing and compilation process and to provide richer user follow-up incremental parse should be initiated. This was experience in integrated development environments (IDEs) achieved by utilizing the partial parsing feature of Code- [8], the concept of incremental parsing and compilation has Compass, which means that the tool is capable of continu- been researched since decades. More recently further ap- ing a previously aborted analysis, by omitting the already proaches, like the involvment of version control systems into parsed files which are present in workspace database. 363 Therefore the main concept of the introduced incremental parsing feature consists of two steps: i) perform a database maintenance operation, where the project workspace is re- stored into a state that ii) the existing partial parsing can finish the procedure. 2.1 Determining file states When a new parse is being done in incremental mode, the state of each file is determined first. Let FDB be the file set stored in the workspace database and FDISK be the file set stored on the disk. An f ∈ FDB ∪ FDISK file may take one Figure 1: Traversal directions of the three states listed as follows. Proof. Let G = (V, W, E) be the directed acyclic graph Added files f is added to the project since the latest parse (DAG) of header inclusions with V containing the file set as if f ∈ FDISK but f / ∈ FDB. vertices and E being the set of upward connections, n := |V |, e := |E|. Let W ⊆ V denote the set of directly changed files, Deleted files f is deleted from the project if f ∈ FDB but k := |W |. f / ∈ FDISK . Modified files f is modified when f ∈ F Let NG(v) be the neighborhood file set of vertex v in G, DB ∩ FDISK at the time of the new parse but its content has changed so w ∈ NG(v) ⇔ (v, w) ∈ E. Therefore for a file v we can since the latest. This can be determined by comparing define the directly included file set as NG(v) and the includer the contents that are stored in the database and on files of v as NGT (v), where GT is the transpose graph of G. the disk, or by their respective hashes for performance optimization. We define up(G, v) and down(G, v) as the file set result of the upward and downward traversal for v ∈ V in G by the 2.2 Header inclusion traversal corresponding traversal model, as formally described below: Specifically when parsing a C or C++ language project, up(G, v) = {v} ∪ ∀w∈NG(v) : up(G, w) (1) changes in header inclusions provide one more challenge to tackle. Upon the modification of a header file all further down(G, v) = {v} ∪ ∀w∈NGT (v) : down(G, w) (2) files in the inclusion chain depending on it should be consid- ered as modified, even without containing any direct changes As a simplification in our model lets assume a uniform themselves. Therefore when determining the modified state distribution of header inclusions among the files. Since of a file as defined in Section 2.1, the set of files defined P deg+(v) = P deg−(v) = e, the average in-degree by the header inclusion relationships transitively should be v∈V v∈V and out-degree for a file v is deg+(v) = deg−(v) = e , which checked for changes. There are two approaches for this, as n will be denoted with d henceforth. As a consequence the described below and shown in Figure 1. length of the longest path in G is logdn, which is the length of the longest header inclusion chain in the project, since G Definition 1. For files a, b and c, given that a is included was defined as a DAG. by b and b is included by c, we say that file a is in an upward connection with b and accordingly file c is in a downward Therefore the asymptotic tight bound both for up(G, v) and connection with b. down(G, v) can be calculated as: Θ(up(G, v)) = Θ(down(G, v)) = dlogdn = n (3) Upward traversal model The upward traversal model depends on the upward connection between files. We define up(G) and down(G) as the upward and downward When resolving the state of file a, its included headers traversal algorithms which determines indirectly changed have to be checked for modifications transitively. files in V through header inclusions from W by the cor- responding traversal model. We define the computational Downward traversal model Similarly, the downward complexity of the algorithms as the number of files checked traversal model uses the downward connections that for changes in their content (or by their hash). Based on can be found between files. If a file a is resolved as Equation 3, the asymptotic tight bound both for up(G) and modified, all files that include a can be marked as mod- down(G) can be calculated as: ified transitively. Note that with this method, the state of any marked files can be considered final and can be X Θ(up(G)) = Θ(up(G, v)) = n2 (4) omitted from further inspections. v∈V X Θ(down(G)) = Θ(down(G, w)) = k ∗ n (5) w∈W Theorem 1. The downward traversal model has better computational complexity over the upward traversal model, and therefore is preferred to be used through the incremental Since k ≤ n and in a typical use case for incremental parsing parsing. k n: Θ(down(G)) < Θ(up(G)). 364 An example for the downward traversal model is showcased in Figure 2. On the left side of the figure the example file set Table 1: Time measures for incremental parsing the is shown with header inclusion dependencies denoted as ar- Xerces-C++ project rows between them. Directly modified files are marked with Parse type Changed files Time a dark background, while files requiring expansion through Full parse – 2 min 49 sec traversal to find indirectly changed files are marked with an 1% change 3 10 sec italic font. Note, that these two categories are equivalent in 5% change 17 21 sec the initial stage. On the right side of the figure the effects 10% change 35 49 sec of downward traversing a.h is demonstrated: files c.h, d.h, f.cpp and g.cpp are also detected as indirectly changed files. While c.h was also a directly modified file, observe that it Table 2: Time measures for incremental parsing the no longer requires downward traversal. LLVM project by one atomic transaction Parse type Changed files Time 2.3 Database maintenance Full parse – 5 h 46 min As mentioned above, incremental parsing includes some 1% change 28 7 min 30 sec maintenance of the existing database depending on the state 5% change 142 1 h 58 min of changed files. 10% change 284 2 h 45 min 1. Added files are perceived as new files to the project • Carry out all deletions from the database in one single and therefore are registered into the database. transaction, so the maintenance is either completely 2. Deleted files need to be purged from the database as executed, otherwise no changes are performed. they have been removed from the project. • Generate multiple file level transactions, so informa- 3. Modified files are handled as if they were a combina- tion regarding a file is either cleaned from the database tion of deleted and added files. First, they are com- or the file is untouched, therefore a consistent state of pletely wiped out from the database – meaning that the database is always kept. all their AST related information and file level rela- tions are erased –, thus considering them deleted, then re-registered like newly added files. Directory level re- Table 2 and Table 3 compare the differences when the lations are not sufficiently maintainable, but these rela- database maintenance is executed through a single and by tions can be effectively computed runtime, on-demand file level transactions. It is clear that the extensive size of from the file level relations. the database rollback log containing all the deletion oper- ations for a larger quantity of files can significantly hinder 3. EXPERIMENTAL RESULTS the effectiveness of incremental parsing, providing signifi- The go-to projects on which CodeCompass is usually tested cant difference in the timespan of incremental parsing for are the Xerces-C++ [3] and LLVM [2] projects. Both large size projects like LLVM. Hence while a single transac- are open source projects that have been under develop- tion may provide stronger guarantees, file level transaction ment for several years and therefore are considered legacy proved to be a more adequate solution, where the required projects. Incremental parsing was also tested on these two time is more or less linear with the quantity of parsed files, as Xerces-C++ is a medium size and LLVM is a large-scale depending on the length and content of the files in question. project and contain enough files (respectively 347 and 2845) to produce a significant difference in runtime between even small portions of changes in the number of files. Table 3: Time measures of incremental parsing the LLVM project by file level transactions Incremental parsing is aimed to reduce the parsing time of Parse type Changed files Time builds, especially nightly builds, therefore it was tested on 1% change 28 9 min 30 sec 1, 5 and 10 percent change of the file set, since no bigger 5% change 142 49 min difference between two builds is presumable. The changeset 10% change 284 1 h 21 min was generated automatically by random selection of files.1 Table 1 shows the results for Xerces-C++, while Table 2 and Table 3 depict the results for LLVM. All measurements 4. CONCLUSIONS were carried out on a standard notebook computer, parsing Incremental parsing was introduced into CodeCompass to on 2 processor cores. reduce the costs of parsing, both time and computational resources, by omitting unchanged files in the project. The In order to keep database consistency in case of a graceful feature distinguishes added, deleted and modified files and abort or unexpected termination of the parser module, the handles them accordingly. The early tests of incremental basic concept is that the maintenance operation of incre- parsing were run on the Xerces-C++ and LLVM projects mental parsing must be performed in a transactional mode, and showed that it works according to its original purpose, in one of the following ways: especially in decreasing the timespan of parsing. While 1Only leaf nodes from graph G introduced in Section 2.2 the results are promising, further challenges include the im- were included in the changeset, so header inclusions did not proved reduction of the timespan required by incremental affect the number of changed files. parsing through parallelizing the process. 365 Figure 2: Downward traversing of a.h demonstrated on a showcase file set. 5. ACKNOWLEDGMENTS [12] T. Tromey. Incremental compilation for GCC. In This work is supported by the European Union, co-financed Proceedings of the GCC Developers’ Summit. Citeseer, by the European Social Fund (EFOP-3.6.3-VEKOP-16- 2008. 2017-00002). [13] T. A. Wagner. Practical algorithms for incremental software development environments. PhD thesis, Citeseer, 1997. [14] T. A. Wagner and S. L. Graham. Efficient and flexible 6. REFERENCES incremental parsing. ACM Transactions on [1] Clang: a C language family frontend for LLVM. Programming Languages and Systems (TOPLAS), https://clang.llvm.org/. 20(5):980–1013, 1998. [2] The LLVM Compiler Infrastructure. https://llvm.org/. [3] Xerces-C++ XML Parser. https://xerces.apache.org/xerces-c/. [4] Zapcc – A (Much) Faster C++ Compiler. https://www.zapcc.com/. [5] T. Brunner and M. Cserép. Rule based graph visualization for software systems. In Proceedings of the 9th International Conference on Applied Informatics, pages 121–130, 2014. [6] M. Cserép and D. Krupp. Visualization Techniques of Components for Large Legacy C/C++ software. Studia Universitatis Babes-Bolyai, Informatica, 59:59–74, 2014. [7] M. Karasick. The Architecture of Montana: An Open and Extensible Programming Environment with an Incremental C++ Compiler. SIGSOFT Softw. Eng. Notes, 23(6):131–142, Nov. 1998. [8] R. Medina-Mora and P. H. Feiler. An incremental programming environment. IEEE Transactions on Software Engineering, (5):472–482, 1981. [9] Z. Porkoláb, T. Brunner, D. Krupp, and M. Csordás. Codecompass: An open software comprehension framework for industrial usage. In Proceedings of the 26th Conference on Program Comprehension, ICPC ’18, pages 361–369, New York, NY, USA, 2018. ACM. [10] V. Savitskii and D. Sidorov. Fast analysis of source code in C and C++. Programming and Computer Software, 39(1):49–55, 2013. [11] R. Szalay, Z. Porkoláb, and D. Krupp. Towards better symbol resolution for C/C++ programs: A cluster-based solution. In IEEE 17th International Working Conference on Source Code Analysis and Manipulation (SCAM), pages 101–110. IEEE, 2017. 366 Visualising Compiler-generated Special Member Functions of C++ Types Richárd Szalay Zoltán Porkoláb Eötvös Loránd University, Faculty of Informatics Eötvös Loránd University, Faculty of Informatics Department of Programming Languages and Compilers Department of Programming Languages and Compilers Budapest, Hungary Budapest, Hungary szalayrichard@inf.elte.hu gsd@elte.hu ABSTRACT expressed, yet relied upon by the most trivial codes. What’s more, In the C++ programming language, special member functions are the compiler is free to lazily evaluate the generation of these mem- either user-defined or automatically generated by the compiler. bers, which results in one such member’s non-availability to only be The detailed rules for when and how these methods are generated reported when its usage was attempted. In case the used software are complex and many times surprise developers. As generated library is outdated or not easily modifiable, or not open source, this functions never appear in the source code it is challenging to com- can result in loss of run-time performance or development effort prehend them. For a better understanding of the details under the wasted on having to redesign parts of the software. For discovery hood, we provide a visualisation method which presents gener- and understanding of the existence and behaviour of these methods, ated special functions in the form of C++ source code that in effect developers can either consult the Language Standard, read Abstract identical to their implicit versions. Syntax Trees (ASTs), or view the disassembly of the binary — none of which is favourable for the average developer. CCS CONCEPTS 1 # i n c l u d e < i o s t r e a m > • Software and its engineering → Source code generation; 2 s t r u c t A { i n t x } ; Software maintenance tools; • Human-centered computing 3 i n t main ( ) { → Information visualization; 4 A a1 ; / / <− D e f a u l t c o n s t r u c t o r c a l l e d . GENERAL TERMS 5 a1 . x = 5 ; 6 A a2 ( a1 ) ; / / <− Copy c o n s t r u c t o r c a l l e d . programming languages, software development, visualisation 7 a1 . x = 6 ; KEYWORDS 8 9 / / W i l l p r i n t " 6 5 " . C++ programming language, compilers, code comprehension, code 10 s t d : : c o u t << a1 . x << " ␣ " << a2 . x ; design 11 } 1 MOTIVATION Listing 1: Example code which uses a default and a copy con- structor. Languages supporting the Object-oriented programming (OOP) paradigm define a central principle of object lifetime which is sur- To aid ongoing development and code comprehension of projects rounded by construction/initialisation and destruction/finalisation. we introduced a tool that allows pretty-printing the visual represen- In the Java programming language, apart from the basic default tation of special member functions that is the closest to how they construction – where everything is initialised to the respective zero would be written by developers. To further this aid, we don’t only value – the developer must explicitly state their intent for different show the compiler-generated special members, but provide a subset construction logic, custom finalisation. A special case is when a of the type’s all member functions which shows both user-written new object is created from an already existing one, where deep copy – e.g. a constructor that initialises from a different data type – and (clone) operations or conversions might be warranted. In C++, how- the standard, implicit ones. We used the open source LLVM/Clang ever, the Language Standard specifies that these aforementioned Compiler Infrastructure [16] for parsing and generation. actions, in the form of special member functions [8], should have a The rest of the paper is organised as follows. In Section 2 we default implementation automatically generated by the compiler discuss the purpose and rules of C++ special member functions. if the user does not explicitly write them. The rules which dictate Then, Section 3 describes the implementation approach and chal- the conditions for generating the special member functions and lenges faced with respect to pretty-printing and presentation to the their behaviour can appear dauntingly complex, and subsequent developers. The paper concludes in Section 4. versions of the language standard may revise and elaborate these rules, increasing their complexity. The most recent, and most sig- nificant such change was with the release of the C++11 standard, 2 C++ SPECIAL MEMBER FUNCTIONS which introduced move semantics [9]. Special member functions in C++ denote the functions that are Modernising code initially written for an older standard can be necessary for the management of instances’ lifetime. [12] These cumbersome as the behaviour of special members are never directly are the constructors, the assignment operator and the destructor. 367 CSS’2018, October 2018, Ljubljana, Slovenia Richárd Szalay and Zoltán Porkoláb 2.1 Constructors data member’s destructor is explicitly hidden – this is a common Constructors are responsible for the initialisation of an object. They practice for scenarios where a controller has to ensure an orderly are usually executed together with the memory allocation for the or batch destruction. instance. Unless the user specifies and provides any constructor function, both C++ and Java will generate a default constructor. In Java, this function initialises every data member to their respective 2.3 Assignment operators zero value, such as integer 0, rational 0.0, the \0 character, or a null reference. In C++, the initial state of the members depend on Contrary to Java, where there exists only primitive types and ref- the storage scope of the object – in most cases, the memory garbage erences, C++ is a language with value semantics. Assigning to a is retained from the memory block where the object is allocated. reference in Java only results in the actual memory modification Unlike Java, however, the default constructor is not created if at of a memory address’ size. The object that is no longer referred by least one data member does not have a default constructor. the assigned-away reference is then left for garbage collection, if Another case of construction is when a new object is initialised applicable. In C++, however, this means that assigning an object from the state of another, already living object of the same type. In to another object of the same type results in the assigned-to object Java, this functionality can be achieved in multiple ways, one of having the assigned object’s state’s copy within its own memory which is by using the special clone() function. This function is region. Traditionally, copy assignment operators have a “destructor” defined in Object, and performs a shallow copy of the instance in part where the current object’s resources and buffers are released, question, only initialising the new object’s members to the same and then a “copy constructor”-like logic where the copy of state value of the cloned one [4, 11]. In case of references to other objects takes place, however, the developer is free to choose a different results in aliasing, the sharing of the same resource – usually an implementation. The compiler-generated copy assignment operator internal buffer – by two separate entities. Another problem with implements a memberwise copy assignment for the entire object. clone() is that the existence of the cloneability marker and the Thus, the copy assignment operator is not generated by the com- respective method must exist through the whole chain of the type piler due to type infeasibility if one of the data members cannot be hierarchy – it is usually referred to as an epidemic [10]. What’s more, copy-assigned. cloning does not actually invoke a construction, but rather creates It is noteworthy to mention that not every language has defined a copy of the memory’s snapshot, which means that business logic the = assignment operator as an operator: in some languages, such strictly bound to a constructor, such as initialisation of read-only as Ada or Pascal, assignment is defined as a statement/instruction, members, cannot be done. In C++, the default behaviour of the copy rather than an operator application. This has led to the inability constructor is to run the copy construction of every data member. to write copy assignment logic in Ada. To avoid use of assignment For fundamental types, this means a copy of the value, and for more on types that are not designed for memberwise copy the limited complex types their respective copy constructors are called. Thus, keyword [18] and type-annotation is used. in case a custom resource which can be properly deep-copied is In C++ it is commonly referred as The Rule of Three that if any used the copy constructor that is generated for the object using this of the copy constructor, copy assignment operator, and destruc- resource will be sufficient. tor is written explicitly by the developer, all of them should be written explicitly. This rule of thumb is not enforced by compil- ers but considered a good practice, because, as discussed earlier, 2.2 Destructor explicitly specifying either will not stop the compiler from automat- The destructor or finalise is called at the end of an instance’s life- ically creating the implicit definitions of the other special member time and is responsible for tearing down the state of the instance. functions. This most commonly means releasing resources, performing clean- up tasks and committing changes, e.g. to a database. In Java, the finalize() method’s implementation is run for an object at an unspecified point in time when the runtime’s garbage collector de- 2.4 Members for move semantics cides that the object is to be reaped. [3] The behaviour differences The release of the C++11 Language Specification has introduced between Java Virtual Machine versions and the general looming move semantics, which allows resources to be directly “stolen” by a of a finalisation never happening for an instance resulted in a con- variable from another, as opposed to a copy-constructed and the sensus on not using finalize() – it has also been deprecated original data’s memory destroyed. [13] This is used heavily with since Java 9. Instead, the AutoCloseable design pattern is used that temporary objects which would get destroyed in the next statement. explicitly requires writing a close() method which executes tear- The move special members’ default implementation executes a move down logic, but can be called arbitrarily by the developers when construction or move assignment of every data member, however, teardown is deemed necessary, such as at the end of finishing a the rules for their existence are more exquisite. Move members database operation. In C++, a destructor can be written by the are not generated automatically if any explicit destructor, copy or user or is automatically generated by the compiler. It is always move member exists, and an explicitly defined move member also executed immediately when an instance’s lifetime ends. The gener- turns off the automatic generation of copy members. ated destructor does nothing in its body, and then the destructor Accordingly, the Rule of Three has been extended to also include of each data member is executed individually – as their lifetimes the two move members, and is referred to as The Rule of Five. also expired. Thus an implicit destructor always exists unless a 368 Visualising Compiler-generated Special Member Functions of C++ Types CSS’2018, October 2018, Ljubljana, Slovenia 3 IMPLEMENTATION lists [5] too. The AST nodes found in the subtrees of these nodes are 3.1 Syntax transliteration then manually converted into a textual, source code representation. We used the open source LLVM/Clang Compiler Infrastructure for s t r u c t A { parsing and generation of special member visualisations because A ( ) { } / / The d e f a u l t c o n s t r u c t o r . Clang’s object-oriented Abstract Syntax Tree (AST) API allows for / / The c o p y c o n s t r u c t o r . an optimised and maintainable application. An example subtree of A ( c o n s t A & r h s ) : x ( r h s . x ) { } the AST corresponding to the source code in Listing 1 can be seen } ; in Listing 2. The copy constructor’s body corresponds to copying the Listing 4: The special members of the example class in List- right-hand record’s single data member into the current record’s ing 1 translated back to source text. corresponding data member. CXXConstructorDecl There are three interesting cases that need to be noted, where i m p l i c i t u s e d c o n s t e x p r A v o i d explicit source code differs from what a compiler generates for itself ( c o n s t s t r u c t A &) n o e x c e p t i n l i n e automatically. First of all, the compiler generates the implicit mem- bers’ arguments without an argument name. One such example ParmVarDecl 20 f 9 0 c 0 u s e d c o n s t s t r u c t A & can be seen in Listing 2, where the ParmVarDecl (parameter vari- C X X C t o r I n i t i a l i z e r F i e l d x i n t able declaration) has no name, and the initialiser’s DeclRefExpr I m p l i c i t C a s t E x p r i n t < LValueToRValue > (declaration reference expression) only refers to this ParmVarDecl MemberExpr c o n s t i n t l v a l u e . x by its memory address, 20f90c0. Such a construct cannot exist in DeclRefExpr c o n s t s t r u c t A actual source code. As a remedy, we manually assign the name rhs l v a l u e ParmVar 20 f 9 0 c 0 ' ' to the variable – or in case multiple parameters would be possible, c o n s t s t r u c t A & number them as arg_1, arg_2, . . . – and use it in the pretty-printed CompoundStmt code. Another such interesting case is about move constructors and CXXConstructExpr < c o l : 7 , c o l : 1 1 > s t r u c t A move assignment operators, namely that the compiler generates the v o i d ( c o n s t s t r u c t A &) n o e x c e p t argument as a temporary, an xvalue, from which move operations can be done. However, T&& rhs written in source code specifies Listing 2: The Clang AST representation of the implicit copy a named variable, an lvalue, from whose members move must ex- constructor’s body, and the call to it in main(). plicitly be specified by using a type annotation std::move, which Other compilers, might use different internal representations, on casts the members to be xvalues which denote variables that are which these transformations would be infeasible to execute – in case essentially transformed into a temporary and their resources can of GNU/GCC, the Register Transfer Language (RTL) is only meant be moved from. The pretty-printer annotates the right-hand sides to be used by compiler-internal applications and code generation is of move initialiser or assignment expressions with std::move to organised into various steps called loops. The example of the same ensure the same semantics. We only do this for record types, as no copy construction can be seen in Listing 3, which has already been fundamental type supports move operations. stripped of semantic information and only the memory access for The third case is with regards to inheritance. In case a class has the data member can be studied from it by humans. It should be at least one superclass, the special members’ default behaviour is to noted that the presented representation is the earliest and shortest cast the current instance to the base class and call the appropriate where copy construction is apparent on the inner data member level. constructor or assignment operator for each base class. A core Previous transformation loops only show the copy constructor’s principle in object-oriented programming is that up-casting – cast call source line in it’s original form, i.e. A a2(a1);. to any base class – is always possible and well-defined, however, this would result in unintelligible source code lines, such as *this ( insn 7 6 8 2 = rhs; – which would lead to an infinite recursion if written in ( s e t (mem/ c : S I ( p l us : DI ( reg / f : DI 82 source code verbatim. The type system allows us to see that this = v i r t u a l − s t a c k − v a r s ) is for the base class, so we explicitly wrap the statement into a cast ( c o n s t _ i n t −8 [ 0 x f f f f f f f f f f f f f f f 8 ] ) ) at the appropriate location to show base class initialisation to the [ 1 a2 +0 S4 A64 ] ) developer. Examples of these cases are depicted in Figure 1. ( reg : S I 9 1 ) ) " / tmp / main . cpp " : 6 −1 We have encountered that the Standard only specifies generating ( n i l ) ) a body for a special member if the currently compiled translation unit ODR-uses [7] the function. While no compiler error is given at Listing 3: The GNU RTL of the copy constructor call in line compilation for an infeasible, implicit deleted special member unless 9 of Listing 1. used, the type system in Clang annotates the forward declaration We have utilised Clang’s architecture to perform a parsing on of the function if it is deleted. Thus by using this annotation and the translation unit, and then performed a traversal on the built the related diagnostics, we can, for each member without a body AST searching for all records, or a particular record with a name achieve either an explicit body generation or printing the reason specified by the user. Once the record is found, we visit every special behind the member being deleted by the type system in a single pass. member’s body, and in the case of constructors their initialiser It should be noted that generating the body for members which are 369 CSS’2018, October 2018, Ljubljana, Slovenia Richárd Szalay and Zoltán Porkoláb 4 CONCLUSION In this paper, we have discussed the rules and behaviour of auto- matically generated special member functions, an intrinsic feature of the C++ programming language. We have introduced an ap- proach to transliterate the compiler’s internal representation of these special members to source text to promote understanding of software projects without resorting to unfavourable techniques such as reading syntax trees manually. We have implemented our solution in the open-source code com- prehension tool CodeCompass [1, 14, 15] — http://github.com/ Ericsson/CodeCompass — as an additional visualisation over C++ files. The upstreaming of this addition is underway at the writing of this paper. ACKNOWLEDGMENTS This work presented in this paper was supported by the Euro- pean Union, co-financed by the European Social Fund in project EFOP-3.6.3-VEKOP-16-2017-00002. REFERENCES [1] CodeCompass. 2012. A software comprehension tool for large-scale software written in C/C++ and Java. http://github.com/Ericsson/CodeCompass [2] Margaret Ellis. 1990. The Annotated C++ Reference Manual. Addison-Wesley, Reading, Massachusetts, USA. [3] James Gosling, Bill Joy, Guy L. Steele, Gilad Bracha, Alex Buckley, and Daniel Smith. 2017. Finalization of Class Instances (1st ed.), Chapter 12.6, 389–393. In [4]. https://docs.oracle.com/javase/specs/jls/se9/jls9.pdf visited on 2018-08-13. [4] James Gosling, Bill Joy, Guy L. Steele, Gilad Bracha, Alex Buckley, and Daniel Smith. 2017. The Java Language Specification, Java SE 9 Edition. https://docs. oracle.com/javase/specs/jls/se9/jls9.pdf visited on 2018-08-13. [5] ISO. 2012. Initializing bases and members, Chapter 12.6.2, [class.base.init]. In Figure 1: Special member overview for a class with two base [6]. http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm? csnumber=50372 classes and a single char data member. [6] ISO. 2012. ISO/IEC 14882:2011 Information technology — Programming languages — C++, version 11 (C++11). International Organization for Standardization, Geneva, Switzerland. 1338 (est.) pages. http://www.iso.org/iso/iso_catalogue/catalogue_ tc/catalogue_detail.htm?csnumber=50372 allowed to have one, and it is only an optimisation that generation [7] ISO. 2012. One definition rule, Chapter 3.2.3, [basic.def.odr]. In [6]. http://www. didn’t take place is a non-functional change and does not affect iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=50372 [8] ISO. 2012. Special member functions, Chapter 12, [special]. In [6]. http://www. the semantics of the generated code – thus this transformation can iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=50372 safely be integrated into other compilation steps. [9] ISO. 2012. Temporary objects, Chapter 12.2, [class.temporary]. In [6]. http://www. iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=50372 [10] Marián Juhás, Zoltán Juhász, Ladislav Samuelis, and Csaba Szabó. 2009. Measur- 3.2 Special member overview ing the complexity of students’ assignments. Annales Universitatis Scientiarum Budapestinensis de Rolando Eötvös Nominatae. 31 (2009), 203–215. To facilitate better code comprehension, we have decided not only [11] Zoltán Juhász, Marián Juhás, Ladislav Samuelis, and Csaba Szabó. 2008. Teaching to show the implicit special members but every related overload of Java programming using case studies. Teaching Mathematics and Computer constructors and assignment operators. This allowed us to show Science. 6(2) (2008), 245–256. [12] Stanley Lippman. 1996. Inside the C++ Object Model. Addison Wesley Longman, a subset of the class’ members which are related to the instance’s Reading, Massachusetts, USA. lifetime. [13] Scott Meyers. 2015. Effective Modern C++: 42 specific ways to improve your use of The full overview proves useful when a special member is de- C++11 and C++14. O’Reilly Media, Sebastopol, California, USA. [14] Zoltán Porkoláb and Tibor Brunner. 2018. The CodeCompass Comprehension faulted. If for example, a class contains some constructors and a Framework. In Proceedings of the 26th Conference on Program Comprehension user-defined copy constructor, the move members will not be gen- (ICPC ’18). ACM, New York, New York, USA, 393–396. https://doi.org/10.1145/ 3196321.3196352 erated automatically, however, the developer can explicitly ask the [15] Zoltán Porkoláb, Tibor Brunner, Dániel Krupp, and Márton Csordás. 2018. Code- compiler to generate the methods with the implicit body rules by Compass: An Open Software Comprehension Framework for Industrial Usage. using the = default specifier, available in C++11 and onwards. In Proceedings of the 26th Conference on Program Comprehension (ICPC ’18). ACM, New York, New York, USA, 361–369. https://doi.org/10.1145/3196321.3197546 This is the suggested approach for modern C++, practised by most [16] The LLVM Project. 2003. Clang: C Language Family Frontend for LLVM. http: open-source projects. In this case, we show these members’ body //clang.llvm.org visited on 2018-08-13. along with the rest of the class with the annotation that the user [17] Bjarne Stroustrup. 1994. The design and evolution of C++. Addison-Wesley, Reading, Massachusetts, USA. requested the body generation. [18] S. Tucker Taft, Robert A. Duff, Randall L. Brukardt, and Erhard Ploedereder. 2000. Another case for the full view is showing the reason why a Consolidated Ada Reference Manual: Language and Standard Libraries. Springer- Verlag, Berlin, Heidelberg, Germany. special member was not automatically generated by printing a hint from the semantic analysis’ diagnostics. 370 How Does an Integration with VCS Affect SSQSA? Bojan Popović Gordana Rakić Naovis d.o.o. University of Novi Sad, Faculty of Sciences Trg Bulevar oslobo ¯ denja 30A Dositeja Obradovića 4 Novi Sad, Serbia Novi Sad, Serbia bojan.popovic@primafin.com goca@dmi.uns.ac.rs ABSTRACT Consequently, software analysis tools integrate support for Contemporary trends in software development almost nec- VCS. Usually this support means possibility to analyze code essarily involve version control system (VCS) for storing and stored to VCS repositories. In some cases tools also rely manipulation of source code and other artifacts. Conse- on advantages of VCS to improve analysis performances or quently, tools supporting the development process such are results. software analysis tools integrate with VCS. In most of cases tools support only analysis of the resources in VCS reposi- In this paper we explore potential advantages of integration tories, while some of them rely on VCS to improve analysis of SSQSA (Set of Software Quality Static Analyzers) plat- process and results. In this paper we explore how an inte- form [9] with GIT [2] as a representative VCS. First, we gration of the SSQSA platform with VCS influences some of introduce a concise background by describing VCS (Section its performances. 2) and SSQSA (Section 3). Prerequisites for the integration and the integration are described in the section 4. We dis- Categories and Subject Descriptors cuss results in the section 5 and possible application models and scenarios in the Section 6, that is followed by comparison D.2.8 [Software Engineering]: Metrics—complexity mea- to related integration solutions (Section 7). We conclude the sures, performance measures paper in the Section 8. This paper is summary of a master thesis described in [8] (in Serbian). Keywords Software quality analysis, intermediate representation, Ver- 2. VERSION CONTROL SYSTEMS sion Control System Version control systems (VCS) might have very broad appli- cation in different areas of content manipulation for personal 1. INTRODUCTION or professional purposes. These are tools used primarily to Quality of a software product is observed through the level of support teams and individuals in development and mainte- satisfied requirements. It could be assessed by its execution nance of a software products. These systems remember all by applying different techniques of dynamic analysis. These the changes of separate files, so that at any time we can re- techniques are applicable when the product is ready for test- cover a specific version, or follow and compare changes over ing which might be late to recognize weaknesses or issues. the time. In this way, all data is safer, good synchronization On the other side, static analysis techniques are travers- between the team members is ensured, the possibilities for ing source code and its various intermediate representations errors are significantly reduced, and therefore the project which makes them applicable already in the early phases of development process is improved. software development process [5]. VCS are divided into two large groups [2]: Contemporary software development practice relies on source code repositories and their synchronization implemented by various version control systems (VCS). VCS are used to store CVCS: Centralized Version Control System where all the data the whole history of activities in the evolution of a software are stored to a centralized server. This approach is cer- product, from version information to the finest details about tainly easier to maintain, but in case of system failure, every individual change in the repository, including informa- all information about the project will be lost. Addi- tion about contributors to the changes. tionally, availability of a network connection is very important. Previously, this was the standard way to execute version control. Representatives of this group are CVS: Concurrent Versions System [4] and Subver- sion [3]. DVCS: Distributed Version Control System where clients map the whole repository. If a server failure occurs, any of the client repositories can be copied back to the server to restore it. However, local copy enables us to work on changes independently of a network connection while 371 Property Git Mercurial derived representations are generated based on eCST, by Simple GUI - + a unique implementation of the derivation process, ensures Getting started for beginners - + their language independence and universality, too. Simplicity branches visualization - + Speed (Windows OS) - + By traversing all or some of these universal intermediate rep- Speed online + - resentations different analysis algorithms are implemented. Changing the history + + Therefore, it is possible to have a single implementation of Using the index + - every functionality that we integrate in the SSQSA which PL independent extensions + - ensures consistency of the results across different languages, Repo. migrating to another system + - but also adaptability to a new language and extendability by a new analysis [9]. Described process and a corresponding Table 1: Comparison between Git and Mercurial platform design is illustrated by the Figure 1. Current version of the SSQSA platform manipulates input the connection is necessary only for saving changes at source code from an local directory (components colored by the remote repository or taking a version from it. Files gray color), while our primary goal in this research is to stored on the hard disk are of small size, and hence this integrate it to analyze the code stored in a Git repository. does not pose a problem problem of a storage space. Additionally, we will explore how usage of Git repository for storing intermediate representation affects SSQSA platform An additional advantage of DVCS is that we can share the and its performances. This level of the integration will en- changes with other team members before they are shared able us to traverse only changed fragments of the structures, globally. On the other hand, there is little advantage of cen- which might further lead to improvement of performances tralized systems compared to distributed ones. Centralized of the analyses. The first prototype includes only results of systems offer us an easier way to control all the people who generation of eCST in the repository. New components that access the server, as well as easy provision of a central point implement integration are yellow-colored in the Figure 1. where all the changes are in place. They also offer us the option of downloading only a piece of code, if we only need 4. THE SSQSA AND GIT INTEGRATION to work on a project module. However, if needed, one copy To enable collaboration of SSQSA with Git, it was neces- of the project in the DVCS can be announces as the main sary to connect eCSTGenerator to Git repository and to one, and thus we can simulate the centralized system. enable it to process the source code stored in it. After the first connection eCSTGenerator is processing the whole con- Distractions that can be addressed to distributed systems tent of the repository and generates its eCST representation. are more technical. For example, in case of a project with Every next time, eCSTGenerator will process only changed many large files that can not be compressed, more storage files. This feature was not easily implementable before the space is required. Additionally if we are working on a large integration with Git. project that contains many customized changes, download- ing a full version of the project can take longer than ex- In addition, SSQSA uses advantages of its integration with pected, and also take up more space on the hard drive than Git at one more level. Namely, after the set of eCST is expected. generated, it is stored to a Git repository so that other com- ponents can also process only changes between versions. For All described differences bring to the decision to conduct these purposes we do not use the same repository as it is a the first experimental integration SSQSA platform with a dedicated development repository, while developers do not DVCS. Therefore, we compare Git [2] and Mercurial [6] have to be affected by the analysis. as the main representatives of DVCS in order to compare their properties to our requirements (Table 1). We can conclude that Mercurial has better characteristics from the 5. RESULTS users point of view, but for our integration these character- To explore applicability aspects of the described integration istics do not have value. On the other hand easiness to inte- solution, we measure time needed for generation of eCST grate with other systems, possibility to migrate to an other representation of a JavaScript project ”proton-native”1. system and speed are extremely important to us. Therefore, in this work, we integrate SSQSA with Git. First, we observe time needed only for generation of eCST representation of the source code from the local folder and 3. THE SSQSA PLATFORM compare it to the time needed to generate it for the code The SSQSA (Set of Software Quality Static Analyzers) [9] stored in a Git repository (Table 2). is a set of tools that enables language independent static software product analysis based on its source code. Lan- As we can see, for the first commit generation process lasted guage independence is ensured by a universal intermediate for significantly longer time. The reason for this is time representation of a source code called eCST (enriched Con- needed for the connection to the Git repository. However, crete Syntax Tree). Once when this representation is pro- even though process spends additional time on the connec- duced for any system, written in any set of programming tion, in later commits we get better results from the version languages, it can be transformed to some of derived inter- integrated with Git. mediate representations such are dependency networks, at different abstraction levels, or flow graphs. The fact that 1https://github.com/kusti8/proton-native 372 Figure 1: SSQSA platform and its integration with Git Version from a from a Time for the ally works according to the pull-request model on the local no. of commit local dir Git repo Git connection level. 7. 744 ms 1250 ms 720 ms 14. 812 ms 1270 ms 754 ms The most practical model for implementing a new imple- 34. 1589 ms 1353 ms 739 ms mentation for the use of Git is the pull-request model. A 80. 1601 ms 1520 ms 870 ms project leader can start an eCST generator on a new repos- 126. 1650 ms 1515 ms 780 ms itory commit to analyze the modified file. If a developer wants to create XML trees, it can also launch an eCST gen- Table 2: Comparison between time needed for eCST erator at each commit. The problem can arise if more teams generation proccess from a local directory and from are made and the eCST generation process is lunched then a Git repository. only. In this case it must be adapted to go through all the commits, not only looking at the latest changes. Eventually, if we include functionality for committing of gen- The ”Director and Lieutenant” model is also suitable for new erated eCST to a repository, time needed for whole process implementation. Each sub-project has its own leader who goes over 6000ms. Obviously, in this scenario integration re- can create XML trees. Also, the leader of the repository duces performances of SSQSA. Still, further integration will may generate eCST when joining new changes to a branch utilize benefits of version control to improve generation of of a project (merge). Also, if developers want to generate derived intermediate representations. Finally, it will be in- XML trees, the same rules apply as with the Pull-Request tegrated with the analyzers. It can be expected that, with model. the growth of data that will be saved up in the exchange, traverse and analysis process, the benefits from the integra- The centralized model is the most unpractical model for us- tion will also grow. Therefore, effects of the integration on ing the new implementation. All team members commit other components still have to be explored (Section 8. their changes to a centralized repository, which in this case contains a lot of commits through which traversal should be 6. APPLICATION SCENARIOS conducted. Depending on a scenario, Git has three common application models: a centralized model, a pull-request model, and a Di- 7. RELATED SOLUTIONS rector and Lieutenants model [2]. In a centralized system, all Many tools also support code analysis from various VCS members of the team synchronize their changes in a central such as BCH: Better Code Hub2 and SonarQube3, primar- repository that stores all source code. In the pull-request ily because the repositories have become the standard code model, developers can make changes to his local repository, storage. However, only some tools rely on versions for more and he commits them to his own repository, and can see the advanced analysis. changes that other team members make. In this model one repository is considered the main repository. In order to ac- Lean Language Independent software analyzer (Lisa) is a complish the changes in it, a request is sent to the project software that analyzes the quality of software projects. The leader to pull the changes. The project leader can add devel- main goal of Lisa is to analyze a large number of project re- oper’s repository as a remote repository, locally test changes, visions asynchronously with minimal redundancy. Analyses and if everything is fine, save them to the main repository. are aimed to cover as many analyzes, and as many program- In a Director and Lieutenants model the project is divided into sub-projects and distributed among teams. Each team 2https://bettercodehub.com/ (sub-project) has its own repository and its leader, and usu- 3https://www.sonarqube.org/ 373 ming languages as possible. These goals are comparable Integration is developed at two levels. At the first level with the goals of SSQSA, as well as the new implementation the platform is connected to the Git repository in order to presented in this paper. However, Lisa currently supports enable processing source code stored in it. At the next level three programming languages, while the SSQSA framework of the integration we use Git repository to store XML file currently allows us to work with more than ten program- containing eCST intermediate representation of the source ming languages. Concerning the subject of this paper, We code so that we can always look only for changes, and not can note certain differences in the approach to the problem traverse all the code, or more precisely, eCST representation and the concrete solution implementation. For the needs of it. This is very important if we have in mind that one of the Lisa analyzer, a special interface called SourceAgent input file (compilation unit) is represented by one eCST. has been developed. It supports the asynchronous access to the Git repository and file revisions [1]. On the other At the first look, the results of the integration are not promis- hand, SSQSA, with the current implementation, uses all the ing. Namely, Git connection used the time that we can save benefits of the Git and the library for interactions with it, by looking only in the changes and not in the whole source looks at the differences between the last two committees, code. However, Without storing trees to the Git repository and reads all the files that have been changed, and gener- we are already saving some processing time. In case when ates XML trees for them. Furthermore, Lisa communicates we store eCST in a Git repository we are spending more directly with the Git repository by making a local copy of time but in the future work we will explore if this cost may the remote repository to a local hard disk, while our imple- be payed off after extending this integration on generation of mentation allows reading from a local disk and thus does derived representations and analyzers. For example, genera- not require an internet connection. Internet connection is tion of dependency network currently traverses all the trees only needed if we want to save the generated XML tree in a while after the full integration with Git it will also look only remote repository. for changes. The similar expectation we have from an inte- gration of analyzers with Git. Therefore, these integration The Analizo is a solution that analyzes source code written activities will be subject of the a future work, as well as in different programming languages, whose emphasis was on analysis of potential costs and benefits, and selection of the C, C ++ and Java. The analysis supports the reading of most suitable usage scenarios. content from remote repositories for each audit in which the source code has been changed in the project [10] and, un- 9. REFERENCES like the SSQSA which currently allows reading of contents [1] C. V. Alexandru, S. Panichella, and H. C. Gall. only from the Git repository, allows reading from the Git Reducing redundancies in multi-revision code analysis. and Subversion repositories, and then generates CSV files. In Software Analysis, Evolution and Reengineering SSQSA also compares file revisions and decides from which (SANER), 2017 IEEE 24th International Conference files to create an XML tree. An advantage over Analizo is on, pages 148–159. IEEE, 2017. that we can monitor file versions on a remote repository. [2] S. Chacon and B. Straub. Pro git. Apress, 2014. Again, the difference is in the number of supported lan- [3] B. Collins-Sussman, B. W. Fitzpatrick, and C. M. guages: Analyzo supports three languages, while SSQSA Pilato. Version control with subversion, 2006. currently supports more than ten programming languages. Accessible in URL: http://svnbook. redbean. com, 2007. [4] D. Grune et al. Concurrent versions systems, a EvoJava is a tool for static code analysis of an input from method for independent cooperation. VU Amsterdam. a Java repository. It uses a VCS to access the code, mines Subfaculteit Wiskunde en Informatica, 1986. the source repository, and calculates metrics. Unlike the SSQSA platform, EvoJava uses Subversion (SVN) and pro- [5] G. O Regan. Introduction to software quality. cesses only .java files. The output file is also in .XML for- Springer, 2014. mat, but containing metric results. EvoJava takes a list of [6] B. O Sullivan. Mercurial: The Definitive Guide: The the code versions that is in the repository and thus creates Definitive Guide. ”O’Reilly Media, Inc.”, 2009. a model based on the XML-generated files [7]. SSQSA, on [7] J. Oosterman, W. Irwin, and N. Churcher. Evojava: A the other hand, observes the latest changes that are commit- tool for measuring evolving software. In Proceedings of ted to a remote repository, finds these files in the file system the Thirty-Fourth Australasian Computer Science and creates XML files based on them. Later it automatically Conference-Volume 113, pages 117–126. Australian commits them to a special local or remote repository, where Computer Society, Inc., 2011. we can track what changes were made during the evolution [8] B. Popović. Integration of a platform for static of our software. We cane also note the variety in supported analysis with a version control system (in serbian). programming languages in SSQSA while EvoJava only sup- Master’s thesis, Faculty of Sciences, University of Novi ports Java programming language. Sad, 2018. [9] G. Rakić. Extendable and adaptable framework for 8. CONCLUSION AND FUTURE WORK input language independent static analysis. PhD thesis, Faculty of Sciences, University of Novi Sad, 2015. Following actual trends in software development and soft- ware analysis SSQSA frameworks goes into a direction of [10] A. Terceiro, J. Costa, J. Miranda, P. Meirelles, L. R. integration with VCS. In this paper we compare character- Rios, L. Almeida, C. Chavez, and F. Kon. Analizo: an istics of different VCS and select Git as a first candidate for extensible multi-language source code analysis and the integration. Furtehr, we describe its integration with visualization toolkit. In Brazilian conference on Git and explore possible benefits from this integration for software: theory and practice (Tools Session), 2010. the performances of the platform. 374 Zbornik 21. mednarodne multikonference INFORMACIJSKA DRUŽBA – IS 2018 Zvezek I Proceedings of the 21st International Multiconference INFORMATION SOCIETY – IS 2018 Volume I Delavnica za elektronsko in mobilno zdravje ter pametna mesta Workshop Electronic and Mobile Health and Smart Cities Uredila / Edited by Matjaž Gams, Aleš Tavčar http://is.ijs.si 8.–9. oktober 2018 / 8–9 October 2018 Ljubljana, Slovenia 375 376 PREDGOVOR V letu 2016 je bil sprejet Raziskovalno Razvojni in Inovacijski (RRI) program EkoSMART v domeni pametne specializacije S4 na področju pametnih mest in skupnosti, kjer EMZ predstavlja enega od šestih nosilnih stebrov programa v obliki RRP (Raziskovalno-Razvojnega Projekta). V okviru javnega razpisa »RRI v verigah in mrežah vrednosti« − sklop 1: »Spodbujanje izvajanja raziskovalno-razvojnih programov (TRL 3-6)« je predvidenih 5,9 milijona evrov nepovratnih javnih sredstev za program EkoSMART. V letu 2017 smo pripravili tretjo delavnico na temo »EM-zdravstva« (elektronsko in mobilno zdravstvo, kratko EMZ), tj. predlog izvedbe infrastrukture in vpeljave uporabe informacijsko in mobilno podprte celostne zdravstvene oskrbe za izboljševanje preventivne, diagnostične in terapevtske obravnave državljanov, ki bi zmanjšala stroške, obenem pa povečala dostopnost zdravstvene oskrbe v obdobju 2016-2020. V letu 2018 smo izpeljali četrto delavnico na temo EMZ in jo združili s pametnimi mesti. Program EkoSMART sestavlja 6 projektov: RRP 1 – Zasnova ekosistema pametnega mesta RRP 2 – Pametna mobilnost RRP 3 – Aktivno življenje in dobro počutje RRP 4 – Elektronsko in mobilno zdravstvo RRP 5 – Integrirane zdravstvene storitve RRP 6 – Prototipi rešitev Projekt EMZ (RRP 4) sestavlja 5 delovnih sklopov oziroma delovnih paketov (DP), ki jih vodijo UKCL, IJS, FERI UM in FRI UL: • Informacijske tehnologije za podporo celostni oskrbi / bolnice • Podpora na domu za zdrave, starejše in za kronične bolnike / doma • Mobilno spremljanje vitalnih in okolijskih podatkov / mobilno • Računalniška podpora, podatki, kreiranje novih znanj /algoritmi • IKT platforma Delavnica EMZ omogoča celoletno pregledovanje in usklajevanje sklopa EMZ znotraj programa EkoSMART. Podobno kot v letu 2018 se bodo partnerji javno predstavili vsem drugim z že precej usklajenim predlogom. Vse predstavitve bomo nato dokončno uskladili in pripravili specifikacijo dela na programu za naslednje obdobje. Določili bomo podskupine partnerjev, ki bodo integrirale svoje prispevke v smiselno celoto in jih skušale premakniti naprej po TRL lestvici v smeri dejanske uporabe. Povezovali jih bomo najprej znotraj delovnih sklopov (delovnih paketov), nato znotraj RRP EMZ, nato pa še znotraj celotnega programa EkoSMART. Na delavnici bomo vse prispevke poskušali povezati tudi z osnovno integrirno platformo oz. z več platformami, ki se razvijajo znotraj EkoSMARTa. Program EM-zdravstvo so vzpodbudile potrebe po uvajanju novih IKT rešitev v Slovenijo, po horizontalnem in vertikalnem povezovanju. Predlagana pobuda EM-zdravje vpeljuje v zdravstveno oskrbo nove koncepte, ki bodo s svojimi multiplikacijskimi in sinergijskimi učinki 377 sprožili hitrejšo in učinkovitejšo prilagoditev obstoječega sistema celostne zdravstvene oskrbe na današnje izzive. Ključna strokovna komponenta je umetna inteligenca, ki bo po napovedih strokovnjakov revolucionirala zdravstvo skupaj z novimi IKT rešitvami. Javno zdravstvo po vsem svetu se otepa izrednih problemov, najboljšo rešitev pa strokovnjaki po svetu vidijo v vpeljavi storitev IKT in umetne inteligence. EM-zdravstvo (EMZ) vidimo kot eno najbolj perspektivnih smeri v več pobudah od zdravstva do pametnih mest. EM-storitve nudijo izboljšano kvaliteto življenja državljanom ob zmanjšanih stroških, hkrati pa omogočajo preboj Slovenije v svet na EM-področju. EM -zdravstvo se bo predvidoma vsebinsko oblikovalo delno kot samostojna pobuda s svojo platformo, organizacijo in projekti, ki bo povezana tako s pametnimi mesti kot z zdravjem. Ključne komponente za uspešno izvedbo EMZ so inovativni človeški viri, njihovo usklajeno delovanje in vpeljava EMZ v Sloveniji. ZDA generirajo dvakrat več pomembnih inovacij v zdravstvu kot EU ter vlagajo štirikrat več sredstev v nova, z medicino povezana podjetja. Kitajska namenja največ sredstev za znanost, medtem ko je Slovenija tretja najslabša po državnem financiranju znanosti v Evropi. Leta 2025 bo več kot milijarda, ali skoraj osmina svetovnega prebivalstva, starejša od 60 let. Stroški za zdravstveno oskrbo starejše populacije predstavljajo v EU skoraj polovico vseh stroškov za zdravstvo, kar pomeni, da grozi zdravstvenemu in gospodarskemu sistemu in kvaliteti življenja zlom, če ne bomo vpeljali storitev EM-zdravja. Druga pomembna komponenta je povezovanje in ustvarjanje kritične mase komplementarnih partnerjev, ki edino omogoča uspešen prodor na svetovna tržišča. Slovenija potrebuje sodelovanje in koordiniranje že zaradi svoje relativne majhnosti, kar dokazuje relativno slaba izkušnja z velikim številom malih in razdrobljenih projektov, ki nimajo dovolj podpore za vpeljavo novih rešitev. Tretja ključna komponenta je vpeljava EMZ v slovensko zdravstvo, ki bo na ta način dobilo novo priložnost, da vzpostavi nacionalno platformo in mednarodne standarde, preseže ujetost v nedopustno dolge čakalne dobe za pregled pri specialistih, poveže razdrobljene in nekompatibilne sisteme in že samo s tem opraviči vložena sredstva. Po zadnjem povečanju sredstev za področje zdravstva so se čakalne vrste povečale, kar kaže, da sedanji tradicionalni pristop ne zmore prinesti realnih izboljšav ob povečanih zahtevah zaradi staranja prebivalstva. Marko Bajec, Matjaž Gams, Aleš Tavčar 378 PROGRAMSKI ODBOR / PROGRAMME COMMITTEE Matjaž Gams, IJS (chair) Marko Bajec, FRI (co-chair) Roman Trobec, IJS (co-chair) Zvezdan Pirtošek, UKCLJ (co-chair) Roland Petek, Marand Jure Bon, UKCLJ Peter Kokol, FERI Andrej Kos, FE Marko Hren, SVRK Aleš Tavčar, IJS Stanislav Erzar, ISKRA Janez Uplaznik, IKTS 379 380 Študija o pooperativni atrijski fibrilaciji Dominik Čarman Tom Cvetkovič Viktor Avbelj študent Medicinske fakultete študent Medicinske fakultete Institut Jožef Stefan Univerze v Ljubljani Univerze v Ljubljani Jamova cesta 39 Vrazov trg 2 Vrazov trg 2 1000 Ljubljana 1000 Ljubljana 1000 Ljubljana viktor.avbelj@ijs.si dominik.carman@gmail.com tom.cvetkovi@gmail.com Jurij Matija Kališnik Janez Žibert Klinični oddelek za kirurgijo srca in ožilja Katedra za radiološko tehnologijo Univerzitetni klinični center Ljubljana Zdravstvena fakulteta UL Zaloška cesta 7 Zdravstvena pot 5 1000 Ljubljana 1000 Ljubljana jmkalisnik@gmail.com janez.zibert@zf.uni-lj.si POVZETEK Atrijska fibrilacija (AF) je najpogostejša kronična motnja ritma. uveljavitev enokanalnih EKG senzorjev, ki bi jih bolniki lahko Predstavlja pogost zaplet po operaciji srca. Eden od vzrokov je imeli nameščene na sebi nepretrgoma po več dni skupaj, lahko porušen avtonomna modulacija, ki jo lahko določamo s pomočjo tudi v domačem okolju. analize variabilnosti srčne frekvence, predvsem nelinearni Po operaciji srca kazalci. Poleg 12-kanalnega elektrokardiograma (EKG) se vse Atrijska fibrilacija je še posebej značilen zaplet po bolj uveljavlja tudi snemanje visokoločljivostnega EKG signala, operacijah na v zadnjem času tudi na brezžični način. srcu. To je do neke mere pričakovano, saj se veliko dejavnikov tveganja za atrijsko fibrilacijo pojavi takoj po operaciji (ishemija Ključne besede: atrijska fibrilacija, Savvy, nelinearni kazalci atrijev, draženje epikarda s sproščanjem faktorjev vnetja, avtonomne modulacije, P-val hipoksija, kirurška poškodba) [10,11,12]. Po dosedanjih ugotovitvah se atrijska fibrilacija najpogosteje pojavlja 2. in 3. 1. UVOD dan po operaciji [2,11,13,14]. Pogosteje se pojavlja po ponovnih operacijah zaradi zapletov, pri podaljšani ventilaciji ali ob 1.1 Atrijska fibrilacija potrebi po ponovni intubaciji [4]. Splošno Dejavnike tveganja med drugim lahko delimo na preoperativne, Atrijska fibrilacija je najpogostejša kronična motnja ritma [1,2], perioperativne in pooperativne. Med preoperativne štejemo katere prevalenca znaša okoli 1% in narašča s starostjo [2,3]. starost, hipertenzijo, sladkorno bolezen, hipertiroidizem, moški spol, predhodno epizodo atrijske fibrilacije, srčno popuščanje, Znaki in simptomi se pri bolnikih razlikujejo. Pri nekaterih povečan levi atrij [2,4,15,16]. Raziskave kažejo, da je pomemben poteka AF popolnoma brez simptomov, drugi so prisiljeni dejavnik tudi interatrijski blok, ki se kaže kot podaljšan in poiskati zdravniško pomoč zaradi utrujenosti, slabše telesne morfološko spremenjen P-val na EKG [13]. Med perioperativne zmogljivosti, palpitacij, bolečine v prsih ali omotice. Najhujši dejavnike sodijo čas pretisnjenja aorte, mesto venske kanulacije, zaplet je trombembolija [4]. Med morfološke dejavnike tveganja bikavalna kanulacija, ishemija miokarda in poškodba srčne spadata tudi povečan volumen in debelejša stena levega mišice med operacijo [17]. Med pooperativne dejavnike tveganja preddvora [3,5,6]. Povečano tveganje predstavljajo tudi stanja, ki pa štejemo respiratorn zmanjšajo hitrost prevajanja oz. skrajšajo refraktarno dobo [8]. e zaplete, ventilacijo, ki je daljša od 6 ur, Glede na trajanje ločimo paroksizmalno (manj kot 7 dni), hipotenzijo, vnetje, atrijske ekstrasistole, neravnovesje avtonomnega živčevja in podaljšan čas perzistentno (več kot 7 dni), dolgotrajno perzistentno (več kot 1 hospitalizacije [4,10,11,13,18,19,27]. leto) in permanentno (ne vzpostavimo sinusnega ritma) [5,7,8,9]. Pri napredovanju iz ene oblike v drugo imata pomembno vlogo 1.2 Variabilnost srčne frekvence električna in strukturna remodelacija [6,8,9]. Avtonomni sistem ima pri nastanku pooperativne AF zelo Diagnostika pomembno vlogo. Eden izmed načinov kako opredeliti njegovo Diagnostika atrijske fibrilacije temelji na anamnezi in kliničnem aktivnost je preko neinvazivnega določanja variabilnosti srčne pregledu, potrdi pa se z EKG posnetkom v vsaj 1 odvodu med frekvence. Slednjo lahko določamo z linearnimi in nelinearnimi aritmijo in z ustaljenimi kliničnimi protokoli nadzora. Tako metodami. Med linearne spadata časovna in frekvenčna analiza, lahko v ta namen uporabimo večdnevno (več)kanalno snemanje ki pa imata slabo napovedno vrednost pooperativne AF, saj ne EKG. Pomembno diagnostično orodje predstavlja tudi ultrazvok. uspeta prikazati najbolj občutljivih sprememb v stanju Sodobni trendi in razvijajoča se tehnologija težijo k poenostavitvi avtonomne regulacije, ki privedejo do pojava aritmije [20,21]. in tem bolj intuitivni uporabi diagnostičnih orodij, zato ne Nelinearne metode po drugi strani temeljijo na teoriji kaosa, ki preseneča, da je v ospredju znanstvenih prizadevanj skuša pojasniti lastnosti signala s t. i. fraktalno geometrijo oz. s fraktalno analizo fizioloških časovnih serij [22]. Tako govorimo o samopodobnosti, kjer neko fiziološko časovno serijo razdelimo na manjše dele, od teh pa vsi odražajo strukturo prvotne celote [21]. Samopodobnost ima določene meje, znotraj katerih govorimo, da omenjena lastnost podenot drži. Pri detrendni fluktuacijski analizi (DFA) jo opisujemo z eksponentom α, za katerega velja, da bližje kot je vrednosti 1, večja je samopodobnost [22]. 381 Variabilnost srčne frekvence vsebuje vrsto pomembnih DFA α1 pooperativno, večjo absolutno razliko DFA α1 ter večjo informacij. Razpad fraktalne organizacije variabilnosti srčne absolutno razliko DFA α2. frekvence v prekomeren red ali v nepovezano naključje je znak zmanjšane sposobnosti prilagajanja srčno-žilnega sistema na 5. RAZPRAVLJANJE zunanje dejavnike in je značilno prisoten pred pojavom atrijske Izsledki kliničnih študij pri bolnikih s ponavljajočo atrijsko fibrilacije, česar pa linearne metode ne pokažejo [21]. fibrilacijo potrjujejo hipotezo, da pojav aritmije lažje opazimo v daljših obdobjih merjenj – 2. HIPOTEZE drugače: dlje trajajoči nadzor pomeni več opaženih motenj ritma [25,26]. Stalni nadzor z enokanalnim Prva hipoteza, ki smo jo preverjali, govori o tem, da brezžični EKG-jem je v naši študiji zaznal skoraj 1/3 več bolnikov s enokanalni EKG senzor Savvy zazna več epizod atrijske pooperativno atrijsko fibrilacijo, kot jih je bilo zabeleženih na fibrilacije kot obstoječi trenutni klinični protokoli. Z drugo temperaturnem listu, kar je v kliničnem smislu pomembno več. hipotezo smo preverjali ali ima skupina bolnikov z atrijsko Pri tem količina dodatnega dela, morebitno bolnikovo neudobje fibrilacijo statistično značilne različne vrednosti trajanja P-vala in stroški niso zelo povečani. To ponuja številne možnosti in in statistično značilno različno obliko P-vala predoperativno od ideje za prihodnje študije, taki načini spremljanja srčnih aritmij skupine bolnikov brez atrijske fibrilacije. S tretjo hipotezo pa predstavljajo v prihodnosti velike možnosti in še neizkoriščen smo preverjali, ali se obe skupini razlikujeta v vrednostih potencial. indeksov DFA α1 in α2 po operaciji in ali sta razliki obeh Pri dveh izmed štirih primerov atrijske fibrilacije, ki smo ju videli parametrov pred in po operaciji statistično značilno različni. samo s Savvyjem, opažamo, da je bil vzrok temu manj intenziven 3. METODE nadzor. Oba bolnika sta bila namreč takrat že na oddelku, hkrati pa sta razvila epizodo atrijske fibrilacije v večernih urah. Boljše Leta 2015 izvedena pilotna študija o pooperativni atrijski spremljanje bolnikov izven enot intenzivne terapije ali nege tako fibrilaciji je bila prvič predstavljena na konferenci Informacijska predstavlja enega izmed pomembnih izzivov sodobne medicine. družba IS 2015 [23], leta 2016 tudi v reviji Informatica [24]. Mehanska ventilacija povzroča spremembe v intratorakalnem Medtem ko je bil pilotni sistem zasnovan za sočasno spremljanje tlaku, ki imajo direkten vpliv na srce, osrčnik in krvne žile, saj s 6 oseb, smo v izvedbi študije leta 2018 uporabili komercialno pozitivnim tlakom poveča intratorakalni tlak in s tem tlak v dostopen EKG senzor Savvy, s katerim smo sočasno merili do 10 desnem atriju. Slednji pa je povezan z nastankom atrijske oseb. fibrilacije. Obenem mehanska ventilacija poveča aktivnost Soglasje komisije za medicinsko etiko za izvedbo raziskave smo simpatika, ki igra pomembno vlogo pri nastanku atrijske pridobili v začetku leta 2018 – za soglasje smo zaprosili v začetku fibrilacije [13,27]. leta 2017 in prošnjo med letom dopolnili. Bistveno krajši čas trajanja P-vala pooperativno glede na Raziskava je bila prospektivne narave. Izvajala se je na predoperativno vrednost pri skupini z atrijsko fibrilacijo govori v Kliničnem oddelku za kirurgijo srca in ožilja UKC Ljubljana, od prid temu, da je tudi povečana prevodnost med atrijema marca do julija 2018. proaritmogena [28]. Študijo je uspešno zaključilo 47 bolnikov. Vsi so sodelovanje v Daljši PQ interval govori o tem, da je AV blok povezan z študiji potrdili s podpisom obveščenega pristanka, prav tako so nastankom atrijske fibrilacije, kar so pokazale pretekle študije vsi izpolnjevali vključitvene kriterije in nobenega od [6,29,30,31]. izključitvenih (kronična atrijska fibrilacija predoperativno, AV Nižja vrednost DFA α1 pooperativno pri skupini z atrijsko blok II. ali III. stopnje, srčni spodbujevalnik, urgentni bolniki, fibrilacijo nakazuje na razpad fraktalne geometrije oz. bolniki iz drugih oddelkov UKC Ljubljana). Meritve so tekle samopodobnosti signala, ki smo ga dobili iz RR intervalov pred predoperativni dan in nato od 1. dne pooperativno nepretrgoma pojavom prve epizode pooperativne atrijske fibrilacije [32,33]. do 5. pooperativnega dne. Predoperativni dan smo bolniku Podobno lahko gledamo tudi na večjo absolutno razliko DFA α1 izmerili 20-minutni visokoločljivostni EKG (Cardiax), obenem smo mu namestili brezžični enokanalni EKG senzor Savvy, ki ga pri skupini z atrijsko fibrilacijo. Ker se je pooperativna vrednost pri skupini z AF relativno bolj zmanjšala glede na predoperativno je imel bolnik nameščenega do jutra pred operacijo. Prvi vrednost, to prav tako kaže na razpad samopodobnosti signala in pooperativni dan smo bolniku zopet namestili brezžični bi lahko v prihodnosti služil kot pomemben napovedni dejavnik enokanalni EKG senzor Savvy in ga pustili na njem do 5. dne za razvoj atrijske fibrilacije [32,33]. Večja absolutna razlika DFA nepretrgoma; 2., 3. in 5. dan smo izmerili tudi 20-minutni α2 pri skupini z atrijsko fibrilacijo nakazuje na, podobno kot visokoločljivostni EKG posnetek (Cardiax). Meritve smo izvajali opisano zgoraj, razpad samopodobnosti signala, tokrat pri nizu z 10 napravami Savvy, 10 pametnimi telefoni oz. tablicami ter visokoločljivostnim RR intervalov večjih od 11. To govori v prid porušenemu 12-kanalnim EKG aparatom (Cardiax). simpatično-vagalnemu ravnovesju pooperativno, kar se odraža v Pri analizi dokumentacije smo bolnike razvrstili v skupino z znižanju obeh kazalcev DFA α [21]. atrijsko fibrilacijo in skupino brez atrijske fibrilacije. V prvi skupini so imeli bolniki na katerikoli način (Savvy ali s klinično 6. ZAKLJUČEK ustaljenimi protokoli) dokazano atrijsko fibrilacijo. AF je morala Z brezžičnim enokanalnim EKG senzorjem Savvy smo odkrili trajati vsaj 30 sekund, da smo jo označili kot epizodo AF. več bolnikov z atrijsko fibrilacijo, kot jih je bilo zabeleženih s 4. REZULTATI klinično ustaljenimi protokoli, zato smo potrdili prvo hipotezo. Pri analizi dolžine P-vala in njegove morfologije nismo odkrili Izmed 47 bolnikov jih je 13 razvilo atrijsko fibrilacijo. Od tega razlik med obema skupinama, zato smo drugo hipotezo zavrnili. smo vseh 13 zaznali z napravo Savvy, medtem ko jih je bilo s Razloge iščemo predvsem v majhnem naboru preiskovancev. klinično ustaljenimi protokoli in napravami zabeleženih 9. Kljub temu smo z analizo razlike trajanja P-vala pred in po Izključno z napravo Savvy smo tako na novo odkrili 4 primere operaciji pokazali relativno večje skrajšanje P-vala po operaciji atrijske fibrilacije. Skupina z atrijsko fibrilacijo je imela glede na stanje pred operacijo pri skupini z atrijsko fibrilacijo. povprečno daljši čas intubacije. Izmed elektrofizioloških Opazili smo pomembno razliko v trajanju PQ intervala po parametrov je imela skupina z atrijsko fibrilacijo večjo razliko v operaciji, s čimer smo potrdili daljše trajanje tega intervala kot trajanju P-vala (računana kot razlika pooperativne in dejavnik tveganja za pojav atrijske fibrilacije. Potrdili smo nižjo predoperativne vrednosti), daljši PQ interval, nižjo vrednost vrednost DFA α1 pooperativno ter večjo absolutno razliko v 382 DFA α1 in DFA α2 pri skupini z atrijsko fibrilacijo, s čimer smo [12] Creswell LL, Schuessler RB, Rosenbloom M, Cox JL. prepoznali porušeno avtonomno modulacijo srca tik pred Hazards of postoperative atrial arrhythmias. The Annals of pojavom atrijske fibrilacije, zato lahko potrdimo tretjo hipotezo thoracic surgery. 1993 Sep 1;56(3):539-49. - avtonomni kazalci so pri bolnikih z atrijsko fibrilacijo različni [13] Maisel WH, Rawn JD, Stevenson WG. Atrial fibrillation kot pri bolnikih, ki atrijske fibrilacije ne razvijejo. after cardiac surgery. Annals of internal medicine. 2001 Dec Naša študija predstavlja pomemben korak v smeri lažje in 18;135(12):1061-73. hitrejše zaznave motenj ritma s pomočjo pametnih tehnologij. [14] Nair SG. Atrial fibrillation after cardiac surgery. Annals of Rezultati kažejo na velik potencial takega načina merjenja in na cardiac anaesthesia. 2010 Sep 1;13(3):196. številne možnosti, ki se odpirajo s prepoznavo novih napovednih [15] Hayashida N, Shojima T, Yokokura Y, Hori H, Yoshikawa kazalcev za pojav atrijske fibrilacije po operacijah srca. K, Tomoeda H, Aoyagi S. P-wave signal-averaged Verjamemo, da nas v prihodnjih letih čaka velik razmah na electrocardiogram for predicting atrial arrhythmia after cardiac področju odkrivanja motenj srčnega ritma. Hkrati upamo, da bo surgery. The Annals of thoracic surgery. 2005 Mar 1;79(3):859- ta študija predstavljala kvalitetno podlago naslednjim 64. obširnejšim raziskavam, ki bodo sčasoma vodile do novih rešitev [16] Aytemir K, Aksoyek S, Ozer N, Aslamaci S, Oto A. Atrial za izboljšanje kakovosti življenja bolnikov. fibrillation after coronary artery bypass surgery: P wave signal 7. REFERENCE averaged ECG, clinical and angiographic variables in risk assessment1. International journal of cardiology. 1999 Apr [1] Benjamin EJ, Wolf PA, D’Agostino RB, Silbershatz H, 30;69(1):49-56. Kannel WB, Levy D. Impact of atrial fibrillation on the risk of [17] Omae T, Kanmura Y. Management of postoperative atrial death: the Framingham Heart Study. Circulation. 1998 Sep fibrillation. Journal of anesthesia. 2012 Jun 1;26(3):429-37. 8;98(10):946-52. [18] Koch CG, Li L, Van Wagoner DR, Duncan AI, Gillinov [2] Schotten U, Verheule S, Kirchhof P, Goette A. AM, Blackstone EH. Red cell transfusion is associated with an Pathophysiological mechanisms of atrial fibrillation: a increased risk for postoperative atrial fibrillation. The Annals of translational appraisal. Physiological reviews. 2011 thoracic surgery. 2006 Nov 1;82(5):1747-56. Jan;91(1):265-325. [19] Kaireviciute D, Aidietis A, Lip GY. Atrial fibrillation [3] Go AS, Hylek EM, Phillips KA, Chang Y, Henault LE, following cardiac surgery: clinical features and preventative Selby JV, Singer DE. Prevalence of diagnosed atrial fibrillation strategies. European heart journal. 2009 Jan 27;30(4):410-25. in adults: national implications for rhythm management and [20] Heart Rate Variability. Standards of measurement, stroke prevention: the AnTicoagulation and Risk Factors in physiological interpretation, and clinical use. Task Force of the Atrial Fibrillation (ATRIA) Study. Jama. 2001 May European Society of Cardiology and the North American 9;285(18):2370-5. Society of Pacing and Electrophysiology. Eur Heart J. 1996; [4] Fuster V, Rydén LE, Cannom DS, Crijns HJ, Curtis AB, 17(3): 354-81. Ellenbogen KA, Halperin JL, Kay GN, Le Huezey JY, Lowe [21] Kšela J. Novejši kazalci avtonomne regulacije srca kot JE, Olsson SB. 2011 ACCF/AHA/HRS focused updates napovedni dejavniki za pojav aritmij po aortokoronarnih incorporated into the ACC/AHA/ESC 2006 c: a report of the obvodih na delujočem srcu [doktorsko delo]. Ljubljana: American College of Cardiology Foundation/American Heart Univerza v Ljubljani; 2009. Association Task Force on Practice Guidelines developed in partnership with the European Society of Cardiology and in [22] Goldberger AL, Amaral LA, Hausdorff JM, Ivanov PC, collaboration with the European Heart Rhythm Association and Peng CK, Stanley HE. Fractal dynamics in physiology: the Heart Rhythm Society. Journal of the American College of alterations with disease and aging. Proceedings of the national Cardiology. 2011 Mar 15;57(11):e101-98. academy of sciences. 2002 Feb 19;99(suppl 1):2466-72. [5] De Caterina R, Atar D, Hohnloser SH, Hindricks G. 2012 [23] Kališnik J M, Poplas-Susič T, Semeja A, Korošec T, Trobec R, Avbelj V, Depolli M, Stanič U. focused update of the ESC Guidelines for the management of Mobile health atrial fibrillation. European heart journal. 2012;33:2719-47. monitoring pilot systems. Proceedings of the 18th International Multiconference Information Society - IS 2015, October 9th [6] Andrade J, Khairy P, Dobrev D, Nattel S. The clinical and 12h, 2015, Ljubljana, Slovenia. Volume G, pp. 62-65. profile and pathophysiology of atrial fibrillation: relationships among clinical features, epidemiology, and mechanisms. [24] Depolli M, Avbelj V, Trobec R, Kališnik J, Korošec T, Circulation research. 2014 Apr 25;114(9):1453-68. Poplas-Susič T, Stanič U, Semeja A. PCARD platform for mHealth monitoring. Informatica. 2016;40:117-123. [7] Allessie MA, Boyden PA, Camm AJ, Kléber AG, Lab MJ, Legato MJ, Rosen MR, Schwartz PJ, Spooner PM, Van [25] Dilaveris PE, Kennedy HL. Silent atrial fibrillation: Wagoner DR, Waldo AL. Pathophysiology and prevention of epidemiology, diagnosis, and clinical impact. Clinical atrial fibrillation. Circulation. 2001 Feb 6;103(5):769-77. cardiology. 2017 Jun;40(6):413-8. [8] Nattel S, Burstein B, Dobrev D. Atrial remodeling and atrial [26] Fetsch T, Bauer P, Engberding R, Koch HP, Lukl J, fibrillation: mechanisms and implications. Circulation: Meinertz T, Oeff M, Seipel L, Trappe HJ, Treese N, Breithardt Arrhythmia and Electrophysiology. 2008 Apr 1;1(1):62-73. G. Prevention of atrial fibrillation after cardioversion: results of the PAFAC trial. European heart journal. 2004 Aug [9] Iwasaki YK, Nishida K, Kato T, Nattel S. Atrial fibrillation 1;25(16):1385-94. pathophysiology: implications for management. Circulation. 2011 Nov 15;124(20):2264-74. [27] Erdil N, Gedik E, Donmez K, Erdil F, Aldemir M, Battaloglu B, Yologlu S. Predictors of postoperative atrial [10] FRCPS PP, Mitchell LB, Crystal E, Heilbron B. Atrial fibrillation after on-pump coronary artery bypass grafting: is fibrillation following cardiac surgery. Can J Cardiol. 2005 duration of mechanical ventilation time a risk factor?. Annals of Sep;21:45B. Thoracic and Cardiovascular Surgery. 2014;20(2):135-42. [11] Almassi GH, Schowalter T, Nicolosi AC, Aggarwal A, [28] Nielsen JB, Kühl JT, Pietersen A, Graff C, Lind B, Struijk Moritz TE, Henderson WG, Tarazi R, Shroyer AL, Sethi GK, JJ, Olesen MS, Sinner MF, Bachmann TN, Haunsø S, Grover FL, Hammermeister KE. Atrial fibrillation after cardiac Nordestgaard BG. P-wave duration and the risk of atrial surgery: a major morbid event?. Annals of surgery. 1997 fibrillation: Results from the Copenhagen ECG Study. Heart Oct;226(4):501. Rhythm. 2015 Sep 1;12(9):1887-95. 383 [29] Cheng S, Wang TJ. Outcomes in Patients With Prolonged PR Interval or First-Degree Atrioventricular Block—Reply. JAMA. 2009 Nov 11;302(18):1967-8. [30] Nielsen JB, Pietersen A, Graff C, Lind B, Struijk JJ, Olesen MS, Haunsø S, Gerds TA, Ellinor PT, Køber L, Svendsen JH. Risk of atrial fibrillation as a function of the electrocardiographic PR interval: results from the Copenhagen ECG Study. Heart Rhythm. 2013 Sep 1;10(9):1249-56. [31] Nielsen JC, Thomsen PE, Højberg S, Møller M, Riahi S, Dalsgaard D, Mortensen LS, Nielsen T, Asklund M, Friis EV, Christensen PD. Atrial fibrillation in patients with sick sinus syndrome: the association with PQ-interval and percentage of ventricular pacing. Europace. 2011 Nov 20;14(5):682-9. [32] Huikuri HV, Mäkikallio TH, Perkiömäki J. Measurement of heart rate variability by methods based on nonlinear dynamics. Journal of electrocardiology. 2003 Dec 1;36:95-9. [33]Shin DG, Yoo CS, Yi SH, Bae JH, Kim YJ, Park JS, Hong GR. Prediction of paroxysmal atrial fibrillation using nonlinear analysis of the RR interval dynamics before the spontaneous onset of atrial fibrillation. Circulation Journal. 2006;70(1):94-9. 384 EMZ in EkoSMART-asistent Mateja Drnovšek Matjaž Gams Department of Intelligent Systems Department of Intelligent Systems Jozef Stefan Institute Jozef Stefan Institute Jamova cesta 39 Jamova cesta 39 1000 Ljubljana 1000 Ljubljana mateja.drnovsek@ijs.si matjaz.gams@ijs.si POVZETEK • RRP 3: Aktivno življenje, dobro počutje V prispevku “EMZ in EkoSMART-asistent” je podrobneje • RRP 4: Elektronsko in mobilno zdravje (EMZ) opisan program EkoSMART, ki je razdeljen na šest projektov, • RRP 5: Pametni sistem integriranega zdravstva in oskrbe med njimi je tudi projekt EMZ (Elektronsko in mobilno zdravje). Le-ta je podrobneje opisan v 3. poglavju, v • RRP 6: Razvoj prototipov nadaljevanju pa je povezan z občinami in zdravjem (4. in 5. Poleg štirih projektov, ki se ukvarjajo z digitalizacijo izbranih poglavje). EkoSMART-asistent, ki je bil razvit na Institutu področij (RRP2-RRP5), sta v program predvidena še dva “Jozef Stefan”, se nahaja na spletni strani EkoSMART [6], v 6. skupna projekta. Eden se ukvarja s skupnim jedrom oziroma poglavju so naštete in podrobno opisane vse aplikacije, ki jih arhitekturo ekosistema (RRP1), eden pa je namenjen razvoju asistent ponuja. prototipnih rešitev in njihovemu preskušanju v relevantnih okoljih (RRP6). Ključne besede S takšno projektno sestavo sledimo dobrim praksam, ki pravijo, elektronsko in mobilno zdravje, virtualni asistent, občine, da je sisteme sistemov, kar pametna mesta so, potrebno graditi EkoSMART-asistent upoštevajoč oba razvojna modela: od spodaj navzgor (RRP2 do 1. UVOD 5, začenši s posameznimi področji in potem z njihovo integracijo v celoto) in od zgoraj navzdol (RRP1, izhajajoč iz Prispevek “EMZ in EkoSMART-asistent” obravnava sestavo strateških usmeritev mesta in potem z osredotočanjem na programa EkoSMART v 6 raziskovalno-razvojnih projektih, posamezna področja). Ključno je enostavno dodajanje novih med njimi je tudi RRP 4 – EMZ (Elektronsko in mobilno storitev pametnih mest, kar omogoča univerzalna skupna zdravje). V današnjem času tehnologije sta umetna inteligenca arhitektura. in IKT zelo pomembna faktorja k napredku v znanosti, k implementaciji na nove produkte ter, kar je bistveno, k aplikativni uporabi v vsakdanjem življenju. Namen projekta EMZ je bil izboljšati varstvo in varnost zdravih, starejših, kroničnih bolnikov in ljudi s posebnimi potrebami z vpeljavo sodobnih informacijsko-komunikacijskih tehnologij (IKT) v celostno elektronsko in mobilno zdravstveno oskrbo (EMZ). Namen je nadgraditi kakovost obravnave in oskrbe omenjenih skupin s pomočjo elektronskih in mobilnih naprav ter s pripadajočimi programskimi rešitvami, zlasti z uporabo telesnih senzorjev in nosljivih pripomočkov, ki omogočajo spremljanje počutja in zdravja na podlagi meritev ter umetne inteligence nad podatki v EkoSMART-platformi. Za lažji dostop do informacij je bil razvit tudi EkoSMART-asistent z vsemi pripadajočimi aplikacijami. Slika 1. Spletna stran EkoSMART, dostopna na [2]. 2. PROGRAM EKOSMART Ekosmart Pametnega Mesta (EkoSMART) [1] je program, ki je bil sprejet na Javnem razpisu »Spodbujanje izvajanja 3. PROJEKT EMZ raziskovalno-razvojnih programov (TRL3-6)«, in sicer Projekt RRP4: EMZ (Elektronsko in mobilno zdravje) [1] je prednostno področje S4: Pametna mesta in skupnosti. Glavni bistveni del programa EkoSMART. Namen projekta EMZ je cilj tega programa je razviti ekosistem pametnega mesta z vsemi izboljšati varstvo in varnost zdravih, starejših, kroničnih podpornimi mehanizmi, ki so potrebni za učinkovito, bolnikov in ljudi s posebnimi potrebami z uporabo modernih optimizirano in postopno integracijo posameznih področij v tehnologij in pristopov. Cilj projekta je bil tudi izdelati sisteme enovit in povezan sistem vrednostnih verig. Z zagotavljanjem za pomoč slepim in slabovidnim, gluhim, kognitivno, vedenjsko kakovostnih storitev za državljane in gospodarstvo želimo in gibalno motenim ter starejšim, ki so sami doma, in omogočiti omogočiti dvig kakovosti življenja in rast splošne blaginje. kvalitetnejše življenje ljudem s kroničnimi boleznimi, ki občasno ali stalno potrebujejo zdravniško oskrbo. Program EkoSMART sestavlja šest projektov, ki vsak po svoje prispevajo k uresničevanju vizije programa (kratica RRP Projekt EMZ je nastal iz širše iniciative elektronskega in pomeni Raziskovalno-razvojni projekt). mobilnega zdravstva, ki zajema okrog 300 partnerjev, ki želijo sodelovati in si želijo biti seznanjeni z dogajanjem na tem • RRP 1: Zasnova ekosistema pametnega mesta področju. Za potrebe izvedbe projekta RRP4 so bile iz nabora • RRP 2: Pametna mobilnost - sistemi za nadzor in upravljanje partnerjev EMZ skrbno izbrane naslednje organizacije: prometnih tokov 385 1. Institut »Jožef Stefan« (IJS) asistentih – kliknete na svojo občino, levo zgoraj, izberete 2. UL - Medicinska fakulteta (MF) aplikacijo »Zdravstvo in sociala«. 3. Univerzitetni klinični center Ljubljana (UKCL) 4. Marand d.o.o. (MAR) 5. UL - Fakulteta za računalništvo in informatiko (FRI) 6. UM - Fakulteta za elektrotehniko, računalništvo in informatiko (FERI) 7. URI-Soča (SOČA) 8. ZD Adolfa Drolca (ZDM) 9. Cosylab (CSL) 10. Elgoline d.o.o. (ELGO) 11. SRC sistemske integracije d.o.o. (SRC) 12. Špica International d.o.o. (ŠPI) 13. Inova IT d.o.o. (INO) 14. Medis d.o.o. (MED) 15. Nela razvojni center d.o.o. (NELA) Slika 2. Del spletne strani Zdravje[5], kjer so podsistemi za 4. EMZ in OBČINE zdravje. Razvili smo ogrodje in večino sistemov občin [3,4] kot korak Asistent vam ponudi sledeče podsisteme: dlje od pametnih mest in oboje sistematično vpeljali v občine, društva upokojencev in druga društva ter v širšo družbo z  Prva pomoč: Tu dobite nasvete v primeru nujne namenom, da Slovenija ponudi boljšo izkušnjo mest oz. občin pomoči (Mobilno Android IOS). Slovenije za vse, domače in turiste. Asistenti za posamezne  Zdravstveni nasveti (slovensko): Če potrebujete občine ponujajo pomembne informacije o občini, storitve in zdravstveni nasvet, si lahko pomagate s strokovnjaki podporo kvalitetnejšemu življenju. Umetna inteligenca in IKT na spletu. napredujeta zelo hitro, raziskovalni oddelki razvijajo zelo dobre  Zdravstveni nasveti (angleško): Poučne spletne nove sisteme, poleg tega pa se razvijajo tudi sistemi, ki jih strani o zdravstvu v angleškem jeziku. lahko namestijo in vzdržujejo lokalne inštitucije in družba.  NIJZ (Nacionalni inštitut za javno zdravje): Povezava na njihovo spletno stran. Sistem sestavljajo naslednji bloki:  PROJEKTI:  - IN LIFE: Pametna varnostna ura je najpomembnejši Občinska televizija – vsakdo lahko razvije svojo občinsko produkt, ki je bil razvit v okviru projekta IN LIFE. televizijo s sledenjem navodilom. Potreben je prenosnik in kamera in nekaj znanja računalništva. Običajno občinska TV Ura je namenjena starejšim ljudem za izboljšanje prenaša ali sprotno dogajanje v živo, ali pa se vrti vnaprej njihove varnosti in podaljšanju avtonomije. Ura je že pripravljena datoteka z informacijami za tekoči teden. pripravljena za prodajo.  - E-gibalec: mobilna aplikacija, razvita z namenom 3D virtualni asistent – ponovno je z nekaj znanja povečanja računalništva po navodilih možno izdelati sistem 3D športnih aktivnosti in gibanja osnovnošolcev. Hkrati smo s tem preprečevali virtualnega asistenta, ki vodi po stavbah, recimo upravni stavbi občine. debelost v otroštvu, ki je v današnjem času čedalje pogostejša [7].  Turizem – sistem omogoča informiranje o turističnih - ASPO: spletna aplikacija za prepoznavanje in znamenitostih v naravnem jeziku in načrtovanje turističnih informiranje o spolno prenosljivih okužbah. obiskov. Sistem vsebuje preko 3000 znamenitosti v Sloveniji in - Zaznavanje stresa: Spletni pogovorni svetovalec je zato izjemno uporaben. OSVET je namenjen zaznavanju stresa in nudenju  Asistenti – za vsako izmed 200 slovenskih občin je narejen psihosocialne pomoči preko spleta. Z anketo lahko občinski asistent, ki odgovarja na vprašanja v naravnem jeziku. preverite stopnjo svojega stresa. Obstajata tudi pokrajinski asistent in slovenski asistent – - EkoSMART, EMZ: namen programa EkoSMART je slednja sta sestavljena iz pripadajočih občinskih asistentov. razviti ekosistem pametnega mesta. Prav tako je bil Asistent za starejše občane (ZDUS) se imenuje Zdusko, le-ta je v pomoč starejšim ljudem, ki iščejo informacije. V poglavju razvit tudi EkoSMART-asistent, ki je podrobneje opisan v spodnjem poglavju. spodaj je bolj podrobno opisan EkoSMART-asistent, vključno z - HEP-Y: Spletna aplikacija za prepoznavanje in namenom razvoja asistenta in njegovimi aplikacijami [3]. Več o asistentih si lahko preberete v prispevku za IS 2018: “Pregled informiranje o virusnih hepatitisih. IJS E9 asistentov”. - Čakalne vrste: Asistent Čakalne vrste je namenjen iskanju čakalnih dob za določene zdravstvene posege pri posameznih ustanovah (več v prispevku “Pregled 5. EMZ in ZDRAVJE IJS E9 asistentov”). Na spletni strani Občin [4] je Zdravje najpomembnejši blok EMZ-ja [3,5]. Tu so informacije o prvi pomoči, zdravstveni 6. EKOSMART-ASISTENT nasveti iz Slovenije in tujine, informacije iz NIJZ, iz programa Kratek opis: Asistent je namenjen posredovanju informacij o pametne specializacije EkoSMART ter podprojekta Elektronsko programu pametne specializacije EkoSMART. V trenutni in mobilno zdravje, repozitorijev domen in prototipov, sistemov realizaciji odgovarja na vsebinska vprašanja o posameznih za nadziranje stresa in o skrbi za starejše. Storitev je dostopna v projektih, ki so vključeni v program. Posreduje organizacijske podrobnosti in podatke o prototipih produktov, storitvah in 386 izdelkov, ki se razvijajo v okviru programa. V končni različici Na spletni strani EkoSMART v ozadju se odpre pa bo asistent omogočal iskanje različnih vrst podatkov v zavihek “Rešitve” in opis projekta RRP 6. Glavni zdravstvenih platformah in razviti integracijski platformi rezultati in cilji so zagotovitev, vzpostavitev in pametnega mesta [2,5]. testiranje prototipov sistemov v ciljnih okoljih, vključno z izvedbami, ki bodo podobne delovanju v Aplikacije: realnih okoliščinah. - Opis: Ekosmart-asistent odpre v ozadju na spletni strani EkoSMART zavihek “Rešitve”, kjer je - Partnerji/inštitucije: Konzorcij programa Ekosmart navedeno, da je program EkoSMART sestavljen iz sestavlja 25 partnerjev z različnih področij delovanja, šestih projektov. V nadaljevanju so projekti našteti, našteti so na spletni strani v ozadju. ob kliku na projekt pa lahko vidimo njegov opis. - Program Ekosmart: Na spletni strani EkoSMART v ozadju se odpre zavihek “O projektu”, kjer si lahko - Partnerji/ljudje: V Ekosmart-asistenta lahko vpišete ogledate splošen opis in namen projekta EkoSMART. ime in priimek osebe zaposlene na projektu EkoSMART, nato pa vam asistent sporoči kontaktne - Prototipi: Povezava na seznam prototipov. podatke. V ozadju se na spletni strani EkoSMART odpre zavihek “Partnerji”, kjer je naštetih 25 partnerjev. - RRP 1 - Zasnova ekosistema pametnega mesta: EkoSMART-asistentu lahko postavite vprašanje glede projekta RRP 1. Na spletni strani v ozadju pa se odpre zavihek “Rešitve” in opis slednjega projekta. V opisih projektov najdete vodjo projekta, namen, ključne partnerje ter glavne cilje in rezultate. Bistvena je povezava pametnega mesta v celovit ekosistem. - RRP 2 - Pametna mobilnost: EkoSMART-asistentu lahko postavite vprašanje glede projekta RRP 2. Na spletni strani v ozadju pa se odpre zavihek “Rešitve” in opis projekta. Projekt se osredotoča na problem rigidnih semaforskih sistemov, ki so neodzivni na realno stanje v prometu. - RRP 3 - Aktivno življenje in dobro počutje: EkoSMART-asistentu lahko postavite vprašanje glede projekta RRP 3. Na spletni strani v ozadju pa se odpre zavihek “Rešitve” in opis projekta. Poudarjena je pomembnost družine kot osnovne celice ter Slika 3. možnosti medgeneracijskega povezovanja znotraj prikazan je EkoSMART-asistent, ki je dostopen na spletni družine. Poslovni cilj je programska rešitev strani [6]. MyFamily, ki bo v ekosistemu EkoSMART omogočala večjo povezanost družine ter vključevanje v medgeneracijsko povezanost za boljšo kakovost življenja. - RRP 4 - Elektronsko in mobilno zdravstvo (EMZ): V ozadju se na spletni strani odpre širši opis projekta. Glavni poslovni in tehnološki cilji RRP 4 so sledeči: izkoristiti potencial gospodarstva, medicine in znanosti za razvoj celovitega Sistema EMZ v 7. ZAKLJUČEK Sloveniji, ki bo znižal naraščajoče stroške na V prispevku “EMZ in EkoSMART-asistent” je bil v celoti področju zdravstva in sociale, okrepiti, obogatiti in predstavljen program EkoSMART, ki je sestavljen iz 6 nadgraditi ponudbo proizvodov in storitev, podprtih z raziskovalno-razvojnih projektov, med njimi tudi EMZ IKT na področju zdravstva ter drugi. (Elektronsko in mobilno zdravje). Glavni namen EMZ je z umetno inteligenco in IKT izboljšati zdravje in kvaliteto - RRP 5 - Integrirane zdravstvene storitve: življenja ljudem, hkrati pa s programskimi rešitvami EkoSMART-asistentu lahko postavite vprašanje glede razbremeniti zdravstveni sistem. Na Odseku za inteligentne projekta RRP 5. Na spletni strani EkoSMART v sisteme na Institutu “Jožef Stefan” je bil razvit tudi ozadju se odpre zavihek “Rešitve” in opis projekta. EkoSMART-asistent s številnimi aplikacijami za lažji dostop do Opisani so tudi glavni rezultati in cilji projekta, med informacij. drugim tudi povečanje kakovosti in varnosti življenja kroničnih bolnikov in daljša oskrba v domačem okolju. 8. ZAHVALA Raziskave in razvoj so bile izvajane v okviru programa - RRP 6 - Prototipi rešitev: EkoSMART-asistentu EkoSMART in so delno sofinancirane s strani Ministrstva za lahko postavite vprašanja glede projekta RRP 6. izobraževanje, znanost in šport ter Evropske unije iz 387 Evropskega sklada za regionalni razvoj (ESRR). Zahvaljujemo https://is.ijs.si/archive/proceedings/2017/files/Zbornik%20 se tudi sodelavcu Alešu Tavčarju, ki je največ prispeval pri -%20I.pdf programu in vsem ostalim sodelavcem. [4] Spletna stran Ui-Občine IJS [20. 9. 2018], dostopno na https://ui-obcine.ijs.si/ [5] Spletna stran Zdravje [20. 9. 2018], dostopno na https://ui- obcine.ijs.si/zdravje/ 9. REFERENCE [6] Povezava na EkoSMART-asistenta [20. 9. 2018], dostopno na http://projekt-asistent.si/ekosmart [1] Bela knjiga EMZ EkoSMART [28. 9. 2018], dostopno na http://library.ijs.si/Stacks/Literature/Bela%20knjiga%20E [7] Projekt »Uživajmo v zdravju«: Za zmanjšanje debelosti MZ%20EkoSMART.pdf otrok in mladostnikov [20. 9. 2018], dostopno na http://www.nijz.si/sl/projekt-uzivajmo-v-zdravju-za- [2] Spletna stran EkoSMART [20. 9. 2018], dostopno na zmanjsanje-debelosti-otrok-in-mladostnikov-0 http://ekosmart.net/sl/ekosmart/ [3] Gams, M. in Tavčar, A. 2017. Zbornik 20. mednarodne multikonference INFORMACIJSKA DRUŽBA – IS 2017 Delavnica za elektronsko in mobilno zdravje, dostopno na 388 Pregled asistentov IJS E9 Mateja Drnovšek Matjaž Gams Department of Intelligent Systems Department of Intelligent Systems Jozef Stefan Institute Jozef Stefan Institute Jamova cesta 39 Jamova cesta 39 1000 Ljubljana 1000 Ljubljana mateja.drnovsek@ijs.si matjaz.gams@ijs.si Aleš Tavčar Gregor Grasselli Department of Intelligent Systems Department of Intelligent Systems Jozef Stefan Institute Jozef Stefan Institute Jamova cesta 39 Jamova cesta 39 1000 Ljubljana 1000 Ljubljana ales.tavcar@ijs.si gregor.grasselli@ijs.si POVZETEK Asistenti so v zadnjih letih čedalje bolj pogosti in dobrodošli na spletnih straneh. V tem prispevku so opisani asistenti, ki so bili razviti na Odseku za inteligentne sisteme na Institutu »Jožef Stefan«. Zajetih je več asistentov, med drugim asistent IJS, asistent za pametna mesta, asistenta za zdravjem, meta asistent, ki povezuje vse občinske asistente, 3D-asistent in asistent ZDUS. Ključne besede Virtualni asistent, asistent IJS, zdravje, pametna mesta, meta asistent, občine. 1. UVOD Virtualni asistenti so koristni na spletnih straneh, saj nas zelo hitro pripeljejo do željenih odgovorov in nam podajo informacije, ki nas zanimajo. V prispevku je narejen kratek Slika 1. Asistent Robi, ki je na spletnih straneh Instituta pregled asistentov in opis asistentovih aplikacij, ki so bili “Jožef Stefan” [1]. razviti na IJS v okviru Odseka za inteligentne sisteme. Razvoj asistentov je bil usmerjen predvsem na točne informacije v V sredinsko polje uporabnik vpiše vprašanje in spodaj se pojavi povezavi s spletno stranjo v ozadju asistenta. Kot najboljši se je odgovor v obliki spletnega besedila. V ozadju se na spletni izkazal asistent IJS, zelo pomembni pa so tudi občinski strani pojavi zadetek iskalnika, tako da uporabnik vidi odgovor asistenti, ki obiskovalca spletne strani informirajo o dogajanju v asistenta in iskalnika v ozadju. Četudi asistent nič ne najde, še občinah. Zanimivi so tudi ostali asistenti, vsi so opisani v 2. vedno dobimo odgovor spletnega iskalnika. poglavju. Sledi opis asistentov. 2. PREGLED ASISTENTOV Vsi asistenti so v obliki spletne aplikacije / spletnega agenta, ki 2.1 Asistent IJS, Robi se po klicu (kliku) prikaže v obliki okenca. V zgornjem delu je Kratek opis: Asistent Robi odgovarja na vprašanja neka slika, povezana s predstavitvijo sistema, nad njo je rob z obiskovalcev IJS in zaposlenih na IJS. Na vprašanja odgovarja aplikacijami levo, pomočjo in izhoda desno. Aplikacije zgoraj v slovenščini, angleščini, francoščini. Zna govoriti slovensko in levo so pomembne, ker povedo, na katere teme se asistent angleško, zna komunicirati preko robota, npr. Nao, zna izvajati posebej spozna. Če izberemo aplikacijo, asistent najprej animacije preko robota animirano na zaslonu in dejansko, kadar pogleda v omenjenem modulu, ali je kje pripravljena primerna je fizično priključen na robota. aktivnost. Recimo v primeru, da imamo izbrano aplikacijo zaposleni (Slika 1), potem bo sistem iskal najprej po imenih, Aplikacije: priimkih in drugih podatkih o zaposlenih in v primeru zadetka - Zaposleni/imenik: Če vpišete ime ali priimek ali kak prikazal tovrstne odgovore. Če pa isto vprašanje napišemo drug podatek o zaposlenem, bo Robi poiskal vse splošno, potem so moduli enakovredni in lahko sistem najde zaposlene, ki ustrezajo opisu. povsem drug odgovor kot bolj smiseln. - Prireditve, novice: Robi vam ponudi koledar prireditev, ki vas usmeri na napovednik dogodkov. - Vprašanja iz podjetij: Robi odpre obrazec, ki je namenjen podjetjem pri navezovanju stikov z IJS. - Organizacijska struktura IJS: Odpre se spletna stran, kjer najdete organizacijske enote Instituta. 389 - Zaposlitve, študentsko delo: Robi vas usmeri na 2.4 EkoSMART-asistent objavo prostih delovnih in študentskih mest. Kratek opis: Asistent je namenjen posredovanju informacij o - Raziskovalna oprema: Spisek pomembnejše programu pametne specializacije EkoSMART. V trenutni inštitutske raziskovalne opreme na strani Nanocenter. realizaciji odgovarja na vsebinska vprašanja o posameznih - Znotraj hiše, notranji akti in obrazci: Robi v ozadju projektih, ki so vključeni v program. Posreduje organizacijske odpre spletno stran, kjer zaposleni dobijo pomembne podrobnosti in podatke o prototipih produktov, storitvah in informacije. Del informacij je dostopen samo izdelkov, ki se razvijajo v okviru programa. V končni različici zaposlenim. pa bo asistent omogočal iskanje različnih vrst podatkov v - Obvestila vzdrževalcem: Zaposleni lahko preko zdravstvenih platformah in razviti integracijski platformi aplikacije posredujete vzdrževalcem podatke o pametnega mesta [3, 5]. okvarah na infrastrukturi IJS. - Počitnice: Robi vas usmeri na počitniške kapacitete, Aplikacije: ki jih ima IJS. - Opis: Ekosmart-asistent v ozadju odpre kratek opis - Malica: Robi vas seznani s tedenskim jedilnikom v projekta Ekosmart. menzi na IJS. - Partnerji/ljudje: Aplikacija za pomoč iskanju ljudi - Računalniški slovarček: Robi odpre spletno stran z zaposlenih na projektu Ekosmart. računalniškim slovarčkom. - RRP 1 - Zasnova ekosistema pametnega mesta: V - Slovarji: Robi vam odpre slovarje v ozadju. ozadju se odpre kratek opis RRP 1. - Vreme: V ozadju se prikaže ARSO petdnevna - RRP 2 - Pametna mobilnost: V ozadju se odpre kratek vremenska napoved. opis RRP 2. - Znamenitosti Slovenije: Robi vam pomaga najti - RRP 3 - Aktivno življenje in dobro počutje: V ozadju željene znamenitosti po Sloveniji. se odpre kratek opis RRP 3. - Turist: Robi odpre spletno stran e-Turist, s katerim - RRP 4 - Elektronsko in mobilno zdravstvo: V ozadju lahko načrtujete izlet. se odpre kratek opis RRP 4. - 3D IJS: Robi odpre 3D IJS robota, ki zna tudi - RRP 5 - Integrirane zdravstvene storitve: V ozadju se zaplesati in vas vodi po IJS kot virtualni vodič. Pelje odpre kratek opis RRP 5. vas do iskane osebe po simuliranem IJS. - RRP 6 - Prototipi rešitev: V ozadju se odpre kratek - IJS TV, napovednik IJS: V ozadju si lahko ogledate opis RRP 6. vsebino, ki se predvaja na IJS kanalu, zasedenost - Partnerji/inštitucije: Konzorcij programa Ekosmart predavalnic itd. sestavlja 25 partnerjev z različnih področij delovanja. - Program Ekosmart: V ozadju si lahko ogledate opis in 2.2 Čakalne vrste namen projekta Ekosmart. Kratek opis: Asistent Čakalne vrste, dostopen na [2], je - Prototipi: Povezava na seznam prototipov. namenjen iskanju čakalnih dob za zdravstvene posege pri posameznih ustanovah. Asistent deluje tako, da v iskalno okno vpišemo poseg, ki ga želimo (asistent nam lahko sam ponudi več opcij), nato izberemo, kako hitro potrebujemo poseg in regijo Slovenije, zatem pa nam asistent ponudi vse možne ustanove, pri čemer jih razporedi v vrstnem redu od najkrajše do najdaljše časovne dobe. Asistent izpiše tudi kontaktne podatke od zdravstvenih ustanov, zato se lahko hitro naročimo na željeni poseg [3]. 2.3 Stres Kratek opis: Cilj študentskega projekta OSVET je spletni pogovorni svetovalec za zaznavanje stresa pri uporabnikih in nudenje psihosocialne pomoči preko spleta. V spodnjem okencu vpišemo svojo težavo oz. počutje, virtualni svetovalec pa nam pomaga s svojo bazo podatkov. Če nečesa ne ve, nas Slika 3. EkoSMART- asistent, dostopen na spletni strani [6]. prosi, da svojo težavo opišemo kako drugače. Svojo stopnjo stresa lahko ocenimo tudi z uporabo ankete [3]. 2.5 Meta asistent Kratek opis: Meta asistent, imenovan Metka, nam da povezavo na ostale občinske asistente. V levem zgornjem kotu asistenta so na voljo vse aplikacije, pod njimi lahko izberemo tudi regije in občine [7, 8] za zožitev interakcije. Aplikacije: - Splošno: Asistent vam da splošne informacije o občini, občinskem svetu, zaposlenih in zanimivostmi občine. - Okolje, prostor in komunala: Asistent vam odgovori Slika 2. Spletni pogovorni svetovalec za zaznavanje stresa na vprašanja v zvezi s prometom, okoljem in [4]. komunalo. 390 - Kultura, šport in izobraževanje: Asistent vam odgovori na vprašanja glede kulture, športa in izobraževanja. - Zdravstvo in sociala: Asistent vam odgovori na vprašanja v zvezi z zdravstvenim domom, zdravniki, oskrbo na domu in socialno službo. - Zaščita in reševanje: Asistent vam odgovori na vprašanja povezana s civilno zaščito, gasilci, policijo, načrti zaščite in reševanja. - Vloge in obrazci: Asistent za vas poišče različne vloge in obrazce. - Kmetijstvo in gospodarstvo: Asistent odgovori na vprašanja glede kmetijstva, gozdarstva, prehrane in gospodarstva. - Storitve in obrtniki: Asistent odgovori na vprašanja glede storitvenih dejavnosti v občini (npr. seznam trgovin, lokacije bankomatov, kinodvorane, banke, Slika 5. Povpraševanje je možno po Sloveniji, pokrajinah ali seznam obrtnikov ipd.). občinah. - Turizem: Asistent vam odgovori na vprašanja glede turizma (prenočišča, znamenitosti, turistično Aplikacije: informacijski centri). - Splošno: Asistent vam da splošne informacije o - Moja občina: Asistent v ozadju odpre spletno stran, občini, občinskem svetu, zaposlenih in zanimivostmi kjer si lahko preberete novice, dogodke in zanimivosti občine. posameznih občin. - Okolje, prostor in komunala: Asistent vam odgovori - Vreme: V ozadju se odpre spletna stran ARSO, kjer na vprašanja v zvezi s prometom, okoljem in lahko vidite petdnevno vremensko napoved. komunalo. - Prevajanje: V ozadju se vam prikaže prevajalnik - Kultura, šport in izobraževanje: Asistent vam podjetja Amebis, kjer lahko prevajate med odgovori na vprašanja povezana z različnimi slovenščino, nemščino in angleščino. športnimi kot tudi kulturnimi društvi in klubi, - Slovarji: V ozadju lahko uporabite slovarje za večje izobraževalnimi ustanovami in o prireditvah. evropske jezike. - Zdravstvo in sociala: Kontaktni podatki za prvo pomoč. - Zaščita in reševanje: Asistent vam odgovori na vprašanja povezana s civilno zaščito, gasilci, policijo, načrti zaščite in reševanja, ravnanjem v primeru naravnih nesreč. - Vloge in obrazci: Asistent vam lahko poišče različne vloge in obrazce. - Kmetijstvo in gospodarstvo: Asistent vam odgovori na vprašanja povezana s kmetijstvom, gozdarstvom, prehrano in gospodarstvom na splošno. - Storitve in obrtniki: Asistent odgovori na vprašanja povezana s storitvenimi dejavnostmi v občini (seznam trgovin, lokacije bankomatov, kinodvorane, banke, seznam obrtnikov, itd.) - Turizem: Asistent vam nudi koristne informacije Slika 4. Meta asistent, ki je dostopen na [8]. glede turizma. - Novice in prireditve: Na spletni strani v ozadju si lahko ogledate novice in prihajajoče prireditve v občini. 2.6 Občine - Moja občina: Spletna storitev Moja občina je namenjena seznanjenju občanov slovenskih občin z Kratek opis: Vsaka izmed 200 občin ima svojega občinskega novicami. asistenta, do seznama asistentov lahko dostopate na spletni - Vreme: V ozadju se odpre spletna stran ARSO, kjer strani [8], pod zavihkom Občine. Po kliku na “Asistenti za si lahko ogledate vremensko napoved. občine” se vam odprejo vse povezave na občinske asistente. - Izberemo občino in odpre se nam spletna stran občine in Slovenski turizem: V ozadju se odpre spletna stran občinski asistent v levem zgornjem kotu. e-Turist, kjer lahko načrtujete ogled. Na Sliki 5 je primer - Občinska TV: V ozadju se odpre spletna stran TV asistenta, ki prikazuje pokrajine. Ponujene aplikacije so pri vseh občinskih asistentih enake. IJS. 2.7 3D-asistent Kratek opis: 3D-asistent je virtualni vodič po IJS. Če ga vprašate, kje je določeni posameznik, vas popelje pred njegova vrata, tako da hodi pred vami v virtualni 3D resničnosti. 391 3D-asistent omogoča tudi, da vsak zgradi svoj virtualni posnetek resničnega prostora in svojega 3D-asistenta. 3D- asistenti povezujejo številne sisteme, razvite na Odseku za inteligentne sisteme IJS, ki omogočajo virtualno izkušnjo. Uporablja se ga lahko preko spletnega portala, enostavno pa se vzpostavi tudi samostojna aplikacija za pametni telefon Android ali iPhone, računalnik Windows ali Mac. 3D-asistent pride do izraza predvsem pri uporabi v občinskih stavbah, podjetjih, inštitutih, muzejih ali znamenitostih, saj do neke mere nadomešča vodiča ali vratarja. Slika 7. prikazuje ZDUS asistenta, ki je dostopen na spletni strani [11]. Aplikacije: - Splošno: Na spletni strani v ozadju si lahko preberete splošno o društvu ZDUS. - Društva: V ozadju lahko vidite vsa društva in klube. Društva so prikazana na zemljevidu, tako da se lažje orientirate. - Aktivno staranje: V ozadju lahko dobite koristne informacije o aktivnem staranju, vključno s predlogi o tem, katere vaje izvajati. - Zdravje: V ozadju lahko dobite koristne informacije o zdravju (pomoč pri različnih boleznih, kaj je priporočljivo jesti, zdravilne rastline). - Bivalna kultura: Na spletni strani v ozadju si lahko preberete več o bivalni kulturi starejših. Slika 6. 3D IJS asistent, ki ga lahko upravljamo s pomočjo - Letovanje: V ozadju lahko dobite informacije o tipk: W, A, S, D. Po kliku na robotovo glavo mu lahko zanimivih izletih in potovanjih. zastavimo vprašanje. - Zakonodaja: V ozadju si lahko preberete več o sprejetih zakonih. - Aktivno državljanstvo: Na spletni strani v ozadju lahko Do 3D-asistenta dostopate tako, da v svojem občinskem asistentu kliknete levo zgoraj “Aplikacije” in dobite koristne informacije o aktivnem državljanstvu, ki nato “3D- asistent”. spodbuja, da starejši ostanejo aktivni in avtonomni. Primer zanimivega 3D-asistenta je 3D IJS asistent, ki - Informatika: V ozadju lahko dobite koristne informacije je prikazan na Sliki 6. Nudi druge funkcije, recimo če kliknete o tem kako starejše naučiti uporabe IKT, ki je v na glavo 3D robota, po želji tudi zapleše [9, 10]. današnjem času nujna. - Medgeneracijsko sodelovanje: V ozadju je opis načinov medgeneracijskega sodelovanja in primeri dobre prakse. - Slovarji: Zdusko vam pomaga razložiti neznane besede. 2.8 ZDUS - Vreme: V ozadju se vam odpre spletna stran ARSO, Kratek opis: kjer lahko vidite vremensko napoved. ZDUS (Zveza društev upokojencev Slovenije) ima na svoji - Slovenski turizem: Zdusko vas poveže s spletno stranjo spletni strani ZDUS asistenta, imenovanega Zdusko. Le ta vam e-Turist, kjer lahko načrtujete svoj ogled Slovenije. odgovori na vprašanja, ki mu jih zastavite. V levem zgornjem kotu asistenta pa so na voljo tudi aplikacije, ki jih lahko izberete. 2.9 SVIZ Kratek opis: Na spletni strani SVIZ (Sindikat vzgoje, izobraževanja, znanosti in kulture Slovenije) je dostopen virtualni asistent, imenovan Svizec. V levem zgornjem kotu lahko izberete aplikacije, v iskalno okno pa vpišete vprašanje. 392 najpomembnejši je bil opisan Asistent IJS (2.1.), v nadaljevanju pa še dva, ki sta povezana z zdravjem, to sta Čakalne vrste (2.2.) in Stres (2.3.). Sledijo asistent za pametna mesta EkoSMART-asistent (2.4.), nato pa Meta asistent (2.5.), ki pokriva Občine (2.6.). Sledi bolj zabaven 3D-asistent (2.7.), do katerega lahko dostopamo iz občinskih asistentov. Na koncu je opisan asistent ZDUS (2.8.), ki je v pomoč starejšim. 4. REFERENCE [1] Povezava na asistenta IJS [20. 9. 2018], dostopna na http://www.projekt-asistent.si/ijs [2] Povezava na asistenta Čakalne vrste [20. 9. 2018], dostopna na https://salty-retreat-14326.herokuapp.com/ Slika 8. Asistent SVIZec. [3] Spletna stran Zdravje [20. 9. 2018], dostopna na https://ui- obcine.ijs.si/zdravje/ Aplikacije: [4] Spletni pogovorni svetovalec za zaznavanje stresa [20. 9. - Splošno: Asistent vam odgovori na splošna vprašanja 2018], dostopno na http://poluks.ijs.si:12345/ o SVIZu. [5] Spletna stran EkoSMART [26. 9. 2018], dostopno na - Vreme: V ozadju se odpre spletna stran ARSO, na http://ekosmart.net/sl/ekosmart/ kateri si lahko ogledate vremensko napoved. - Prevajanje: V ozadju se pokaže prevajalnik podjetja [6] Povezava na EkoSMART-asistent [28. 9. 2018], dostopno Amebis, kjer lahko prevajate stavke med slovenščino, na http://projekt-asistent.si/ekosmart nemščino in angleščino. [7] Spletna stran Asistenti [28. 9. 2018], dostopno na - Slovarji: V ozadju se odpre slovar za večje evropske https://ui-obcine.ijs.si/asistenti/ jezike. [8] Spletna stran Meta asistent [28. 9. 2018], dostopno na - Računalniški slovarček: V ozadju se nahaja slovarček http://projekt-asistent.si/meta-asistent računalniških izrazov. - [9] Spletna stran 3D-asistent [28.9. 2018], dostopno na Matematika / Znanje: Odpre se spletna stran Wolphram Alpha, zmogljivo orodje za reševanje https://ui-obcine.ijs.si/3d-asistent/ matematičnih, fizikalnih, kemijskih in drugih [10] Povezava na 3D-asistent [28. 9. 2018], dostopno na znanstvenih nalog. ,http://3d.ijs.si/ - Počitnice: V ozadju se odpre spletna stran Odpočij.si, [11] Povezava na asistenta ZDUS [26. 9. 2018], dostopno na kjer lahko najdete počitniški kraj. http://www.projekt-asistent.si/zdus - Predpisi in zakoni: Dostopni splošni zakoni na spletni strani v ozadju. 3. ZAKLJUČEK V prispevku je bil predstavljen pregled večine asistentov, ki so bili razviti na Odseku za inteligentne sisteme Instituta “Jožef Stefan”. Skupno jih je med 200 in 300. Kot prvi in 393 Varnostna ura IJS Mateja Drnovšek Tine Kolenik Matjaž Gams Odsek za inteligentne sisteme Odsek za inteligentne sisteme Odsek za inteligentne sisteme Institut »Jožef Stefan« Institut »Jožef Stefan« Institut »Jožef Stefan« Jamova cesta 39 Jamova cesta 39 Jamova cesta 39 1000 Ljubljana 1000 Ljubljana 1000 Ljubljana mateja.drnovsek@ijs.si tine.kolenik@ijs.si matjaz.gams@ijs.si POVZETEK 2. Opomniki: Opomnike lahko nastavite na spletnem Starejša populacija se vse pogosteje spoprijema s težavami portalu ali jih pošljete preko SMS-a. Lahko so kakovostnega, varnega in samostojnega življenja. Situacija enkratni ali periodični. odpira vrata za inovativne rešitve z uporabo umetne inteligence. 3. Socialni klici (prijatelji): Če od domačega zaslona s Na Institutu »Jožef Stefan« na Odseku za inteligentne sisteme prstom po zaslonu potegnete 2-krat v desno stran, že vrsto let razvijamo sisteme za pomoč starejšim. Med njimi je prispete do kontaktov. Pokličete lahko do 4 shranjene tudi pametna ura, ki omogoča podaljšanje samostojnega številke. življenja, hkrati pa poveča varnost in udobje. Zaradi slednjega je primerna tudi za druge ciljne skupine, kot so odročni delavci, 4. Zunanji klici: Kdorkoli lahko pokliče na uro in začne reševalc pogovor – kot z mobilnim telefonom. i, raziskovalci, avanturisti ipd. V prispevku je splošno predstavljena najnovejša različica ure, ki je zmožna nenehnega 5. Spremljanje aktivnosti: Pametna ura spremlja spremljanja uporabnika. Med najpomembnejšimi funkcijami aktivnost uporabnika. Če se nenavadno malo ali sta zaznavavanje padcev in mirovanja, obenem pa ura vsebuje preveč giblje, kot se običajno, ura to sporoči več kot 15 funkcij, ki so predstavljene v prispevku. skrbniku. Lahko se nastavi, da se sprememba dnevne aktivnosti sporoči na določeno številko. Ključne besede 6. Geo-lociranje: Lociranje uporabnika v primeru EMZ, pametna ura, skrb za starejše, aktivno staranje, nevarnosti. zaznavanje padcev, zaznavanje mirovanja, srčni utrip. 7. Peskovnik: Če uporabnik zaide preko določenega območja, ura o tem obvesti skrbnika. 1. UVOD 8. Števec korakov: Dnevno štetje korakov. Ura nas Razvoj medicine in izboljšanje življenjskih pogojev sta opomni, če se moramo več gibati ali pa če smo že pripomogla k podaljšani življenjski dobi, a tudi hitrejši tempo naredili dovolj korakov v dnevu. življenja ter povečan obseg dela in zmanjšanje rodnosti, zaradi česar se je prebivalstvo začelo starati [1]. Daljše življenje je 9. Merjenje srčnega utripa: Ura meri srčni utrip eno velik civilizacijski dosežek, eden od ključnih izzivov minuto in nato izpiše povprečni utrip. pa je, kako ohraniti samostojnost in varnost starejših ter kako narediti 10. Avtomatski klic na pomoč v primeru padca: starost dostojno in prijetno tako za starejše kot njihove Pametna ura pokliče skrbnika, če se zgodi padec. sorodnike [2]. S tem namenom je bila razvita pametna ura za starejše 11. Avtomatski klic na pomoč v primeru mirovanja: . Razvoj pametne ure temelji na večletnem Pametna ura pokliče skrbnika, če uporabnik obmiruje raziskovalnem delu prepoznavanja aktivnosti in padcev ter dolgotrajnega in natančnega testiranja različnih prototipov pri 1-5 min (odvisno od načina delovanja). številnih ciljnih uporabnikih [3,4,5]. Končni rezultat je pametna 12. Klic na pomoč s pomočjo gumba: Ročni klic na ura, ki uporabniku ne poveča le varnosti, ampak tudi izboljša pomoč, ki ga lahko izvede uporabnik sam, če življenje s številnimi uporabniškimi funkcijami. potrebuje pomoč. 13. Prekinitev klicanja na pomoč. Če se sproži klic na Sistem povezuje starejše in njihove skrbnike ter jim pomaga pri pomoč, ga lahko uporabnik prekine. komunikaciji in hitri pomoči pri nezgodah. Primeren je za starejše, ki želijo dlje 14. Nastavljanje ure s komandnim modulom. Preko časa bivati v domačem okolju, za živeče v domovih za starejše in za uporabnike, ki delajo na tveganih komandnega modula je možno nastavljati uro. območjih. V prispevku predstavimo osrednjo komponento 15. Nastavljanje ure. Uro je možno nastaviti na več našega sistema, pametno uro, ki je namenjena izboljšanju načinov delovanja, parametrov itd. Nastavitve je varnosti uporabnika. možno doseči na skrit način, da ne bi npr. dementni uporabniki narobe nastavljali ure. 2. PAMETNA URA 16. Tovarniške funkcije pametne ure. Ura predstavlja 2.1 Osnovne funkcije nekoliko okrnjen mobilni telefon in ima vse osnovne Varnostna pametna ura IJS ima skupaj s komandnim modulom funkcije tovrstnih naprav. Možen je dostop do naslednje osnovne funkcije: tovarniških funkcij, vendar na skrit način. 1. Čas in datum: Na domačem zaslonu vidite čas, Nekaj teh funkcij je podrobneje opisanih v nadaljevanju. datum, signal, baterijo, števec korakov ter način delovanja. 2.2 Varnost Glavna funkcionalnost in razlog, zakaj si uporabniki sploh želijo takšen sistem, je varnost. En poglavitnih strahov, zakaj se 394 starejši odločijo za zapustitev domačega okolja ter prehod v poročanje o morebitnih alarmih in stanju ure ter enostaven varovano okolje (dom za ostarele, varovana stanovanja), je vpogled svojcem ter skrbnikom. ravno strah pred poškodbo in nezmožnostjo klica na pomoč. Žal Zaradi boljše pokritosti in zanesljivejšega delovanja je primarni se dandanes, kljub vsej sodobni tehnologiji, pogosto dogaja, da sistem, ki se uporablja za komunikacijo s strežnikom, SMS, se starejša oseba poškoduje (pade) v domačem okolju, ni pa naknadno pa smo omogočili tudi uporabo podatkovnega zmožna vstati in poklicati na pomoč. Zaradi dolgotrajnega (tudi prometa z uporabo REST-protokola. večdnevnega) ležanja na tleh s poškodbo se zdravstveno stanje izredno poslabša, okrevanje se podaljša, v najhujših primerih pa Zaradi varovanja osebnih podatkov se na strežniku shranjuje le lahko to privede do smrti. uporabniško ime (ki je poljubno), geslo ter telefonska številka ure, ki je hkrati tudi identifikacija naprave. Uporabnik lahko Sistem je zasnovan tako, da prepoznava določene tipe sam izbira, ali želi določeni osebi (uporabniku, skrbniku) nevarnosti in samodejno pokliče na pomoč v primeru, da dodeliti vpogled do podatkov. Da pa bi bilo delovanje sistema prepozna nevarnost. razširljivo, smo omogočili določene API-klice, ki omogočajo 2.2.1 Zaznavanje padcev povezljivost z drugimi sistemi. To je predvsem primerno za klicne centre, ki želijo ohraniti svoj obstoječ sistem za Zaznavanje padcev temelji na prepoznavanju značilnih pospeškov prejemanje klicev in hranjenje podatkov ter ga na enostaven , ki nastanejo med padanjem in po padanju. Glavni način nadgraditi z dodatnimi funkcionalnostmi, ki jih omogoča senzor, ki ga za to uporabljamo, je pospeškomer. Z uporabo pametna ura. dodatnih senzorjev, kot so barometer, giroskop in PPG, pa lahko natančnost algoritma še izboljšamo. Žal imajo osnovne (cenejše) ure le pospeškomer, višji cenovni razred pa omogoča 3. NAVODILA dodatne senzorje ter s tem višjo natančnost in manj lažnih Sistem umetne inteligence skuša izvajati nekatere funkcije, ki bi alarmov. jih skrbnik-človek s pridom uporabljal. Ura poveča varnost starejših, odraslih in otrok ter podaljša samostojnost življenja Glavna ideja algoritma je prepoznavanje treh obdobij padca; starejših. Možni so trije načini uporabe: gibanje pred padcem, visok pospešek, ki nakazuje padec in mirovanje po padcu (glej sliko 1). Na ta način lahko a) aktivni način, ko se sproži alarm ob vsakem daljšem prepoznamo vse hujše padce, po katerih je oseba tako mirovanju (npr. na izletu, ob zahtevnih hišnih poškodovana, da se ne more več premikati (niti poklicati na opravilih, za varnostnike, vlakovodje, delavce; pomoč). V primeru, da se oseba še lahko premika, trajanje baterije 8 ur), predpostavimo, da lahko pokliče na pomoč z uporabo gumba. b) normalni način, ko se sproži alarm ob velikem Na ta način drastično zmanjšamo število lažnih alarmov, pospešku in nato mirovanju, tj. padcu (npr. izlet; obenem pa zagotavljamo visok nivo varnosti. trajanje baterije 24 ur), in c) varčni način, ko ura ne zaznava padcev in mirovanja, ampak samo SOS-gumb (trajanje baterije 3 dni). Video na YouTube: Varnostna ura IJS (https://youtu.be/bokP2_x-Q3s) Več informacij na spletni strani: http://ura.ijs.si 3.1 Vklop/izklop 1. Za vklop pritisnemo in držimo srednji gumb (slika 2), dokler se na zaslonu ne izpiše »Lemfo 5«, to je znak, da se ura prižiga (ugaša). Po nekaj sekundah bo Slika 1. Prepoznavanje padcev iz pospeškov. Ob padcu nastanejo izstopajoči pospeški, po njem pa se stanje umiri. pametna ura v stanju pripravljenosti in na zaslonu se bo prikazal domači zaslon z uro in dnevom. 2. Za izklop zopet pritisnemo in držimo srednji gumb, 2.2.2 Prepoznavanje mirovanja dokler se na zaslonu ne pokaže slika 2. Pogosto se zgodi, da osebi postane slabo, omedli ipd., a ker ne  Če želimo uro ugasniti, pritisnemo modri gumb na stoji, pri tem ne pride do padca, zato sistem za prepoznavanje zaslonu (glej sliko 2). padcev ne sproži alarma. Z namenom, da bi prepoznali takšne  Če želimo uro ponovno zagnati, pritisnemo zeleni težave, smo razvili metodo za prepoznavanje nenavadnega gumb na zaslonu (glej sliko 2). mirovanja. Metoda ves čas spremlja premikanje osebe  Na sredini zgoraj je tudi SOS-gumb za ročni alarm. (zapestja). Ker merimo mikro-gibe, ki so prisotni skorajda vedno, ko je oseba budna (npr. tudi ko tipkamo po tipkovnici), enostavno določimo mejo, ki predstavlja, da je oseba omedlela. Z namenom zmanjševanja lažnih alarmov sistem v takšnem primeru najprej opozori uporabnika z 10-sekundnim vibriranjem. Če je uporabnik priseben, premakne zapestje, kar pomeni, da je vse v redu. V primeru, da se to ne zgodi, sistem sproži alarm. 2.3 Povezljivost Majhen zaslon na uri onemogoča enostavno upravljanje kompleksnih nastavitev, to je še posebej poudarjeno pri starejših uporabnikih, ki imajo pogosto težave z vidom. Zato smo se odločili, da uro povežemo s spletnim portalom (komandnim modulom), ki omogoča upravljanje ure na daljavo, 395 Slika 4. Polnjenje pametne ure. 3.4 Prehod med načini delovanja Način delovanja nastavimo tako, da na domačem zaslonu s prstom potegnemo od desne proti levi. Pojavi se okno »Moduli« in tri možne izbire (kot je opisano na začetku): a) aktivni način, tu je možnost izbire dveh:  zaznavanje padcev  zaznavanje mirovanja b) normalni način c) varčni način Izbrati je mogoče eno možnost izmed a), b) in c) s pritiskom znotraj ustreznega kroga. 3.5 Klic prijateljev Do kontaktov lahko dostopamo tako, da na domačem zaslonu s Slika 2. Pametna ura s prikazanim SOS-gumbom. prstom dvakrat potegnemo z leve proti desni. Pojavi se okno »Kontakti« z vnaprej shranjenimi številkami. Prijatelje 3.2 Kako pogledamo čas in datum? pokličemo tako, da držimo prst na željenem kontaktu.  Po prižigu ure se pokaže domači zaslon. Na sredini Opomba: Za klice mora biti vstavljena SIM kartica. zaslona se samodejno izpišejo čas, datum, signal, baterija, dan, števec korakov in način delovanja (glej 3.6 Samodejni klic na pomoč v primeru sliko 3). padca  Ura varčuje z energijo, zato se zaslon po nekaj V primeru, da padete in obmirujete 20 sekund ali eno minuto sekundah zatemni. Če je zaslon temen, ga oživimo s (se nastavi v »Napredne nastavitve«), bo pametna ura pritiskom na srednji gumb. samodejno poklicala na pomoč in bo klicala toliko časa, dokler koga ne prikliče. Navodila za preklic klica so spodaj. Pomembno: Uro nosite na roki, v roki, v žepu oz. nekje na sebi, da lahko zazna padec. 3.7 Ročni klic na pomoč Če želimo sami poklicati na pomoč, pritisnemo najprej srednji gumb, da je ura v stanju pripravljenosti, nato pa tri sekunde držimo spodnji gumb. Ura bo poklicala na pomoč, na zaslonu pa se bo pojavila slika 5: »Kličem na pomoč. Pritisni in drži za preklic.« Če želimo, lahko klic na pomoč prekinemo – glej navodila spodaj. 3.8 Preklic klica na pomoč Slika 3. Domači zaslon pametne ure. Ko ura kliče na pomoč, se na zaslonu pojavi slika 5: »Kličem na pomoč. Pritisni in drži za preklic.« Klic prekličemo tako, da 3.3 Polnjenje baterije pritisnemo na zaslon in držimo. Barva zaslona se bo spremenila Uro polnimo preko magnetnega polnilnika. Kabel za z rdeče na zeleno (slika 5) in klic se bo prekinil. polnjenje ima na eni strani magnet ovalne oblike, ki ga prislonimo na rumene kontakte na zadnji strani ure (glej sliko 4). Na drugi strani kabla je USB-priključek, ki ga vstavimo v adapter, le-tega pa v vtičnico. Ob začetku polnjenja pametne ure zaslišimo zvočni signal, govor in vibriranje, na zaslonu pa se pojavi znak za polnjenje. Slika 5. Prikazan je postopek za sprožitev in preklic klica na pomoč. 3.9 Srčni utrip Okno za merjenje srčnega utripa se nahaja levo od okna »Domači zaslon«. Ko je na zaslonu izpisan »Domači zaslon«, s prstom po zaslonu potegnemo z leve proti desni, tako da se na 396 zaslonu prikaže slika 6. Za začetek merjenja srčnega utripa 5. ZAHVALA pritisnemo in držimo na zaslon. Ura približno 25 sekund meri Raziskave in razvoj so nastale v okviru programa EkoSMART srčni utrip in ga nato prikaže na sredini zaslona. in so delno sofinancirane s strani Ministrstva za izobraževanje, Za izhod na »Domači zaslon« s prstom potegnemo po zaslonu znanost in šport ter Evropske unije iz Evropskega sklada za od desne proti levi. regionalni razvoj (ESRR). Zahvaljujemo se vsem, ki so sodelovali pri razvoju pametne ure: Jani Bizjak, Anton Gradišek, Hristijan Gjoreski, Luka Stepančič, Nejc Mlakar, Nejc Kovač, Tadej Magajna, Kristian Remsak, Samo Remec ter ostali sodelavci odseka E9. 6. REFERENCE [1] Lutz, W., Sanderson, W., in Scherbov, S. 2008. The coming acceleration of global population ageing. Nature. 451, 7179 (2008), 716–719. DOI= https://doi.org/10.1038/nature06516. Slika 6. Zaslon pred začetkom merjenja srčnega utripa. [2] WHO, 'Active ageing: a policy framework', 2002. [Splet]. Dostopno na: 4. ZAKLJUČEK http://www.who.int/ageing/publications/active_ageing/en/. V prispevku je opisana najnovejša različica Varnostne ure IJS. [Dostop: 21– Sept– 2018] Ura vsebuje več kot 15 funkcij, med katerimi sta najbolj [3] 'Varnostna ura ijs – Safety watch JSI', 2018. [Splet]. pomembni funkciji za avtomatsko zaznavanje padcev in Dostopno na: http://ura.ijs.si. [Dostop: 21– Sept– 2018] mirovanja. Pametna ura je bila razvita na podlagi večletnih izkušenj in številnih preizkusov prototipov na različnih [4] Bizjak, J., Gradišek, A., Stepančič, L., Gjoreski, H., and stopnjah razvoja z dejanskimi uporabniki, kar je tudi pripeljalo Gams, M. 2017. Intelligent assistant carer for active aging. do trenutnega izdelka, ki je že preizkušen in primeren za končne EURASIP Journal on Advances in Signal Processing. uporabnike. S pomočjo spletnega API-ja je tudi omogočena 2017, 1 (2017), 76. enostavna integracija v že obstoječe sisteme za e-oskrbo ter [5] Bochanovski, M., Gjoreski, H., Bizjak, J., and Gams, M. pomoč na daljavo, s čimer upamo na hiter in učinkovit prodor 2016. Smartwatch fall detection. V Proceedings 19th na trg, najprej v Sloveniji, nato pa v Evropi. international multiconference IS 2016, M. Bajec, Ur. Jožef Stefan Institute, 17–20. 397 New HRV biofeedback technique and monitoring its effects in a psychotherapeutic group of coronary patients with SAVVY mini ECG devices installed Dušan Enova Roman Trobec Katja Mesarič UMC Ljubljana, Division of Internal Institut »Jožef Stefan« UMC Ljubljana, Division of Internal Medicine Jamova 39 Medicine Zaloška cesta 7, 1525 Ljubljana 1000 Ljubljana, Slovenija Zaloška cesta 7, 1525 Ljubljana +386 1 522 81 52 roman.trobec@ijs.si katja.mesaric@kclj.si dusan.enova@kclj.si Katarina Slevec Borut Ambrožič Tej Enova UMC Ljubljana, Division of Internal UMC Ljubljana, Division of Internal Gimnazija Bežigrad Medicine Medicine Peričeva ulica 4, 1001 Ljubljana Zaloška cesta 7, 1525 Ljubljana Zaloška cesta 7, 1525 Ljubljana tej.enova@gimb.org katarina.slevec@gmail.com ambrozicborut@gmail.com ABSTRACT 2. NEW USE OF THE SAVVY DEVICE We believe that we should have a possibility to use the SAVVY In the past years, two techniques for raising heart rate medical device (mini ECG device, developed at the Jožef Stefan variability (HRV) were presented; 1) resonant breathing with a Institute) for the purpose of psychophysiological coronary frequency of 0.1 Hz and 2) rhythmic isometric contraction of rehabilitation. With these devices applicated on each patient, we skeletal muscles with a frequency of 0.1 Hz. Research has could simultaneously measure heart rate variability (HRV) on shown greater efficiency for resonant breathing techniques. the whole group of cardiac patients during psychophysical In our study, we want to test Combined respiratory and relaxation exercises and during exercises aimed at increasing muscular HRV technique, or CBM-HRV, in which muscular the HRV. The next step, however, would be the possibility of and respiratory techniques are combined. For this technique, lending these devices to patients for home use during a period preliminary experiments have shown greater efficiency, as of three months of coronary rehabilitation. In this way, it would compared to other techniques, since it has greatest contribution be easier to monitor the progress and cooperation of patients in to the increase in HRV. Because the technique is not physical means of regular practicing of these exercises at home. We demanding, it could be practiced by patients in the first days believe that this would also increase the motivation of patients after myocardial infarction (MI) in the lying position. In to carry out exercises at home. The results of home HRV addition, a possibility to use the SAVVY medical device (mini measurements could be sent by patients to Psychophysiological ECG device, developed at the Jožef Stefan Institute) for the rehabilitation clinic via e-mail or in some other, even more purpose of psychophysiological coronary rehabilitation is elegant way- for instance telemetry. stressed. 3.1. NEW HRV BIOFEEDBACK Keywords TECHNIQE Heart rate variability, Biofeedback, Resonant breathing, One of the authors of the HRV biofeedback training, professor Rhythmic isometric contraction of skeletal muscles, Combined Vaschillo E.G., presented two techniques for increasing the respiratory and muscular HRV technique, SAVVY medical heart rate variability: resonant breathing with a frequency of 0.1 device Hz and a rhythmic isometric contraction of skeletal muscles with a frequency of 0.1 Hz [4]. In his research, he found that 1. INTRODUCTION the technique of resonance abdominal breathing with a There is a worldwide occurring trend of increasing the number frequency of 0.1 Hz was more effective than the aforementioned of scientific studies on biofeedback training aimed to increase muscles contraction technique. By reviewing professor heart rate variability (HRV-BFB) in patients following a heart Vaschillo's article [4], authors Dušan Enova and Tej Enova attack, as well as in other coronary patients, and in patients with found that participants in the study of isometric muscular some other somatic and psychiatric illnesses [1,2,3]. In the contractions technique did not have a fixed, specific, breathing Internal Clinic, UKCL (Ljubljana UMC), at the Department of rhythm during the exercise, but they breathed spontaneously, Cardiac Rehabilitation, we are also preparing a study in each with their own rhythm. From the results of one of the cooperation with colleagues from the USA, in which we will participants in the study, it is evident that the latter was investigate the impact of psychophysiological coronary breathing with an average rhythm of 0.3 Hz or approx. 18 rehabilitation (standard physiotherapeutic rehabilitation + stress breaths per minute. Dušan and Tej Enova have, by trials on management + HRV-BFB) on the reduction of psychogenic and themselves, came up with biofeedback technique: simultaneous oxidative stress in patients following a heart attack (MI). isometric tensioning of the muscles of the crossed lower limbs 398 in a sitting position, namely for 4 seconds during the inhalation phase; and relaxation of all muscles for 6 seconds in the exhalation phase (the entire respiratory cycle is 10 seconds, 6 breaths per minute, frequency 0.1 Hz). Thus, the respiratory and muscular techniques are combined, both in the rhythm of 0.1 Hz (Combined respiratory and muscular HRV technique or CBM-HRV). It turned out that this new HRV technique is significantly more effective than resonant breathing technique. We are now planning a study to confirm these preliminary results on a sample of young and healthy people. In case of confirmation of this hypothesis, we will use this new HRV technique in the aforementioned planned study at the UKCL Internal Clinic. The technique is not physically demanding and could already be practiced (in the lying position) in the first days after the MI by hospitalized patients. 3.2. PRELIMINARY RESULTS OF THE NEW HRV BIOFEEDBACK TECHNIQUE Figures 1 to 4 show the results of HRV measurements parameters of the four techniques performed by the author himself in a state after experiencing stress (Saturday, 22.9.2018, in the morning). Measurements were done with the Nexus 10, MindMedia medical device. Results of the combined CBM1 technique (tensioning the leg muscles during the inhalation phase with a rhythm of 0.1 Hz) are presented in the final Figure 4. It is evident that, when practicing this technique, SDNN as a Figure 1. Breathing with a frequency of 0.2 Hz (average classical measure of heart rate variability, is significantly higher breathing 12 breaths per minute). SDNN 9.94; Stress Index than when practicing other techniques. This technique was 1091.11 performed after a breathing phase of 12 breaths per minute (0.2 Hz). The other two techniques were carried out later. Duration of each of four measurements presented in Figures 1. to 4. was 3 minutes, but only those intervals without artifacts were counted for calculation. Figure 2. Breathing with a frequency of 0.1 Hz (resonant therapeutic breathing 6 breaths per minute). SDNN 17.55; Stress Index 492.95 399 3.3.PROPOSED PHYSIOLOGICAL EXPLANATION OF THE COMBINED HRV- BFB TECHNIQUE FUNCTIONING According to the explanation in professor Vaschillo's article, it is evident that the physiological effect of isometric contractions of skeletal muscles slightly increases the overall body blood pressure. With a delay of about 5 seconds, the heart rate is adjusted after this change of blood pressure [5]. Therefore, it is clear that, by the principle of respiratory sinus arrhythmia (RSA), the heart rate rises slightly during the inhalation phase and slightly decreases during the exhalation phase, and that the amplitude of this variability is maximally increased during breathing with a frequency of 0.1 Hz - there occurs some sort of cardiac-vascular-vegetative nervous system resonance. By tensioning skeletal muscles, blood pressure is slightly increased, after 5 seconds the heart rate is slightly reduced. Within the resonance breathing with 0.1 Hz frequency, the ratio of 4sec / 6 sec was selected between inhalation and exhalation phase. With muscles tensioning for 4 seconds from the start to the end of the inhalation phase (CBM1), and after a time interval of 5 seconds (counted from the start of the inhalation), an additional slight increase in blood pressure in the exhalation phase (lasting 6 seconds) occurs. Consequently, the heart rate frequency in the exhalation phase is further reduced, which leads to a higher amplitude in the curve of heart rate variability. In the opposite case, when using CBM2 combined technique, muscles tensioning is present only in the exhalation phase. As a Figure 3. Combined CBM2 technique (tensioning skeletal consequence, there occurs some inhibition of heart frequency muscles during the exhalation phase with a 0.1 Hz rhythm). variability - thus a lower efficiency of this combined technique SDNN 18.93; Stress Index 616.99 in comparison with CBM1. Therefore, we suggest that only CBM1 technique is really more effective. 4.CONCLUSION In the paper, we presented the possibility of using a new HRV biofeedback technique to increase the heart rate variability, and at the same time suggested that the effects of this technique could be measured with Savvy medical device (mini ECG device, developed at the Jožef Stefan Institute) for the purpose of psychophysiological coronary rehabilitation. It turned out that this new HRV technique is significantly more effective than resonant breathing technique. We are now planning a study to confirm these preliminary results on a sample of young and healthy people. 5.REFERENCES [1] Yu, L., Lin, I., Fan, S., Chien, C. and Lin, T. 2018. One- Year Cardiovascular Prognosis of the Randomized, Controlled, Short-Term Heart Rate Variability Biofeedback Among Patients with Coronary Artery Disease. International Journal of Behavioral Medicine 25, 3 (Jun. 2018), 271-282. DOI= https://doi.org/10.1007/s12529-017-9707-7 [2] Lehrer, P. M., Vaschillo, E. G. and Vaschillo, B. 2000. Resonant frequency biofeedback training to increase cardiac variability: Rationale and manual for training. Applied Psychophysiology & Biofeedback, 25, 3 (Sep. 2000), 177-191. Figure 4. Combined CBM1 technique (tensioning skeletal [3] Vaschillo, E. G., Lehrer, P. M., Rishe, N. and Konstantinov, muscles during the inhalation phase with a rhythm of 0.1 M. 2002. Heart rate variability biofeedback as a method for Hz). SDNN 35.71; Stress Index 159.62 assessing baroreflex function: a preliminary study of resonance in the cardiovascular system. Applied Psychophysiology & Biofeedback, 27, 1 (Mar. 2002), 1-27. [4] Vaschillo, E. G., Vaschillo, B, Padina, R. J. and Bates, M. E. 2011. Resonances in the Cardiovascular System Caused by 400 Rhythmical Muscle Tension. Psychophysiology, 48, 7 (Jul. Significance of Resonance in the Heart Rate and Vascular Tone 2011) 927-936. DOI = 10.1111/j.1469-8986.2010.01156.x Baroreflexes, In Biostec, A. Fres, J. Filipe and H. Gamboa, Eds. Springer-Verlag, Berlin Heidelberg, 224-237. [5] Vaschillo, E. G., Vaschillo, B., Buckman, J. F., Padina, R. J. and Bates M. E. 2010. The Investigation and Clinical 401 The Summoner – "Izbir čnež" Aljaž Glavač Jernej Zupančič Matjaž Gams Fakulteta za računalništvo in “Jožef Stefan” Institute and “Jožef Stefan” Institute informatiko Jožef Stefan International Jamova cesta 39 Večna pot 113 Postgraduate School Ljubljana, Slovenia Ljubljana, Slovenia Jamova cesta 39 matjaz.gams@ijs.si aljaz.glavac97@gmail.com Ljubljana, Slovenia jernej.zupancic@ijs.com ABSTRACT The paper continues with Sections 2 and 3, where the proto- More and more human to human interactions are these days types and “summon” heuristics are described, in Section 4 we converted into the human to computer interaction. So called describe the agent selection, in Section 5 the user experience agents are replacing human role in serving the users with is addressed and Section 6 concludes the paper. basic information and executing basic tasks. These inter- actions are getting better over time, approaching the point 2. FIRST PROTOTYPE where humans cannot make the difference, whether there The first approach in creating a prototype was to ask all is a real human or a computer agent (virtual assistant) on available 200+ agents a given question and then calculate the other end of the dialog. Similar to a society of humans, a weight, based on which the Summoner would choose the there are also societies of virtual assistants. In this paper most suitable answer. The most suitable answer gives the we describe a meta-agent, i.e. a meta virtual assistant, that user the most information about the question, or at least given a question asks a society of virtual assistants and sum- guides the user in the right direction, where the answer can mons the best reply from them all. The “Summoner”, as we be found. The process is presented in Figure 1. call our system, is therefore an online service that finds the best answer to the users’ question from a predefined list of systems according to the chosen heuristics. Keywords best answer, assistant, text distance, intersect, information retrieval 1. INTRODUCTION Virtual assistants [2] represent a fast developing field with several companies developing at least one of them, e.g. Mi- crosoft, Apple, Google, etc. In Slovenia, Department of in- telligent systems of Jožef Stefan Institute developed several hundred virtual assistants [1]. For 220 Slovenian munici- palities we managed to design 200 virtual assistants, where the knowledge base was extracted from the municipalities’ web pages. 20 municipalities did not have any web page to start with. We call virtual assistants agents and vice versa continuously. Each agent can answer questions for their mu- Figure 1: Basic concept flow chart of the Summoner nicipality. If a user wants to know something about Ljubl- jana, the user needs to find the agent of Ljubljana and ask The user input is processed by removing symbols and stop- this agent a specific question. However, the users who don’t words and running it through lemmatization and stemming know which agent to ask have the problem of finding the services. Out of this step three sets of words are created: right agent. We wanted to resolve this gap so that the users raw (original), lemmas and stems. could get the most out of the already working 200 agents. The original question is then sent to each agent according This project had another goal – to combine all of the already to the municipality agent application programming interface working agents into one general agent. Previous experience (API) specification. Each agent, when asked, responds with demonstrated that often some agents find a good reply, but a Javascript object notation (JSON) object with included it is hard to predict which one will provide the best one the answer ID, the answer text and the corresponding link. for a particular question. Therefore, dynamic selection from The link is present if the agent does not find the answer or if several reply suggestions of different agents is needed. To the agent wants to display more information about the an- achieve this, several heuristics had to be applied, tested and swer. Each answer is then processed in the same way as the improved in order to get the desired performance. question – removing symbols and stop-words and perform- 402 ing lemmatization and stemming. Each answer is stored in a hash map, where the key is the unformatted answer and value is the score or the weight of the answer. Only the user input and agent answers are used for calculat- ing answer weight. We use the simple method of word inter- section between formatted user input and formatted answer, which are a set of lemmes and stems. The weight is simply the size of the intersect. The answer with the most words that also co-occur in the user input is ranked the highest. Figure 2: JSON response format 3. SECOND PROTOTYPE After the testing of the first prototype, it turned out that to the agent title. Doing so reduces the number of suitable retrieving answers from all 220 agents took most of the re- agent significantly for almost every question. The similar- sponse time. First improvement was to use threads for send- ity is again computed using the text distance calculations, ing and retrieving questions from all 220 agents. This feature which might help to improve agent selection. After some reduced the answer fetch time in average to about a quarter simple testing we discovered that we had to compare stems of the original time (Table 1). and lemmas of both, user input question and agent title too. We also added full match intersect, i.e. set of words inter- sect without the transformations of stemming and lemmati- Table 1: Use of threads and without threads, when zation. Additionally, we added a function over this weight requesting all agents with the same question that takes into account the agent title length. Each agent No threads Threads score is defined as the sum of sizes of intersect (raw, stems Time(s) 201 54 and lemmas) each divided by the number of words in the agent title. The final answer score is calculated as the sum In order to measure the performance of the Summoner, a of the weights obtained in computing the agent title similar- benchmarking system was set-up, which includes 100 (ques- ity to the question and agent answer similarity to the user tion, correct answer) pairs. The benchmarking system in- question, where the second weight is computed as described serts all 100 questions into the Summoner and compares the in the first prototype (Figure 3). See Table 3 for benchmark obtained answers to the correct ones that were defined man- results. ually. Using the prototype described in Section 2, i.e. cal- culating the size of word intersect, we get 93 of 100 answers right. Table 3: Benchmark score after implementing text distances into agent selection In order to improve this score, we included several more Text Jaro Stems and Stems, lem- complex text distance1 calculations that compare the user distance lemmas mas and full input and the answer. The computed text distance is then match used as the weight assigned to each answer. The perfor- Score 94 98 97 mance of different text distances was then assessed using the benchmarking system (Table 2). According to the benchmarking results the best score is ob- tained when only the agents, whose title similarity to the Table 2: Different text distances used for weight user question is computed as the interesect of lemmatized calculation and their performance in the benchmark and stemmed words, are asked a question. After the agent Text Jaccard Sorensen Bag Overlap Overlap, selection process we get a list of agents that are most suitable distance qval is for questioning. We identify three typical situations: None Score 91 92 92 93 92 1. Only one agent is found. We ask this agent and return According to the results none of the added text distance the answer. metrics improved our score. The best solution for matching user input with answer remained the simplest one – set of 2. More than one and less than ten agents are found. In the processed words intersection. The response is provided this situation only the found agents are queried for in the JSON format (Figure 2). the answer. The best answer is chosen according to the heuristic described in Section 2. 4. AGENT SELECTION Many of the user questions already have some information 3. If more than ten agents are found all the agents are about which municipality a question is referring to. So the queried for answer. In order to obtain all the answers next improvement idea was to remove the agents that we a special care has to be taken so that the requests to know will not provide a suitable answer. The question is for- the API are not timed out. warded to an agent only if the user input has any similarity 1https://pypi.org/project/textdistance 4.1 Response time 403 Figure 4: Profiling sorted by cumulative time The agent endpoint returns an answer as well as a back- ground website link. This link adds additional information to the answer. If the question is about the mayor of the mu- nicipality then the answer contains information about this mayor and the background website gives the user further and more in depth information about the mayor and the municipality. When investigated, these website links fall in one of three categories. Each of them is treated differently: • First category. Website is fully working and display- ing correct information about the user question. No further processing is made and the link is stored into the response. Figure 3: Summoner flowchart after implementing the agent selection • Second category. When checking the website availabil- ity, status code 404 is returned. Several municipality website links stored in the virtual assistants’ databases A study2 shows that when computer program takes more are outdated and those pages don’t exist anymore. than 0.1 second but less than 1 second to respond, user When this happens, link gets parsed and only base notices the short delay but stays focused on their current of the link is stored into the response, presenting the train of thoughts. More than 10 seconds and the flow is user the main municipality web page. To test or filter broken. Users will often leave the site rather than wait. a website link, a request to the municipality website The goal is to provide the correct answer in less than 10 has to be done. seconds, however, the response time in cases when all the • Third category. The website link is a redirect to na- agents are queried is very long (Table 4). jdi.si3, a Slovenian search engine. This happens when the agent does not have the answer to the given ques- tion. By redirecting the user to the search engine, the Table 4: Time it takes to return answer in different agent hopes that the user will get the information he situations of agent selection is searching for. To overcome this problem, we simply Situation 1 agent se- 2 – 10 agents >10 agents ignore the answers that include search engine results. lected selected selected Time(s) 1.1 6.4 54 5. USER EXPERIENCE We ran the profiling of the source code to determine what Two main goals of every virtual assistant are to give the user function or part of the program takes the most time. Evi- correct answer to his question and to respond as quickly as dently, the time to get all the agents to respond takes the possible. In order to enable easy user interaction the visual most time (Figure 4) – it is the only function that runs in look of the user interface needs to give a user an intuitive way threads. of using the assistant: providing input and answer display fields and the feedback about the state of the system. 4.2 The “unknown answer” problem We provide a simple user interface (Figure 6). On the bot- tom of the page there is the user input field, above it are the 2https://www.nngroup.com/articles/powers-of-10- time-scales-in-ux/ 3https://najdi.si 404 Figure 7: Different system status banners: top – in progress, middle – success, bottom – error We have developed a meta-assistant that enables choosing the correct answer for a given question, when several an- swers are provided by the agent society. Development pro- cess and system heuristic for choosing the right agent and answer were described together with a short presentation of the web application that enables the user to interact with Figure 5: Unknown answer problem visualized the Summoner system. Virtual assistants don’t possess true human intelligence, but response messages from the system, followed by the system only try to imitate human interaction and respond to some status line and the web page with additional information sort of user request based on some heuristics. Disadvan- regarding the system or the website that corresponds to the tage of this type of virtual assistants is that they, unlike provided answer. real artificial intelligence4, cannot learn on their own. For example, with municipality elections new mayor might be elected. Consequently, a person would need to rewrite some entities of the assistants’ knowledge database, which would then enable the Summoner to obtain the correct answer. Looking forward, this type of virtual assistants is improving with user interaction, since the interactions are stored in the database, a system could be designed in the future that would learn to answer user questions on its own. However, in order to implement such a system, several hundred question- answer pairs have to be obtained first. Our Summoner has room for improvement, with the first step of improving the underlying agents by decreasing the response time and improving the knowledge base. Future work will also include improving the user experience in using the web application. Additionally, the goal is to provide search across the municipalities in Slovenian, English and German language, enabling wider user base. Further, other publicly available virtual assistants could be included into the agent society, enabling the Summoner to provide the answer to a wider range of questions. 7. ACKNOWLEDGMENTS The work was co-funded by Cooperation Programme Inter- Figure 6: Summoner web site user interface reg V-A Slovenia-Austria 2014-2020, project AS-IT-IC. As mentioned before, if requests are made to all agents, the 8. REFERENCES long time to retrieve the answer can make the user leave the [1] D. Kužnar, A. Tavčar, J. Zupančič, and M. Duguleana. site, leaving the impression that the system does not work. Virtual assistant platform. Informatica, 40(3):285, 2016. To give a proper feedback about the state of the system we [2] M. Ožek, M. Gams, and J. Krivec. Analiza delovanja added an animated status bar, that gives the user visual virtualnega svetovalca. In Proceedings of the 12th feedback when the Summoner is processing the input. The International Multiconference Information Society - IS animation is a banner above answer display field which uses 2009, volume A, pages 116–119, Ljubljana, Slovenia, different colors, depending on the state of the system. Blue 2009. Jožef Stefan Institute. animation is displayed when the Summoner is processing the answer and it makes the user feel like the website is thinking. When the answer from the service is retrieved, successful or not, the color changes according to the status. Green marks successful and red unsuccessful response (Figure 7). 4http://www.alanturing.net/turing_archive/pages/ 6. CONCLUSION Reference%20Articles/What%20is%20AI.html 405 Zaznavanje srčnega popuščanja z analizo srčnih tonov Anton Gradišek Martin Gjoreski Borut Budna Institut “Jožef Stefan” Institut “Jožef Stefan” Institut “Jožef Stefan” Ljubljana, Slovenija Ljubljana, Slovenija Ljubljana, Slovenija anton.gradisek@ijs.si Monika Simjanoska Matjaž Gams Gregor Poglajen Faculty of Computer Science and Institut “Jožef Stefan” Univerzitetni klinični center Ljubljana Engineering Ljubljana, Slovenija Ljubljana, Slovenija Ss. Cyril and Methodius University Skopje, Makedonija POVZETEK slabšanje simptomov in znakov srčnega popuščanja največkrat že Kronično srčno popuščanje je pogosta kronična bolezen predvsem pomeni, da gre za polno razvito epizodo poslabšanja SP. V pri starejših od 65 let. Ob poslabšanju zdravstvenega stanja je zadnjem času pa ugotavljamo, da se nekateri fiziološki parametri (npr. dodatni srčni toni, porast tlakov v pljučnem krvotoku) pogosto potrebna hospitalizacija, zato razvijamo novo pričnejo spreminjati že nekaj tednov preden pride do klinično telemedicinsko metodo, ki bo na podlagi analize zvoka srčnih tonov sposobna prepoznati poslabšanje stanja in tako preprečiti očitnega poslabšanja SP. Prve analize [2] že kažejo, da zgodnje prepogoste hospitalizacije. V prispevku se osredotočimo na prvi ugotavljanje spreminjanja teh parametrov pomeni precej učinkovitejšo obravnavo SP, saj je epizod polno razvitega korak te metode, na prepoznavanje zdravih posameznikov in pacientov s poslabšanjem srčnega popuščanja. Metoda je poslabšanja srčnega popuščanja manj, s tem pa je tudi manj sestavljena iz filtriranja in segmentacije posnetkov, izračuna potrebe po hospitalni obravnavi teh bolnikov. značilk, gradnje modelov strojnega učenja in združevanja V zadnjem času smo priče hitremu razvoju telemedicine, ki klasifikacijskih algoritmov. Na bazi posnetkov 158 zdravih omogoča spremljanje pacientov na daljavo in tako znatno posameznikov in 40 bolnikov smo dosegli 98,2 % natančnost. razbremeni tako pacienta kot tudi zdravstveni sistem. V pričujočem prispevku predstavimo začetke študije, v kateri Ključne besede uporabimo analizo zvoka srčnih tonov za prepoznavanje Srčno popuščanje, analiza zvoka, strojno učenje. poslabšanja srčnega popuščanja. Cilj je razviti metodo, s pomočjo katere bo uporabnik lahko sam spremljal svoje zdravstveno stanje. Uporabnik bo občasno z mikrofonom posnel bitje svojega srca, 1. UVOD algoritem pa bo ocenil, če se je stanje srčnega popuščanja Kronično srčno popuščanje (angleško: Chronic Heart Failure, poslabšalo do te mere, da je potreben posvet z zdravnikom. CHF) je kronična progresivna bolezen, pri katerem srce ni Dolgoročno bo ta pristop omogočil pravočasno prepoznavanje sposobno črpati dovolj krvi, da bi zadostila potrebam telesa [1]. V poslabšanja in posledično zmanjšal število hospitalizacij. Koncept razvitem svetu bolezen prizadene 1-2 % splošne populacije in je prikazan na Sliki 1. 6-10 % starejših nad 65 let. Glede na trenutno veljavna priporočila Evropskega združenja za kardiologijo srčno popuščanje (SP) diagnosticiramo s pomočjo kliničnega pregleda, z določitvijo nivoja biološkega označevalca SP v krvi (NT-pro BNP ali BNP) ter z ultrazvokom srca. Kljub velikem napredku na področju medikamentozne in nemedikamentozne obravnave kroničnega srčnega popuščanja je prognoza te bolezni še vedno slaba, saj se tudi v zadnjih letih 5-letno preživetje giblje okrog 50 %. Tipično se v kliničnem poteku CHF izmenjujejo obdobja dobrega počutja (tj. kompenzirane faze) in obdobja poslabšanja SP (tj. faze dekompenzacije), kjer bolniki opazijo izrazitejše poslabšanje telesne zmogljivosti, težje dihanje in otekanje v okončine in/ali trebuh. Pogosto je potrebno poslabšanje SP zdraviti hospitalno, saj ti bolniki potrebujejo intravenozna zdravila za učinkovito rekompenzacijo. Zgodnja prepoznava poslabšanja SP pa omogoča, da lahko z ustreznimi terapevtskimi ukrepi (največkrat z omejitvijo vnosa tekočine in povečanjem Slika 1. Shematski prikaz razvoja poslabšanja srčnega odmerka zdravil za odvajanje vode) bolniku hospitalizacijo popuščanja. V zgornjem primeru mora pacient zaradi prihranimo. Izkušen zdravnik lahko nastajajoče poslabšanje SP poslabšanja v bolnišnico, v spodnjem primeru poslabšanje prepozna glede na slabšanje bolnikovega počutja in glede na poslabšanje znakov SP, ki jih lahko ugotovi ob pregledu bolnika. zaznamo dovolj zgodaj, da lahko z ustrezno intervencijo preprečimo nadaljnjo Pri nekaterih bolnikih s poslabšanjem SP lahko pri osluškovanju poslabšanje. (fonokardiografija) srčnih tonov opazimo dodatne srčne tone (tipično tretji ton, ki se pojavi 0,1-0,2 s za drugim tonom). Žal 406 V tej študiji se osredotočimo na prvi korak te metode, tj. na algoritme strojnega učenja na teh 15 značilkah, iz tega dobimo prepoznavanje zdravih posameznikov in dekompenziranih končno odločitev ali gre za zdravega ali za bolnika. pacientov. Za snemanje srčnih tonov smo uporabili profesionalni digitalni stetoskop, za prepoznavanje pa metodo, ki temelji na strojnem učenju. V nadaljevanju opišemo metodo in trenutne rezultate. 2. MATERIALI IN METODE 2.1 Baza podatkov V študiji smo zbrali posnetke 158 zdravih posameznikov in 40 posnetkov bolnikov z razvito epizodo srčnega popuščanja (dekompenziranih). V vzorcu zdravih je bilo 104 moških in 54 žensk, povprečna starost 33,2 let. Z izrazom zdravi tu označujemo posameznike, ki niso imeli nobenih medicinskih stanj, ki bi imele za posledico neobičajne srčne tone. V vzorcu bolnih je bilo 22 moških in 18 žensk, povprečna starost 47,7 let. Bolnike smo posneli v bolnišnici med hospitalizacijo. Za snemanje smo uporabili profesionalni medicinski digitalni stetoskop 3M Littmann Electronic Stethoscope Model 3200. Posnetek, fonokardiogram, je bil vedno sneman na Erbovi točki, nad tretjim medrebrnim prostorom levo od grodnice. Posamezen posnetek je dolg do 30 s, pri nekaterih posameznikih smo posneli več kot en posnetek, da smo povečali količino podatkov v učni množici. Pred začetkom študije smo pridobili pozitivno mnenje Komisije za medicinsko etiko. 2.2 Metoda Metoda za prepoznavanje bolnikov in zdravih je sestavljena iz treh ključnih korakov, kot prikazuje Slika 2. Prvi korak obsega filtriranje in segmentacijo signala. Za filter je bil izbran nizkopasovni Butterworthov filter s pragom 1 kHz, ta prag je bil izbran na podlagi študije [3], kjer so ugotovili, da večina relevantnih srčnih tonov nastane pod tem frekvenčnim območjem. Signal smo razbili na segmente, dolge 1 s, pri tem so se zaporedni segmenti prekrivali po pol sekunde (segmentacija z drsečim oknom). Nato odstranimo segmente z energijo, manjšo od mediane posameznega posnetka – tako odstranimo segmente, kjer se bitje srca slabo sliši. V naslednjem koraku iz vsakega segmenta izračunamo vrsto značilk. Pri tem uporabimo odprto knjižnico značilk OpenSmile [4]. Knjižnica vsebuje značilke tako v časovni kot tudi v frekvenčni domeni, skupaj jih je 1582. Po izračunu značilk je tako vsak segment predstavljen z vektorjem 1582 vrednosti, ki jih uporabimo za gradnjo modelov strojnega učenja. Tretji korak je sestavljen iz treh faz. V prvi fazi zgradimo modele strojnega učenja na podlagi posameznih segmentov. Pri tem testiramo vrsto različnih algoritmov, kot so J48, Naivni Bayes, Random Forest, kNN, SVN in drugi. Tu nas vodi ideja, da se Slika 2. Posamezni koraki metode za prepoznavanje srčnega lahko posamezni algoritmi drugače odzivajo na določene strukture popuščanja. v podatkih. V drugi fazi združimo napovedi modelov na posameznih segmentih v napoved na celotnem posnetku. Tu 3. REZULTATI izhajamo iz dejstva, da vsi segmenti istega posnetka pripadajo Delovanje metode smo preizkusili po načelu »izpusti en istemu razredu (zdravi ali bolni). Združevanje poteka z uporabo minimuma, maksimuma in povprečja napovedi modelov na posnetek«. To izvedemo tako, da iz baze podatkov izločimo en posameznih segmentih. Če denimo uporabljamo pet različnih posnetek, uporabimo preostale posnetke kot učno množico, na modelov in vsak posnetek razdelimo na deset segmentov, bo vsak kateri naučimo metodo, potem pa jo uporabimo za določitev model za vsak segment določil verjetnost, da gre za bolnika. Iz teh izločenega posnetka. Postopek ponovimo za vsak posnetek v bazi. verjetnosti izračunamo povprečno vrednost, minimum in Pri evaluaciji natančnosti algoritma naletimo na manjši problem z maksimum, torej skupno 15 značilk. V tretji fazi poženemo interpretacijo. Ker smo uporabili izrazito neuravnotežen vzorec 407 (158 zdravih in 40 bolnikov, pri tem, da smo pri zdravih pogosto posneli vsaj dva posnetka, da smo povečali učno množico), je večinski razred skoraj 90 % - zanesljivost algoritma, ki je nad 5. ZAHVALA 90 %, je tako pravzaprav pričakovana. Tu zato navedemo rezultate, ki smo jih dobili na nekoliko manjšem set podatkov iz Raziskave in razvoj so bile izvajane v okviru programa prejšnjega koraka te študije [5]. Na množici 99 zdravih in 23 EkoSMART in so delno sofinancirane s strani Ministrstva za bolnikov, kjer smo imeli večinski razred 81 %, izobraževanje, znanost in šport ter Evropske unije iz Evropskega smo dosegli 96 % natančnost. Pri študiji se pokaže, da z uporabo da sklada za regionalni razvoj (ESRR). Zahvaljujemo se tudi vsem, pristopa s kombinacijo klasifikatorjev vsakič dobimo bistveno boljše ki so sodelovali pri zbiranju posnetkov. klasifikacijske rezultate kot če bi upoštevali samo večinski razred. Če se osredotočimo samo na bolnike, vidimo, da naša metoda 6. REFERENCE pravilno zazna 87 % primerov, pri zdravih pa je ta natančnosti 97 [1] J. J. V. McMurray et al., 2005, Heart failure, Lancet, 365, %. 1877-1889. [2] W. T. Abraham et al., 2016, Sustained efficacy of pulmonary 4. ZAKLJUČEK artery pressure to guide adjustment of chronic heart failure Predstavili smo metodo za prepoznavanje srčnega popuščanja na therapy: complete follow-up results from the CHAMPION podlagi zvoka srčnih tonov. Metoda uporablja kombinacijo randomised trial, Lancet, 387, 453-461. algoritmov strojnega učenja in lahko dobro razpoznava med [3] S. Choi and Z. Jiang, 2008, Comparison of envelope zdravimi posamezniki in bolniki z razvito epizodo srčnega extraction algorithms for cardiac sound signal segmentation, popuščanja. Začetni rezultati so obetavni, seveda pa je za resno Expert Systems with Applications, 34, 2, 1056–1069. oceno potrebno testiranje na večji množici posnetkov. Poleg tega se moramo zavedati, da smo v tej začetni študiji ločevali le med [4] F. Eyben, M. Wöllmer, and B. Schuller, 2010, Opensmile: the munichversatile and fast open-source audio feature ekstremnima primeroma – med popolnoma zdravimi posamezniki extractor, in Proceedings of the 18th ACM international in med hospitaliziranimi pacienti. V nadaljevanju študije načrtujemo analizo posnetkov pacientov z različno izraženimi conference on Multimedia. ACM, 1459–1462. stopnjami poslabšanja srčnega popuščanja. To nam bo [5] M. Gjoreski, M. Simjanoska, A. Gradišek, A. Peterlin, M. omogočalo, da bomo zaznali poslabšanje in pacienta napotili k Gams, G. Poglajen, 2017, Chronic heart failure detection zdravniku, še preden bi bila potrebna hospitalizacija. V tej fazi from heart sounds using a stack of machine-learning smo za snemanje srčnih tonov uporabili profesionalni stetoskop. classifiers. The 13th International Conference on Intelligent Cilj je, da bo pacient lahko za snemanje uporabil mikrofon, ki ga Environments, 14-19. priklopi na telefon, metoda za zaznavanje popuščanja pa bo tekla na telefonu ali v oblaku. 408 Elektrokardiografski senzor: uporaba med družinskimi zdravniki Staša Kocjančič Viktor Avbelj Zdravstveni dom Murska Sobota Institut Jožef Stefan Grajska ulica 24 Jamova cesta 39 SI-9000 Murska Sobota SI-1000 Ljubljana stasa.kocjancic@gmail.com viktor.avbelj@ijs.si IZVLEČEK Obravnava bolnika z motnjo ritma v Motnje ritma so pogosto prisotne pri bolnikih v ambulantah ambulanti družinskega zdravnika družinske medicine in tako so družinski zdravniki in tudi Pri obravnavi vsakega bolnika z aritmijo je potrebno odgovoriti specialisti drugih kliničnih strok soočeni s problemom, kako na naslednja vprašanja: katera motnja ritma je prisotna; ali ga ta obravnavati take bolnike. V Zdravstvenem domu Ljubljana smo motnja ritma ogroža trenutno in ali ima dolgoročne posledice; ali pod okriljem Instituta Jožef Stefana v oktobru 2016 začeli s bolnik motnjo ritma dobro prenaša (kakšni so simptomi in pilotnim sistemom za presejanje bolnikov s sumom na nereden vrednost krvnega tlaka); ali je potrebno takojšnje ukrepanje; ali srčni utrip. Z raziskavo smo želeli dobiti vpogled v praktično je potrebna takojšnja hospitalizacija in ali je potreben posvet s uporabo elektrokardiografskega senzorja pri bolnikih, specialistom ter če ja, kdaj [3]. obiskovalcih ambulant družinske medicine, ki v anamnezi navajajo motnje srčnega ritma. Ta metoda lahko revolucionizira Palpitacije so eden najpogostejših problemov bolnikov, ki jih oskrbo bolnikov s srčno-žilnimi obolenji v domačih ambulantah, zdravniki na primarni ravni napotujemo h kardiologom; po nam privarčuje denar in premosti prepad med primarnim in podatkih študije, ki je zajemala 500 bolnikov, naj bi jih zaradi sekundarnim nivojem. palpitacij bilo pregledanih 16 % [4]. Vendar pa dejanske motnje ritma najdemo pri manj kot polovici [5]. Palpitacije so senzorni Ključne besede simptom, ki ga spremlja neprijeten občutek močnega, hitrega ali nepravilnega bitja srca; bolniki radi opisujejo, kot da jim razbija motnje ritma, EKG senzor, telekardiologija, napotovanje v prsih ali v vratu [6]. Diferencialna diagnoza palpitacij je obširna; v študiji, v katero je bilo vključenih 190 bolnikov, so 1. UVOD ugotovili vzroke pri 84%. Pri 43% bolnikov je bil vzrok Motnja srčnega ritma ali aritmija je vsak ritem, ki ni normalni kardialni, pri 31% psihogen in pri 10% mešani vzroki kot sinusni ritem z normalnim prevajanjem preko prevajalnega tireotoksikoza, uživanje kofeina ali kokaina, anemija in podobno sistema; normalni ritem se torej začne z impulzom v [7] . sinoatrialnem vozlu v zgornjem delu desnega atrija, srčni utrip je Da lahko postavimo diagnozo pri bolniku, ki navaja motnje v normalnem območju, torej od 60 do 100 utripov na minuto, P- ritma, v praksi uporabimo EKG. Po enkratnem posnetku, ki valovi so normalni na EKG-posnetku in ritem je stabilen. Motnje nujno ne odraža motnje ritma, ki je lahko prehodnega značaja, se ritma so pogosto prisotne pri bolnikih v ambulantah družinske poslužujemo bolj kontinuiranih metod. V široki uporabi pri medicine in tako so družinski zdravniki in tudi specialisti drugih specialistih kardiologih je holter monitoring, kjer gre za 24- ali kliničnih strok soočeni s problemom, kako obravnavati take 48- urno nepretrgano merjenje EKG-ja [8]. V izogib bolnike. nepotrebnemu napotovanju in zaradi dolgih čakalnih vrst lahko Po podatkih Zdravstvenega statističnega letopisa je bilo leta 2014 v ambulanti uporabimo osebne prenosne merilnike kot npr. event na primarnem zdravstvu opravljenih 1.229.195 preventivnih in loop monitor, zio patch ali implantabilni loop rekorder; gre za 7.525.179 kurativnih pregledov, kar znaša 596 preventivnih in manjše prenosne naprave, ki jih ima bolnik nameščene dlje časa 3.650 kurativnih obiskov na 1.000 prebivalcev. Tako se [9]. Slovenija z 6,5 obiskov pri zdravniku na prebivalca letno uvršča na sredino seznama evropskih držav. Gledano po razlogih za 2. RAZISKAVA obisk na primarni ravni predstavljajo bolezni srca in ožilja 5,73% vseh obiskov letno, to je 290.086 pregledov, in so na šestem Namen, cilji in metode mestu po pojavnosti [1] . V Zdravstvenem domu Ljubljana smo pod okriljem Instituta Med najbolj pogoste aritmije štejemo: supraventrikularne Jožefa Stefana v oktobru 2016 začeli s pilotnim sistemom za prezgodnje utripe, ventrikularne prezgodnje utripe, bradikardijo, presejanje bolnikov s sumom na nereden srčni utrip. Z raziskavo ventrikularno tahikardijo, atrijsko fibrilacijo in undulacijo, smo želeli dobiti vpogled v praktično uporabo EKG-senzorja supraventrikularno tahikardijo, avtriventrikularne bloke ali (Savvy, [10]) pri bolnikih, obiskovalcih ambulant družinske ventrikularno tahikardijo in fibrilacijo. Simptomi določajo, kako medicine, ki v anamnezi navajajo motnje srčnega ritma. Pri njih je treba k bolniku pristopiti in so lahko odvisni od aritmije same smo želeli ugotoviti uporabnost EKG-senzorja, določiti delež (npr. palpitacije) ali hemodinamskih posledic aritmije (npr. bolnikov, ki v anamnezi navajajo motnje ritma in so le-te dispneja, vroglavica). Lahko so podobni drugim medicinskim objektivno potrjene s pomočjo EKG-senzorja, ugotoviti delež stanjem in vključujejo palpitacije, vrtoglavico, sinkopo, bolečine posameznih tipov motenj ritma, ki jih registrira EKG-senzor, v prsih in vratu, dispnejo, slabost in anksioznost. Gre pa lahko ugotoviti skladnost anamnestičnih motenj počutja z dejanskimi tudi za posledico nekega srčnega obolenja, kot so kongestivno motnjami srčnega ritma, ki jih zabeleži EKG-senzor in ugotoviti srčno popuščanje, ishemija ali trombembolični zapleti [2]. način ukrepanja zdravnikov glede na izvid EKG-senzorja. Raziskava je prospektivna, tipa primer – kontrola. Vključeni so bili nekateri družinski zdravniki in njihovi bolniki, ki so navajali 409 motnje srčnega ritma in so ob pregledu imeli normalni izvid 12- 4. ZAKLJUČEK kanalnega EKG-ja. V pilotni raziskavi pacientov s sumom na nereden srčni utrip se Pacienti, ki so sodelovali v raziskavi, so bili izmenoma vključeni je pokazalo, da so pacienti, ki so uporabljali EKG-senzor, tega v eno od dveh skupin. Prva je bila testna skupina, kjer so bolniki dobro sprejeli, saj so ga celo v 64% nosili z veseljem. Zdravniki prejeli EKG-senzor in navodila za ravnanje z njim, druga skupina so bili zadovoljni v še večjem odstotku, saj so v 83% odgovorili, je bila kontrolna, katere bolniki so bili obravnavani brez EKG- da jim je metoda olajšala oziroma zelo olajšala obravnavo senzorja po ustaljeni poti. Vsi v raziskavo vključeni pacienti so bolnikov. Ker se podatki o kontrolni skupini, ki EKG-senzorja ni prejeli dnevnik opažanj, ki so ga pisali doma in se po 5 do 10 imela, še zbirajo, bodo rezultati direktne primerjave med dneh vrnili na kontrolni pregled. Odvisno od situacije je lahko skupinama znani po pridobitvi vseh podatkov. Obširneje o študiji posamezen bolnik ponovil EKG meritve in tako senzor dobil glej v [11]. večkrat, a največ trikrat. Sestavni del EKG-senzorja je bil tudi mobilni telefon, ki pa je bil 5. ZAHVALA v raziskavi uporabljen le kot snemalna naprava za EKG. V pripravah na pilotno študijo smo ugotovili, da bo prikaz EKG na Zahvaljujemo se vsem zdravnikom in njihovim bolnikom, ki so sodelovali v naši raziskavi ter marljivemu osebju SIM zaslonu telefona paciente verjetno preveč motil in smo zato ta -centra. prikaz pacientom onemogočili. Sam EKG Posebna zahvala gre Antoniji Poplas Susič, ki je bila gonilna sila -senzor Savvy sicer omogoča meritve do 7 v tem projektu; brez njene pomoči in spodbujanja na večih dni, telefon pa so bolniki polnili vsako noč, če je meritev trajala več dni. ravneh nam ne bi uspelo tako uspešno uresničiti zadanih ciljev. 3. REZULTATI 6. LITERATURA [1] Zdravstveni statistični letopis 2014. Zdravstveno varstvo na Od oktobra 2016 do januarja 2018 je bilo v raziskavo vključenih primarni ravni. Inštitut za varovanje zdravja RS Ljubljana 30 zdravnikov iz ZD Ljubljana in ZD Murska Sobota ter njihovih [citirano 2018 Jan 23]. 110 bolnikov, od katerih je imelo senzor nameščen 100 bolnikov DOI=http://www.nijz.si/sites/www.nijz.si/files/uploaded/publikaci (90.9%), podatki kontrolnih skupin so trenutno na voljo le pri 10 je/letopisi/2014/5.1_primarna_raven_2014.pdf. bolnikih. Podatki so bili zbrani od vključenih zdravnikov in [2] Levy, S. Olshansky, B. 2017. Arrhytmia management for primary njihovih bolnikov. care clinicians. Uptodate;. Topic 961, Version 11.0 [citirano 2018 Zdravnikov, ki so sodelovali, je bilo 30, od tega 23 žensk in 7 Jan 23]. DOI= https://www.uptodate.com/contents/arrhythmia- moških; 2 sta specializanta družinske medicine, 2 pediatra, ostalo management-for-the-primary-care-clinician. specialisti družinske medicine; povprečna starost je bila 49,18 [3] Shen, WK., Sheldon, R.S., Benditt, D.G., et al. 2017. Guideline let, povprečna delovna doba je 22,12 leti. Odgovarjali so na fort he Evaluation and Management of Patients With Synkope: A vprašanja o uporabnosti te metode: zadovoljni oziroma zelo report of the American College of Cardiology/ American Heart zadovoljni z jasnostjo odčitka so bili v 93,5%, z enostavnostjo Association Task Force on Clinical Practice Guidelines and the odčitka v 77,1%, s Heart Rythm Society. J AM Coll Cardiol. 70 (16). postavljeno diagnozo v 88,7%; 83,9% jih je odgovorilo, da jim je metoda olajšala oziroma zelo olajšala [4] Kroenke, K., Arrington, M.E., Mangelsdorf, A.D. 1990. The obravnavo in 83,9% zdravnikov je bilo zadovoljnih oziroma zelo prevalence of symptoms in medical outpatients and the adeguacy of therapy. Arch Intern Med 150:1685 zadovoljnih z uporabo te metode. Vključenih je bilo 110 bolnikov, od tega je do sedaj zbranih [5] Klein-Wiele, O., Faghih, M., Dreesen, S., et al. 2016. A novel cross-sector telemedical approach to detect arrhythmia in primary podatkov za 10 kontrolnih bolnikov, ostali podatki so v fazi care patients with palpitations using a patient-activated event pridobivanja. Bolniki, ki so bili vključeni v raziskavo, so imeli recorder. Cardiol J. 23 (4): 422–8 pri zdravniku normalen posnetek EKG-ja, vseeno pa je obstajal [6] Zimetbaum, P., Josephson, M.E. 1998. Evaluation of patients with sum na motnjo ritma, bodisi anamnestični (72,2%), klinični palpitations. N Engl J Med 338: 1369–73. (18,7%) ali oba (9,1%). Med bolniki je bilo 71,8% žensk, stari so bili od 18 do 61 let, v povprečju 49,1 let; 41,2% jih je zaposlenih, [7] Weber, B.E., Kapoor, W. 1996. Evaluatin and outcomes of patients with palpitations. Am J Med 100: 138 28,2% upokojenih, 21,2 % nezaposlenih; 71% jih je nekadilcev, za 58,7% jih je njihov zdravnik opredelil kot tvegane pivce [8] Barry, J., Campbell, S., Nabel, E.G. 1987. Ambulantory alkohola. Bolniki so kot najpogostejše vzroke za prih monitoring of the digitized electrocardiogram for detection and od k early arning of transient myocardial ischemia in angina pectoris. zdravniku navedli razbijanje srca (45,3%), nato preskakovanje Am J Cardiol 60: 483 srca (20,3%), ter tiščanje v prsih, bolečina v prsih, omotica in težko dihanje. Bili so tudi povprašani o tem, kako se jim je zdela [9] Joshi, A.K., Kowey, P.R., Prystowsky, E.N. 2005. First experience with Mobile Cardiac Outpatient Telemetry system for uporaba te metode: 69,7% se je zdela namestitev enostavna, the diagnosis and management of cardiac arrhytmia. Am J Cardiol 68,7% se je zdela uporaba enostavna; 53,5% jih nameščen senzor 95: 878 ni motil, 64,3% ga je z veseljem nosilo in 67,1% jih je ocenilo, da večjih težav s senzorjem niso imeli. [10] Savvy. Navodila za uporabo Personal Cardiac Activity monitoring system [citirano 2018 Jan 23]. DOI=http://www.savvy.si Po odčitku posnetka smo ugotovili, da je vzrok bolnikovim [11] Kocjančič, S. 2018. Elektrokardiografski senzor: uporaba med težavam v 39,3% v benignih motnjah ritma, kot prezgodnji utripi, družinskimi zdravniki. Med Razgl. 57 Suppl 2: 37–41 sinusna tahikardija, paroksizmi supraventrikularne tahikardije, v 1% je šlo za paroksizem atrijske fibrilacije, v 13,1 % je šlo za ventrikularne motnje ritma, v 30,3% pa je šlo za anksioznost in panično motnjo. Zdravnikovo naslednje ukrepanje bilo za 63,5% bolnikov nadaljnje opazovanje, 6,7% jih je bilo poslanih na dodatne diagnostične preiskave, 6,7% jih je dobilo novo medikamentozno terapijo; 18,3 % jih je bilo napotenih h kardiologu. 410 Pametno okolje za učinkovito ščetkanje zob Peter Kokol Matjaž Colnarič Stanislav Moravs Fakulteta za elektrotehniko, Fakulteta za elektrotehniko, Fakulteta za elektrotehniko, računalništvo in informatiko, računalništvo in informatiko, računalništvo in informatiko, Koroška Cesta 46, 2000 Koroška Cesta 46, 2000 Koroška Cesta 46, 2000 Maribor Maribor Maribor stanislav.moraus@um.si peter.kokol@um.si matjaz.colnaric@um.si Milan Zorman Grega Žlahtič Jernej Završnik Fakulteta za elektrotehniko, Milan Zorman Zdravstveni dom dr. Adolfa Drolca računalništvo in informatiko, Fakulteta za elektrotehniko, Maribor, Ulica Talcev 9, 2000 Koroška Cesta 46, 2000 računalništvo in informatiko, Maribor Maribor Koroška Cesta 46, 2000 jernej.zavrsnik@zdm-mb.si milan.zorman@um.si Maribor grega.zlahtic1@um.si Helena Blažun Vošner Marko Turčin Tadej Završnik Zdravstveni dom dr. Adolfa Zdravstveni dom dr. Adolfa Zdravstveni dom dr. Adolfa Drolca Drolca Maribor, Ulica Talcev 9, Drolca Maribor, Ulica Talcev 9, Maribor, Ulica Talcev 9, 2000 2000 Maribor 2000 Maribor Maribor helena.blazun@zdm- marko.turcin@um.si tadej.zavrsnik@student.um.si mb.si Simon Jurič Bojan Slemnik Jernej Detela Inova IT, Pesnica pri Mariboru Inova IT, Pesnica pri Mariboru Inova IT, Pesnica pri Mariboru 32e, 2211 Pesnica pri Mariboru 32e, 2211 Pesnica pri Mariboru 32e, 2211 Pesnica pri Mariboru simon.juric@inova.si bojan.slemnik@inova.si jernej.detela@inova.si Ključne besede 2.PAMETNO DRŽALO ZA ŠČETKE pametno zobozdravstvo, pametni ročaj za ščetko, resna igra Prototip pametnega držala ščetke (slika 1), omogoča merjenje položaja, pospeškov in pritiska ščetke, s čimer je omogočeno 1.UVOD merjenje dodatnih parametrov pravilnega čiščenja zob v Področje pametnih zobnih ščetk je v zadnjem letu z vidika primerjavi z že obstoječimi rešitvami. Druga poglavitna razlika tehnologij doživelo napredek [1]. Če je še v letu 2016 družina med naštetimi pametnimi ščetkami in novo rešitvijo je, da ščetk Oral-B SmartSeries, kot ena najpametnejših, ob povezavi imajo vse naštete komercialno dostopne ščetke glavo na s pametnim telefonom omogočala spremljanje časa, lokacije in električni pogon, medtem ko rešitev z držalom omogoča pritiska zobne ščetke v ustih, je v začetku 2017 zobna ščetka uporabo običajne ščetke in s tem povečuje fleksibilnost uporabe Kolibree Ara zajem podatkov o čiščenju zob preselila iz in niža ceno uporabe. pametnega telefona na zobno ščetko, kar je koristno v situacijah, ko pametni telefon ni v dosegu povezave Bluetooth. Namen pametnega držala za zobne ščetke je zagotoviti Podatke zajemajo preko 3D senzorjev, obdelujejo pa jih s instrument za preverjanje pravilnosti čiščenja zob pri otrocih pomočjo metod umetne inteligence. Še korak dalje so naredili ter učenje le-tega. V ta namen je držalo ščetke opremljeno s snovalci ščetke Onvi Prophix, ki za 400USD (skoraj 2-kratnik senzorji, ki naj omogočajo detekcijo gibanja (pospeškometri) in do sedaj najdražjih pametnih ščetk) ponujajo pametno zobno merjenje pritiska ščetke na zobe. Držalo med čiščenjem javlja ščetko s štirimi različnimi nastavki in kamero HD, katera preko podatke na mobilno napravo ali osebni računalnik, od tod pa v aplikacije na pametnem telefonu snema čiščenje zob. Z vidika oblak, kjer se ti podatki analizirajo in se pridobijo vse primerjave s tukaj razvito rešitvijo je zanimiva pametna ščetka relevantne informacije. Za dodatni nadzor pravilnega čiščenja Grush. Intelov modul Curie skrbi za zajem podatkov te pametne zob pri kliničnih preizkusih, je sistem nadgrajen s sistemom ščetke, katera s pomočjo pametnega telefona spremeni čiščenje spremljanja položaja uporabnika na osnovi naprave Kinect. zob v igro za otroke. Podatki obeh sistemov se združujejo (fuzija podatkov) in omogočajo analizo podatkov na bistveno višji ravni. V okviru projekta EkoSmart za razliko od zgoraj navedenih pristopov razvijamo pametno okolje za učinkovito ščetkanje in Prototip je bil zasnovan na osnovi že izdelanih elektronskih vzdrževanje zobne higiene, ki sestoji iz pametnega držala, resne modulov povezanih v celoto znotraj ohišja in vsebuje: igre in Kinect aplikacije za spremljanje položaja ščetke.  absolutno orientacijo držala v prostoru (3 osni pospeškometer, absolutno orientacijski senzor v prostoru), 411  modul za pretvorbo signala iz merilnih lističev (LOAD CELL),  Arduino UNO,  bluetooth modul BlueFruit BLE Shield,  signalne lučke in vibracijski aktuator,  akumulator in indukcijski napajalnik,  tipka za vklop/izklop. Slika 1. Pametno držalo za zobne ščetke – strukturni prikaz Delovanje: Mehanski deli, pospeškometer in pretvornik signalov iz merilnih lističev so montirani v preprostem ohišju. Senzorske signale ob uporabi ščetke (pospeški in sila) sprejema Arduino in jih preko protokola Bluetooth BLE (Low Energy) prenaša na mobilne naprave. Svetlobni indikatorji in vibracijski Slika 2. Prikaz delovanja Kinect aplikacije aktuator se uporabljajo za signalizacijo neustrezne uporabe ščetke, signalizacijo vklopa/izklopa in kot indikator V času umivanja zob zbrani podatki, ki jih senzorji držala zobne napolnjenosti akumulatorja. Držalo je opremljeno tudi z ščetke sicer pridobivajo in hranijo povsem avtonomno (tudi v času, ko/če ščetka ni v ustih), so zmeraj ustrezno validirani in indukcijskim polnilcem, ki se polni preko stojala za ščetko. za kasnejšo obdelavo zapisani le za čas, ko je ščetkanje zob tudi Ohišje prototipa je bilo izdelano z metodo 3D tiska. dejansko potekalo. Na ta način zagotavljamo verodostojnost izmerjenih podatkov, česar sama pametna ščetka ne zmore. Držalo podatke senzorjev pretvori v podatke o položaju ščetke v prostoru (Eulerjevi koti). Za določitev začetnega (nultega) Potrebna oddaljenost obraza osebe, ki ščetkanje izvaja, glede položaja ščetke, je predvidena uporaba Kinect naprave v na mesto postavitve Kinect, je med 1,2 in 2,5 m, medtem ko je kliničnih eksperimentih in tipka v primeru neodvisnega priporočljiva višina postavitve med 1,0 in 1,8 m. delovanja držala. Zaradi preproste prenosljivosti in dobre podprtosti s strani Ker gre za prototip za preverjanje koncepta, bo miniaturizacija algoritmov za kasnejše obdelave, izmerjene podatke hranimo v potekala v naslednjih fazah. Namesto ločenih vezij se bo formatu JavaScript Object Notation (JSON). uporabilo eno, na novo zasnovano vezje, ki bi primerno tudi za V skladu z določili aktualne uredbe o varstvu osebnih podatkov masovno proizvodnjo. Narejena je bila tudi prva študija stroškov za izdelavo držala na osnovi brizgane plastike. (GDPR) smo preverjanje identitet oseb (mladostnikov) za katere bo t. i. Longitudinalni management statusa zobovja S prototipom smo dosegli potrditev, da je mogoče z razmeroma izvajan, omejili izključno na nivo številk ZZZS. Demografski cenenimi senzorji zajeti dovolj relevantne podatke, iz katerih je podatki istih oseb bodo pooblaščenim zdravstvenim delavcem mogoče ugotoviti frekvenco ščetkanja ter smer gibanja in silo, tako dostopni le v povezavi z drugimi zdravstvenimi sistemi na s katero ščetka pritiska na zobe in dlesni. isti Id osnovi. V okviru projekta EkoSMART smo pripravljeno rešitev tudi 4.PAMETNA RESNA IGRA patentirali. S pomočjo zunanje patentne pisarne je bila oddana V prvotni fazi razvoja igre smo naredili pregled orodij s prijava patenta, patent pa je že v fazi objave. katerimi bi jo lahko razvili [2,3]. Tako smo po pregledu V nadaljevanju projekta bo izdelanih deset prototipov držala, ugotovili, da je trenutno eden izmed bolj priljubljenih in prav ki bodo uporabljeni za klinične teste čiščenja zob z otroki. tako fleksibilnih in stabilnih orodji Unity. Orodje najbolj prepriča z dobro dokumentacijo in neverjetno velikim naborom 3.KINECT APLIKACIJA ličnih iger vseh možnih variant. Prav tako se vse v orodju Unity Prototip opreme (slika 2) spremljanje položaja ščetke dela kot objekt kar omogoča hitre spremembe, ki so tako predstavlja naprava Microsoft Kinect for Windows v2 (Kinect), uporabne v sprotnem razvoju, kot v morebitnih nadgradnjah ali z osebnim računalnikom (Microsoft Windows), z vrati USB spremembah delovanja in izgleda iger. (3.0), integrirano tehnologijo Bluetooth Low Energy (BLE ali Pri izbiri tipa iger smo se odločili za tako imenovan »Tower Bluetooth 4.0) in odjemalsko aplikacijo vrste WPF App (.NET defense«, saj se lahko smisel igre lepo preslika na temo ustne Framework). Slednja skrbi za potrebno sprotno identifikacijo in higiene ali bolje rečeno »Ščitenje zob«. V takšnem tipu igre vizualizacijo pravilnosti položaja dlani desne ali leve roke, ki mora igralec namreč trdnjavo braniti pred številnimi skupinami drži ščetko v ustni votlini, vključno z njeno dopustno napadalcev s pomočjo postavljanja stolpov. Pri tem je (predhodno nastavljivo) oddaljenostjo od obraza osebe, ki pomembno taktično postavljanje stolpov, saj so surovine meritev izvaja. 412 omejene in jih je potrebno čim bolj učinkovito porabiti. V našem primeru bo seveda trdnjavo predstavljal zob/zobje, napadalci bodo bakterije, stolpi pa pripomočki za ustno higieno (Slika 3). V igri bo tudi, primerno za otroke, zamenjano uničevanje (ki se lahko interpretira kot ubijanje) bakterij s pretvorbo škodljivih bakterij v pozitivne bakterije, ki se lahko uporabijo za branjenje zob. Prav tako bo s sličicami predstavljeno kako vpliva tip zaužite hrane na število škodljivih bakterij, ki bodo napadale naše zobovje. Za samo motivacijo in omejitev igranja igre bo s pomočjo podatkov pridobljenih s strani pametne ščetke, odvisno od kakovosti umivanja zob, otrok prejel več življenj katera potrebuje za nadaljevanje igranja v kolikor mu jih zmanjka. V primeru zelo dobrega rezultata umivanja zob, dobi otrok dodatno nagrado, ki mu omogoči uporabo predmeta, ki ga v sami igri drugače ne more dobiti. Ta nagrada je predstavljena z jabolkom, ki zmanjša število škodljivih bakterij v naslednjem napadu. Razvili smo tudi model nagrajevanja s točkami na podlagi trajnosti in kakovosti oralne higiene za spletno varianto igre in primerjavo z drugimi igralci, kar bo povečalo motivacijo otrok za ščetkanje zob. Slika 3. Prikaz resne pametne igre LITERATURA [1]. Dowson T. 7 Dental Industry Trends in 2017 & What They Mean For Practice Growth [Internet]. [cited 2017 May 13]. Available from: https://titanwebagency.com/blog/dental- industrytrends/ [2]. LudoScience - Classifying Serious Games: The G/P/S Model (Broacasting our studies) [Internet]. [cited 2017 May 9].Available from: http://www.ludoscience.com/EN/diffusion/537-Classifying- Serious-Games-The-GPS-Model.html [3]. Baranowski T, Blumberg F, Buday R, DeSmet A, Fiellin LE, Green CS, et al. Games for Health for Children—Current Status and Needed Research. Games Health J. 2015;5:1–12. 413 Time Series or Relational Database for Edge and IoT Anže Luzar Sašo Stanovnik Matija Cankar XLAB Research XLAB Research XLAB Research XLAB d.o.o. XLAB d.o.o. XLAB d.o.o. Pot za Brdom 100 Pot za Brdom 100 Pot za Brdom 100 1000 Ljubljana, Slovenia 1000 Ljubljana, Slovenia 1000 Ljubljana, Slovenia anze.luzar@xlab.si saso.stanovnik@xlab.si matija.cankar@xlab.si ABSTRACT This paper explores the efficiency of relational and time se- In Fog and Edge computing data management and process- ries databases on edge devices by measuring and comparing ing is moving from the Cloud closer to the IoT devices. response times and memory footprints of two representa- To perform the work in edge devices, different, more light- tives. From the results a reader can conclude which type of weight, small-footprint and specialized tools need to be em- the database is better for a specific edge device or fog-like ployed. In this paper we perform a side-by-side comparison environment. of relational and time series databases of their speed and re- source consumption. The Results show better performance The research of the database performance will be further of time series over relational databases. used in a fog-to-cloud application called Smartboat, which is developed as a use case for EU H2020 funded project called Keywords mF2C[5]. The application’s goal is to establish support for boats that would simplify sailing and detect different types database, SQL, timeseries, relational, cloud, fog, IoT, edge of threats across the sea. The IoT sensors that are installed onto the boats are used to collect certain amount of data, for 1. INTRODUCTION example they can retrieve the temperature, GPS position, Nowadays plenty of databases are available for storing actual pressure, humidity, they can detect whether doors are open timestamped data to a database. In the past several years, or not, generate flood alarms and so on. Based on that data there are growing appeals for reading and storing data from it is important to take different actions. And since it’s im- IoT devices [8]. In the close future, storing and manipulat- portant when to take these actions, a database that supports ing time series data will play an important role in IoT [1]. storing and aggregating the data annotated with timestamps A common technique is to gather the data using cloud or is required. Different databases that fulfill the requirements fog devices that read IoT sensors [4] and temporarily store for the project were reviewed in order to select the best one it in relational or time series databases. As not all sensor for the use case considering this article. Then a comparison readouts are required to be stored in the Cloud, it is useful between databases and testing of parameters, most impor- to filter and process the sensors on the Edge near them and tantly time efficiency and memory footprint, were made. thus save Cloud resources and bandwidth. To achieve this, one of the most important and desired abilities for Edge de- The paper continues as follows: Section 2 presents the prob- vices is to handle volumes of time series data quickly with lem, its background and the key parameters to evaluate the minimum latency and footprint in order to give the observer database. The experiments and results are presented in Sec- results as quickly as possible [11], [9]. IoT devices can rely tion 3 followed by the discussion and conclusion in the last on different database types behind them and the best choice section. mostly depends on the type and format of the data that is being stored and on the requirements of the edge device 2. DATA STORAGE IN EDGE DEVICES [6]. Among available database types that can be used for Storing data always requires time and has a memory foot- IoT devices are NoSQL (e.g. MongoDB) with its subtype print – that means some CPU and memory usage. On top time series database (e.g. InfluxDB, Prometheus, Time- of that there are many performance problems that can arise caleDB) and relational database (e.g. PostgreSQL, MySQL, due to several reasons that are occasionally hard to deter- MSSQL). Global trends unveil that time series databases are mine. The following section provides additional information currently the fastest growing database type [7]. for understanding the problem of storing the data and the problem itself. 2.1 Sensors, edge devices and cloud storage In a combined fog and cloud environment the processing of data is distributed between edge devices and cloud. Process- ing in the cloud has no resource restrictions such as opposed to processing at the edge. To provide the best and to the user transparent experience of using cloud and edge envi- ronment, the appropriate software has to be applied to each 414 segment of the fog to cloud hierarchy. A similar stack is Supported languages More supported languages are a plus, presented in Figure 1, which is similar to the one proposed but our main focus was on Java and Python support. by mF2C project. We focus on data management close to sensors, i.e. edge devices, such as routers or small computers Data types The support or special/faster handling of floats like Raspberry Pi devices that store, filter and transmit data and timestamps was considered as a better option. collected from IoT sensors. Beside being able to store and License An open-source solution is preferred due to better transmit data, edge devices can serve light-weight services flexibility and potential costs if the databases would and issue notifications based on thresholds. These devices run on a large amount of edge devices. are capable in variety of functions, but do not have an ex- cess of resources, therefore the software needs to be selected carefully. All databases, relational and time series, were evaluated by those parameters and the best candidates of each type were selected for the testing phase. The attributes for databases are collected in the Table 1. 2.3.2 Relational database selection process We chose PostgreSQL as the initial relational database for our endeavors because of its standards compliance, it of- fering a native JSON object storage which we aspired to use elsewhere in the application and because it was eas- ily integrated into other frameworks already in use. Other databases may also be appropriate for this purpose, how- ever PostgreSQL proved to be the most compatible choice Figure 1: The proposed mF2C architecture. at that point in development. The key parameters of our comparison, based on two comparative sources [12, 2] are 2.2 The performance degradation issue shown in Table 1. From the experience we gained by working on Fog, Edge and IoT use-cases, we found that the relational database, 2.3.3 Time series database selection process PostgreSQL, can became unstable and an overkill due to These days, time series data applications such as sensors the lack of resources. The issue occurs when data is con- used in IoT analytics, are growing rapidly due to their sim- tinuously being written into the database for a long period plicity and SQL based query language. For the comparison of time. Performance degradation seems to be a common we have chosen 8 time series databases (Table 1) and finally problem in PostgreSQL [10] and sometimes hard or even selected InfluxDB as the best candidate mainly because of impossible to solve [3]. For shorter continuous periods of official Raspberry Pi Docker support. Other databases were recording (e.g. single day) issue does not manifest due to not selected because they did not fulfill expectations regard- the small amount of the data and also because the database ing Docker or Raspberry Pi support (OpenTSDB, TimescaleDB), requires a reasonable amount of RAM. When recording lasts a proprietary license (Kdb+), low data type flexibility (Prometheus, longer (e.g. more than one day) writing becomes slower. To RRDtool) and a lack of Java support (Graphite, Druid). solve the issue different approaches of saving and different databases were taken into consideration. 3. EXPERIMENT AND RESULTS 2.3 Benefits of using relational or time series 3.1 The test between time series and relational databases and their comparison databases 2.3.1 Criteria for filtering the databases The performance was evaluated by integration of InfluxDB into our application and comparing it with the performance Before performing the evaluation of the databases, a selec- of PostgreSQL. tion of the testing candidates was required, one from each type of database. Our methodology preferred databases with better support for the following attributes: 3.2 Measurement environments The databases could be manipulated through their own ter- minal clients or by libraries that provide support for differ- Supported platforms Applications should run on all ma- ent programming languages. To eliminate the probability of jor platforms like Linux, Windows and macOS, there- poorly written library or additional latencies based on the fore we required to be sure that there will not be any language overhead, the tests were performed in both envi- complication for the applications to use the database. ronments – through a Java program and through the ter- minal with official client. According to the presented lim- Official Docker support Docker, currently the most pop- itations, the following tests were performed: reading and ular container technology for Linux that allows cre- writing to the InfluxDB and PostgreSQL databases using ating and packaging an application along with all its different methods like Java, bash console, reading from file dependencies, was also very important for the imple- and so on. So we tested the database and created a ta- mentation of the services in our project. ble (Table 2) showing first stage results for measuring time Rapsberry Pi Our project was focused and prototyped taken for writing and reading. All times presented in the around the Raspberry Pi, therefore the options that table are for writing ten million records to database or for include this were preferred. retrieving one million lines from the base. 415 Table 1: Relational and time series database feature comparison table time with some deviations that are occurring periodically. Table 2: Time taken table with first-stage results. Most of the data in InfluxDB (Figure 2) is stored to database very quickly and it takes between 0–1 milliseconds. There are also some deviations of records that take around 50 mil- liseconds to be stored to database. The curve that shows the average time taken to write to database is also more diverse than in PostgreSQL plot. To perform thorough testing an InfluxDB faster batching was enabled to be included into the evaluation. The threshold for storing was set to 10000 points per batch or every 200 milliseconds. The result was lower times for writing and faster program execution. However, comparing InfluxDB results with and without faster batch- ing (Figure 2) showed similar performance. The results un- doubtedly show advantage of InfluxDB over the PostreSQL 3.3 IoT characteristics for databases focusing on time consumption. PostgreSQL has less variance IoT devices, especially sensors, usually have the ability of in time taken for transaction, but nevertheless InfluxDB is gathering the data accompanied by data analysis to detect better in the average case. anomalies. Those devices write to database in bursts and are often operating on a lot of data. Table 3: Time for writing million lines to database Database Time taken [s] 3.4 The experiment metrics InfluxDB 58.92580 The metric chosen for performing the experiment is speed, InfluxDBFasterBatching 5.25127 measuring the time for writing a million rows to the database PostgreSQL 300.48325 in chunks of 15 points, which appears to be a common re- quest on an Edge device connected around a dozen IoT de- vices. 3.5.1 Database setup times and resource consump- tion 3.5 The results Beside runtime performance, setup time and resource con- The results in Figure 2 present a comparison between In- sumption were measured. The database setup times, includ- fluxDB and PostgreSQL. The x-axis shows the consecutive ing connecting to the database and building tables and in- block number of 15 records and the y-axis shows time re- dexes is significantly faster with InfluxDB, which is evident quired to save the block. While the results are dispersed in Table 3. The resource consumption comparison was per- a trend curve showing the expected time taken is added to formed by writing and querying data for PostgreSQL and In- the graph. The plots present three cases. From the Post- fluxDB via CLI (Command Line Interface). 100000 records greSQL plot (Figure 2) below we can see that most of the were written to database using CLI and then retrieved back. blocks of fifteen records take 5–8 milliseconds to be stored The results of this test are presented in Table 4, where it is in database, which means that time is constant most of the evident that InfluxDB uses less storage, prepares database 416 Figure 2: Plots for InfluxDB and PostgreSQL and InfluxDB faster batching. faster, a query takes only 30 % of time. It seems that RAM postgres-performance-mystery-51544ceea584, Apr and CPU consumption is higher for InfluxDB. 2018. Accessed on 2018-09-06. [4] J. W. Flory. How time-series databases help make sense of sensors. https://opensource.com/article/ Table 4: Different test methods performed in CLI 17/8/influxdb-time-series-database-stack, Aug InfluxDB PostgreSQL 2017. Accessed on 2018-09-03. Writing time 0.694 s 223.374 s [5] Horizon2020. mf2c project. Query time 1.492 s 5.617 s http://www.mf2c-project.eu/, Jan 2017. Accessed on Database size 5.5 MB 15 MB 2018-09-05. Memory usage 63.98 MB 20.53 MB [6] R. Kumar. 4 steps to select the right database for your CPU usage 107 % 98.3 % internet of things system. https: //thenewstack.io/4-steps-to-select-the-right- database-for-your-internet-of-things-system/, 4. CONCLUSION Apr 2018. Accessed on 2018-09-12. This paper presented the approach towards comparing rela- [7] M. Risse. The new rise of time-series databases. tional and time series databases including comparing charac- https://www.smartindustry.com/blog/smart- teristics and performance. We explored several time series industry-connect/the-new-rise-of-time-series- databases and their use related to the Cloud, sensors and databases/, Feb 2018. Accessed on 2018-09-05. IoT. The results indicates why time series database can be [8] D. G. Simmons. Pushing iot data gathering, analysis, a better solution when it comes to storing IoT-generated and response to the edge. data. We concluded that InfluxDB is a more suitable op- https://dzone.com/articles/pushing-iot-data- tion for handling data gathered from IoT sensors and is also gathering-analysis-and-response-to-the-edge, significantly faster in comparison to a relational database. Apr 2018. Accessed on 2018-09-02. [9] E. Siow, T. Tiropanis, and W. H. Xin Wang. 5. ACKNOWLEDGMENTS Tritandb: Time-series rapid internet of things This project has received funding from the European Union’s analytics. https://arxiv.org/abs/1801.07947v1, Jan Horizon 2020 research and innovation programme under grant 2018. Accessed on 2018-09-13. agreement No. 730929. [10] A. Vorobiev. Performance degradation of inserts when database size grows. 6. REFERENCES https://www.postgresql.org/message-id/BANLkTi% [1] A. Bridgwater. Iot time series data is ‘of the hour’. 3DVKBmRLVLDjy8qxpWx_6-rmbUaXg%40mail.gmail.com, https://internetofbusiness.com/iot-time-series- May 2011. Accessed on 2018-09-04. data-hour-influxdata/, Dec 2017. Accessed on [11] D. G. Waddington and C. Lin. A fast lightweight 2018-09-03. time-series store for iot data. [2] DB-Engines. Db-engines ranking of relational dbms. https://arxiv.org/abs/1605.01435, May 2016. https: Accessed on 2018-09-12. //db-engines.com/en/ranking/relational+dbms, Sep [12] Wikipedia. Comparison of relational database 2018. Accessed on 2018-09-05. management systems. [3] J. DiLallo. Solving a postgres performance mystery. https://en.wikipedia.org/wiki/Comparison_of_ https: relational_database_management_systems, Sep //medium.com/flatiron-engineering/solving-a- 2018. Accessed on 2018-09-05. 417 Spletni obrazec in REST API za naročanje receptov Denis Pavliha Aleš Kalan Nataša Planinc SRC d.o.o. SRC Infonet d.o.o. SRC Infonet d.o.o. Tržaška 116 Cesta talcev 39 Cesta talcev 39 1000 Ljubljana 4000 Kranj 4000 Kranj +386 1 6007000 +386 4 6007600 +386 4 6007600 denis.pavliha@src.si ales.kalan@infonet.si natasa.planinc@infonet.si POVZETEK ambulanti in pacientu, potrditve in zavrnitve naročil, preklici, Razvili smo rešitev za elektronsko naročanje receptov za stalne preusmeritve v drugo ambulanto,…). Izdelali smo ogrodje za posredovanje sporočil (routing) prek treh kanalov: terapije. Zasnovana je kot Representational State Transfer elektronske pošte ( (REST) Application Programming Interface (API) vmesnik, e-mail), platforme za avtomatizacijo marketinških opravil pred katerim se nahaja grafični uporabniški vmesnik ( angl. (Mautic [1]) in prek REST API klica (za npr. klic vmesnika Graphical User Interface – GUI) v obliki preproste spletne platforme Think!EHR [2]). Sistem smo opremili s aplikacije za naročanje zdravil. Za komunikacijo (tj. pošiljanje sinhronizatorjem podatkov iz centralne baze zdravil (CBZ) ter naročil) smo izdelali ogrodje za posredovanje sporočil (routing) implementirali predpomnjenje (caching) seznama. prek treh kanalov: elektronske pošte (e-mail), platforme za avtomatizacijo marketinških opravil (Mautic) in prek REST Tabela 1. Nabor klicev REST API vmesnika in njihove API klica (za npr. klic vmesnika platforme Think!EHR). metode Ključne besede API klic Metoda spletno naročanje, elektronski recept, REST API vmesnik order GET order/new POST 1. UVOD order/status GET Naročanje receptov pri splošnem zdravniku je vsaj za stalne order/last GET terapije, tj. ponavljajoče se že predpisane recepte, zgolj patient GET administrativno opravilo. Pacient mora telefonirati v ambulanto, kar ponavadi traja dolgo zaradi zasedenih telefonskih zvez, nato patient/confirm GET pa zgolj posredovati potrebo po podaljšanju recepta, ki ga prek patient/register POST sistema e-Recept izvede zdravnik na osnovi že postavljenih diagnoz in prejšnjih izdaj zdravil. Gre za postopek, ki jemlje service GET dosti časa vsem udeleženim, zaradi preprostosti pa bi ga lahko service/drugs GET poenostavili z uporabo digitalnih rešitev. ward GET 2. REŠITEV ward/search GET Če želimo postopek digitalizirati, lahko to storimo tako, da del ward/suggest POST procesa (tj. sporočanje prek telefona ali osebno) pretvorimo v ward/register POST izpolnjevanje spletnega obrazca, ki ga nato zdravnik prejme v svoj zdravstveni informacijski sistem. Zato smo pričeli z ward/login POST zasnovo ogrodja za spletno naročanje receptov stalnih terapij. GET Najprej smo definirali podatkovni model, ki bo implementiran v ward/login/forgotten obliki podatkovne baze za hrambo naročil receptov. Nato smo POST rešitev razdelili na zaledni del ( angl. backend), ki je navzven ward/edit POST dostopen prek Representational State Transfer (REST) ward/edit/status POST Application Programming Interface (API) vmesnika, in grafični uporabniški vmesnik ( angl. Graphical User Interface – GUI), ward/edit/forwarding POST ki omogoča pacientom uporabo rešitve v obliki spletne DELETE aplikacije. ward/edit/forwarding/apply POST cron GET 3. REST API VMESNIK ward/confirm POST Zaledni sistem ( angl. backend) smo zasnovali kot ogrodje (framework), ki izpostavlja REST API vmesnike za izvedbo ward/authorize POST vseh podprocesov, povezanih s procesom spletnega naročanja. ward/disable POST Pripravili smo vse potrebne posamezne REST API vmesnike (Tabela 1) ter ustrezne razporejevalnike ( cronjob-driven 4. SPLETNI OBRAZEC (GUI) scheduler) opravil (naročila, pacienti, predlogi novih ambulant, Nazadnje smo razvili še spletni grafični uporabniški vmesnik stanja ambulant). Izdelali smo HTML predloge in vsebinska (GUI), ki je bil implementiran v HTML/JS/CSS tehnologiji z uporabo AngularJS ogrodja [3] (Slika 1). Izdelali smo sporočila, ki se v okviru e-mail sporočil posredujejo ob posamezne podsklope GUI (uporabniški vmesnik, registracija, različnih dogodkih v procesu (oddaja naročila, obvestila potrditveni pogled, administracijski vmesnik) in ustrezne 418 komunikacijske vmesnike za povezovanje z vsemi razvitimi REST API vmesniki posameznih podsklopov (naročila, pacienti, storitve, ambulante, predlogi, zdravila). GUI smo optimizirali tudi za mobilni prikaz (responsiveness). 5. PREIZKUŠANJE Vse razvite REST API vmesnike iz nabora smo preizkusili na način funkcionalnega testiranja: definirali smo nabor pozitivnih in negativnih vhodnih podatkov ter preizkušali odzive posameznih API vmesnikov nanje. S preizkušanjem smo potrdili pravilno delovanje razvite spletne rešitve za naročanje receptov. 6. NADALJNJE DELO Za potrebe povezovanja rešitve v platformo EkoSmart bo potrebno dodatno preizkušanje komunikacijskega vmesnika za pošiljanje sporočil prek REST API klica v platformo Think!EHR [2]. REFERENCE [1] Mautic, Open-Source Marketing Automation, www.mautic.org [2] Think!EHR, Marand, www.marand.com/thinkehr [3] Angular, Google, www.angular.io Slika 1. Grafični uporabniški vmesnik za naročanje receptov. 419 Postopki in priporočila za izgradnjo govorne zbirke za potrebe sinteze slovenskega govora Tomaž Šef Institut “Jožef Stefan” Jamova cesta 39 1000 Ljubljana +386 1 477 34 19 tomaz.sef@ijs.si POVZETEK 2. ANALIZA IZKUŠENJ IN PRAKS Govorna zbirka, uporabljena v sistemu eBralec [4], je bila v osnovi V članku predstavljamo raziskave in razvoj postopkov izgradnje zasnovana za korpusno sintezo govora. Vsebina besedila za govorne zbirke za potrebe sinteze slovenskega govora z vidika snemanje je bila pridobljena s statistično obdelavo vseh besednih pridobivanja govornih posnetkov in določanja optimalnih korpusov, s katerimi smo razpolagali, brez kakšne posebne snemalnih pogojev. vnaprejšnje selekcije. Določila se je pogostost posameznih glasov Za potrebe sinteze govora predlagamo govorno zbirko z branim in glasovnih nizov v besedilu. V zbirko so bili vključeni vsi stavki govorom. Na podlagi analize izkušenj in praks pri gradnji obstoječe (povedni, velelni, vprašalni ipd.), in sicer glede na statistično zbirke, uporabljene v sistemu eBralec, podajamo priporočila za zastopanost v besednem korpusu. razvoj nove zbirke s primarno podporo HMM-sintezi in sodobnejši Pri izbiri povedi se je težilo k temu, da so bile le-te fonetično čim WaveNet-sintezi z možnostjo preproste prilagoditve na korpusno bolj bogate, da so zagotavljale vse možne difone in da je vsaka sintezo. dodatna poved doprinesla čim več novih polifonov (trifonov, Ključne besede štirifonov ipd.). Dolžine izbranih stavkov naj bi čim bolj ustrezale statistični porazdelitvi dolžin stavkov iz besednega korpusa. Govorna zbirka, govorni bralnik besedil, sinteza slovenskega govora. Osnovni statistični podatki govorne zbirke eBralca so podani v tabeli 1. 1. UVOD Velikost besednega korpusa 7.145.345 povedi Govorne zbirke vsebujejo računalniško berljive posnetke govora in 77 milijonov besed priložene podatke z opisom posnetega govora (govorni dejavniki, Obseg govorne zbirke 4.000 povedi dejavniki govorcev, zapisi in označitve posnetega govora) [1, 2]. (46.785 besed) Takšne zbirke so nepogrešljive pri raziskovalno-razvojnem delu na Število različnih difonov 1.883 področju govornih tehnologij in predstavljajo pomemben člen Število različnih trifonov 21.369 osnovne infrastrukture za razvoj govornih tehnologij za posamezno (št. kombinacij v korpusu) (24.702) jezikovno področje [3]. Tabela 1: Statistični podatki o govorni zbirki Razlikujemo med dvema vrstama govornih zbirk: sistema eBralec.  brani govor: med drugim vključuje izvlečke iz knjig, radijske novice, sezname besed in zaporedja števk …  Snemanje govorne zbirke je potekalo v studiu RTV Slovenija ob spontani govor: vključuje dialog (med dvema ali več osebami), prisotnosti izkušenega tonskega tehnika. Med 10 profesionalnimi pripovedovanje (npr. raznih zgodb), opise (npr. poti na govorci smo izbrali najustreznejši moški in ženski glas. Med zemljevidu), dogovarjanje (npr. glede termina sestanka) ipd. branjem besedila so govorci imeli nameščene elektrode Za potrebe sinteze govora predlagamo govorno zbirko z branim Laryngographa, s katerimi smo spremljali nihanje glasilk za lažje govorom. Takšen govor ustreza najpogostejšim oblikam rabe kasnejše označevanje period govornega signala. sintetizatorjev govora. Poleg tega je lažje izdelati transkripcijo Samo snemanje je zaradi obsežnosti besedila, ki ga je bilo potrebno takšnega govora. Snemanje je bolj nadzorovano in predvidljivo. Pri prebrati, trajalo več mesecev. Pri tem smo skušali zagotoviti, da so spontanem govoru je govorno zbirko težko fonetično in prozodično uravnotežiti. nastavitve opreme ves čas ostale čim bolj nespremenjene. To ni bilo najlažje, saj je bila oprema v studiu že precej stara, studio je bil Najpomembnejši preostali dejavniki, ki jih je potrebno upoštevati konstantno v uporabi, konfiguracija prostora pa se je prilagajala pri snovanju govorne zbirke za potrebe sinteze govora so: izbira trenutnim potrebam RTV Slovenija. Pred vsakim snemanjem je vsebine posnetkov, izbira govorcev, snemanje govorne zbirke in govorec poslušal svoje predhodne posnetke, s čimer se je skušalo označevanje posnetkov. zagotoviti čim bolj enak način govora, z enako intonacijo ipd. Izbira velikosti govorne zbirke je posledica kompromisa med Zbirka sintetizatorja eBralec je bila označena na treh nivojih: želenim številom variacij glasov oz. njihovim pokritjem na eni grafemski zapis besedila, fonetični zapis prebranega govora z strani ter časom in stroški, vezanimi na razvoj, na drugi strani. mejami posameznih fonemov, oznake period zvenečih glasov. Upoštevati je treba tudi čas za kasnejše preiskovanje govorne zbirke in potreben prostor za njeno hranjenje. 420 Govorna zbirka oz. njen fonetični prepis se je popravljal s  poleg najpogostejših besed je treba »pokriti« tudi različne programom Transcriber (Slika 1). Govorci vseh besed niso prozodične kontekste, v katerih se te besede običajno izgovorili tako, kot smo pričakovali oz. tako, kot bi jih izgovoril pojavljajo, sam sintetizator govora. Zato je bilo potrebno fonetični zapis  čim bolj se skušamo izogibati besedam, ki niso vsebovane v uskladiti z dejansko prebranim. slovarjih izgovarjav, s katerimi razpolaga projektna skupina, saj je zanje potrebno ročno zagotoviti pravilen fonetični prepis,  v besedilo je priporočljivo vključiti pogoste leksikalne termine oz. pogoste besede, kot so denimo telefonske številke, ekonomsko terminologijo (bančništvo), različne valute, terminologijo s področja računalništva in interneta, medicine, pogosta lastna imena, nekatera tuja imena in izraze, glavne in vrstilne števnike, črkovanje, značilne jedi, turistične znamenitosti, lahko tudi države in njihova glavna mesta; pri tem stavke, ki pokrivajo te termine, kreiramo sami,  v zadnjem času je pomembno, da govorna zbirka pokriva še različne situacije, ki nastopajo v dialogu (aplikacije dialoga in simultanega prevajanja; npr. raba v virtualnih asistentih),  večji zastopanosti raznovrstnih povedi, predvsem vprašalnih in velelnih (pogostost teh povedi mora biti večja, kot je v samem besedilnem korpusu, iz katerega se zajema besedilo za branje),  besedilo naj zajema različne zvrsti novic, razne napovedi (npr. vremenske napovedi) in podajanje informacij (npr. stanje na cestah, borzne informacije) ter navodil (npr. napotki za vožnjo), Slika 1. Postopek popravljanja fonetičnih oznak v govorni  bolj podrobno označevanje govorne zbirke, ki naj vsebuje tudi zbirki eBralca. prozodijske oznake in razne dogodke, vezane na govorne organe, Največje izrazite identificirane pomanjkljivosti govorne zbirke  izbiri ustreznega ženskega glasu – ta naj bo nekoliko nižji in eBralca so naslednje: bolj aspiriran (povprečna osnovna frekvenca naj bo nižja kot  prevelika osredotočenost na čim večje število različnih pri aktualnem ženskem glasu), polifonov (npr. štirifonov, petfonov ipd.),  zagotavljanju enakih snemalnih pogojev med posameznimi  stavki omejenih dolžin znotraj vnaprej določenih intervalov ne sejami snemanja, pokrivajo dovolj vseh situacij, s katerimi se srečamo med  obseg govornega korpusa naj bo večji od obstoječega. dejansko sintezo,  premalo vprašalnih in velelnih povedi, 3. PRIPOROČILA ZA VSEBINO BRANIH  vsebina govorne zbirke ni bila povezana z najbolj pogostimi BESEDIL vsebinami besedil, ki se sintetizirajo v praksi oz. z najpogostejšimi aplikacijami rabe sintetizatorjev govora, Umetno generirani govor mora zveneti naravno in biti prijeten za  poslušanje. Pomembne so tudi nastavitve za hitrost branja in jakost govorna zbirka je brez posebnih prozodijskih oznak, označena zvoka ter možnost uporabe različnih glasov. sta dva nivoja (grafemski in prozodični),  sploh niso oz. pomanjkljivo so označeni razni dogodki, Izbor vsebine posnetkov oz. branih besedil govorne zbirke za povezani z govorilnimi organi, kot so tlesk, zapora, odpora, sintezo govora poteka v več korakih [5]: pripora, pridih,  ustvari se obsežna tekstovna zbirka besedil, ki pokriva različne  govorna zbirka ne vsebuje posebej posnetih števk, števil, zvrsti (dnevni časopisi, revije, leposlovje ipd.), števnikov, ideogramov, aktualnih slovenskih in pogostih tujih  tokenizacija – iz zbirke besedil se odstranijo vse oznake, vezane imen, priimkov, krajev, imen ulic ipd., prav tako ni podprto na oblikovno podobo (glava besedila, tabele ipd.), črkovanje (razumljivo črkovanje je v nekaterih aplikacijah in  okrajšave, števila ipd. se pretvorijo v polno besedno obliko za posebne skupine uporabnikov še posebnega pomena; npr. (normalizacija besedil), slepi in slabovidni uporabniki, uporaba v bralnikih zaslona),  besedila se pretvorijo v predvideni fonetični prepis (grafemsko-  predvsem ženski glas ni najbolj »posrečeno« izbran za potrebe fonemska pretvorba), sinteze slovenskega govora,  optimizira se obseg zbirke glede na vnaprej pripravljene  oddaljenost govorca od mikrofona ni bila najbolj ustrezna; kriterije (metoda požrešnega iskanja); doseči želimo statistično prevelik razpon med konicami signala in preostalim delom ustrezno vzorčenje izbranega področja govorjenega jezika. onemogoča večjo glasnost sintetiziranega govora,  Izbira povedi ne sme potekati naključno, pač pa mora biti skrbno pri snemanju ni bil pred mikrofon postavljen ustrezen filter, ki načrtovana [6]. Priporočamo, da se za čim optimalnejšo izbiro bi zadušil neželene zvoke; npr. razne tleske,  povedi uporabi naslednji postek, ki ga sestavlja več korakov: premajhna pozornost pri zagotavljanju enakih snemalnih 1. Statistična obdelava besedil: pogojev med posameznimi seansami snemanja.  Statistično obdelamo celoten besedni korpus in določimo Pri izgradnji nove govorne zbirke bo na podlagi preteklih izkušenj pogostost pojavljanja posameznih glasov in glasovnih treba več pozornosti nameniti: nizov v besedilu.  večji prozodični pestrosti posnetega besedila, ki bo pokrivala  Vključimo vse stavke (povedne, velelne, vprašalne itd.) in najrazličnejše situacije rabe sintetizatorja govora, besedilo izdelamo statistiko posameznih vrst povedi oz. stavkov. mora vsebovati tudi zelo kratke in zelo dolge povedi, 421 2. Izdelava spiska glasovnih nizov z oceno zaželenosti sintetizator govora najpogosteje uporabljal. Tako se sintetizator posameznega niza: govora še posebej prilagodi izbranim domenam, za katere prebrani  V spisek vključimo nabor vseh teoretično možnih umetni govor potem zveni še posebej naravno in ga »na hitro« kombinacij difonov. skoraj več ni moč ločiti od človeškega govora.  V spisek vključimo vse trifone, štirifone in (po potrebi) Za izbor fonetično in prozodično uravnoteženega besedila ostale zaželene (najpogostejše) polifone, na katere smo predlagamo uporabo besedilnega korpusa Gigafida , ki vsebuje 1,2 naleteli pri statistični obdelavi besedil. milijarde besed v slovenskem jeziku.  Utež oz. ocena zaželenosti niza je odvisna od pogostosti njegovega pojavljanja v besedilu. 4. PRIPOROČILA ZA IZBIRO 3. Postopek izbire povedi:  Ocenimo doprinos glasovnih nizov za vsako poved iz GOVORCEV tekstovnega korpusa. Posname naj se vsaj 10 različnih govorcev. Te krajše posnetke  Doprinos povedi je enak vsoti vseh ocen zaželenosti nizov (nekaj sto stavkov z dobrim pokritjem difonov) se nato strojno (iz spiska), ki se v povedi pojavijo. označi in preizkusi na aktualnem sintetizatorju govora. Po možnosti  Doprinos posamezne povedi normiramo z dolžino povedi se izvede tudi anketa med več poslušalci glede naravnosti in (št. besed v povedi ali št. fonemov v povedi). razumljivosti govora, pa tudi glede subjektivne ocene, kateri glas  Določimo takšno utež, da bodo dolžine izbranih stavkov se jim preprosto zdi najprijetnejši za poslušanje. čim bolj ustrezale statistični porazdelitvi dolžin stavkov iz Dobro je, da je glas takšen, da se v postopku sinteze mesta korpusa. »lepljenja« čim manj opazijo oz. slušno zaznajo. Pri izbiri govorca  Izberemo poved z najvišjim normiranim doprinosom. je potrebno upoštevati tudi njegovo sposobnost sledenja napotkom,  Iz spiska odstranimo vse glasovne nize, ki jih izbrana poved potrebne ponovitve med snemanjem, čas snemanja ipd. vsebuje. Potrebujemo vsaj en moški in en ženski glas. Smotrno je, da  Ponovno ocenimo vsako poved in izberemo najboljšo sintetizator govora razpolaga s po dvema glasovoma za vsak spol. (glede na novi spisek, v katerem so izločeni tisti glasovni Pomembno je, da se kandidate vnaprej seznani z namenom nizi, ki smo jih že pokrili) ter popravimo spisek. snemanja in možne uporabe tako pridobljenih glasov. Pred  Postopek ponavljamo, dokler ne izberemo želenega števila snemanjem morajo izbrani govorci podpisati pogodbo oz. privolilo, povedi. da dovolijo rabo posnetkov za potrebe sinteze govora. 4. Ovrednotenje rezultatov:  Vsakih 1000 povedi izdelamo statistiko difonov, trifonov, DEJAVNIKI GOVORCEV štirifonov in drugih polifonov, ki jih že pokrivamo (gre za Pri izbiri govorcev moramo upoštevati nekatere značilnosti, ki so glasovne nize, ki smo jih do takrat že izločili iz zgoraj povezane z njihovim govorom. Pri tem razlikujemo med [7]: omenjenega spiska).  prehodnimi značilnostmi oz. tranzienti in 5. Dodatne izboljšave algoritma:  trajnimi značilnostmi.  Ker mora zbirka vsebovati vse možne kombinacije difonov, Prehodne značilnosti so morebitne psihološke in fiziološke motnje algoritem popravimo tako, da difone dodatno utežimo glede na ostale polifone. Na takšen način bo algoritem na (npr. počutje, bolezen, psihično stanje). začetku dajal prednost povedim, ki bodo pokrile čim več Med trajne značilnosti pa uvrščamo fiziološke in anatomske novih difonov. Predvidoma se vsi difoni pokrijejo že po ca. značilnosti (npr. spol, starost, težo, okvare na govorilih, kadilske in 100 stavkih. pivske navade) ter geografske in socialno-jezikovne vplive na  Pri trifonih in štirifonih upoštevamo pri robnih glasovih govor (izobrazbo in poklic govorcev, narodnostno-narečno tudi podatek o glasovni skupini, ki ji pripadajo (npr. štirifon področje trenutnega in morebitnega predhodnega bivališča "krak" ne bo doprinesel prav dosti novega v našo zbirko, če govorcev, narodnostno-narečno področje govorčevih staršev). ta že vsebuje štirifon "krat"; zato oceno koristnosti takega GOVORNI DEJAVNIKI štirifona popravimo navzdol). To lahko naredimo preprosto tako, da v spisek vnesemo dodatne nize skupaj z njihovimi Med govorne dejavnike uvrščamo prozodijske značilnosti govora, frekvencami pojavljanja v korpusu (primer takega štirifona: ki se nanašajo na trenutno razpoloženje govorca ter njegov pristop "k"+"r"+"a"+"pripornik"). k tvorjenju govornih posnetkov [7]. Posneti govor lahko tako  Algoritem z različnim uteževanjem izboljšamo tako, da označimo za hiter, počasen, napet, sproščen, odrezav, natančen, končni nabor vsebuje različne povedi (povedne, vprašalne, površen ipd. velelne, enostavne, sestavljene, naštevanje itd.). Tako V praksi se izkaže, da so nekateri glasovi preprosto bolj primerni lahko isti korpus učinkovito uporabimo tudi za generiranje za izdelavo sintetizatorjev govora kot drugi. Pri tem je zelo težko prozodičnih parametrov pri sintezi govora. vnaprej napovedati, ali je nek glas primeren ali ne, pri tem ni splošno veljavnih pravil. Fonetični prepis besedila se izvede z modulom za grafemsko - fonemsko pretvorbo, s katerim razpolagajo projektni partnerji. Posnetih naj bo več 10 (do 50) ur govora. Prva ura posnetkov naj 5. PRIPOROČILA ZA SNEMANJE bo označena na zgoraj opisani način (čim boljša fonetična GOVORNE ZBIRKE uravnoteženost). Besedilo za preostale posnetke do 10 ur se izbere Besedilo, ki vsebuje vsa želena zaporedja alofonov, je tako, da so stavki čim bolj pestri in zajemajo različne prozodične najpriporočljiveje prebrati naenkrat. Zelo pomembno je namreč, da variante. Na takšen način dobimo dovolj učnega gradiva za govorec skozi vso besedilo govori na enak način, z enakim glasom, pravilno nastavljanje prozodičnih parametrov sintetizatorja. z enako intonacijo, skratka z enakimi parametri govora. Razliko do preostanka polnega obsega govorne zbirke se zapolni z Snemanje besedila po kosih v daljšem časovnem obdobju je manj branjem različnih zvrsti besedil, za katere se predvideva, da se bo priporočljivo, saj se govorcu lahko glas zaradi različnih zunanjih 422 vzrokov (vreme, drugačne nastavitve pri snemanju, spremenjen izgovarjav, različne hitrosti govora ipd. popravimo ob snemanju spekter in intenziteta motenj iz okolice) ali notranjih vzrokov naslednje seje. (razpoloženje, bolezen) spremeni, govorna zbirka pa ni v celoti posneta, kar oteži kvalitetno sintezo govora. 6. OZNAČEVANJE GOVORNIH Snemanje govornega gradiva naj poteka ob prisotnosti izkušenega POSNETKOV snemalnega operaterja z namenom, da se preprečijo neustrezne Predlagamo, da se uporabi tri nivoje anotacij oz. prepisov izgovarjave besed in napake pri snemanju govora. Govorcu je govorjenega besedila: grafemski prepis, fonetični prepis in potrebno pred snemalnimi sejami podati ustrezna navodila in ga zaprositi, da povedi prebira razločno in enako prozodijske oznake. merno hitro. Med branjem besedila imajo govorci nameščene elektrode laringografa, Ker je ročna segmentacija govora na fonetičnem nivoju naporna in s katerimi se spremlja nihanje glasilk zaradi lažjega kasnejšega dolgotrajna, pri tem uporabljamo vsaj delno avtomatizirane označevanja osnovnih period govornega signala. postopke, ki so bolj učinkoviti, če vnaprej poznamo grafemski Samo snemanje celotne govorne zbirke zaradi obsežnosti besedil prepis govorjenega gradiva. a, ki ga je treba prebrati, navadno traja več mesecev. Pri tem morajo Avtomatskim metodam in postopkom po potrebi sledi »ročno« nastavitve snemalne opreme ves čas ostati nespremenjene. Pred popravljanje oznak, kar je ne glede na hiter razvoj tehnologije še vsakim snemanjem naj govorec posluša svoje predhodne posnetke, vedno zelo zamudno. s čimer se skuša zagotoviti čim bolj enoten način govora med posameznimi snemalnimi sejami. Na začetku snemanja posamezne 7. ZAKLJUČEK seje naj govorec prebere nekaj vnaprej določenih fiksnih stavkov, Pri razvoju metod in priporočil za izgradnjo govorne zbirke ki omogočajo primerjavo glasnosti in višine govora med CityVOICE smo posebno pozornost namenili določanju optimalnih posameznimi snemalnimi sejami. pogojev za snemanje govornih zbirk, določanju optimalnih Med snemanjem govora je priporočljivo preverjati, ali govorec fonetično in drugače uravnoteženih vsebin za snemanje govornih ustrezno izgovarja predloženo besedilo. To lahko storimo s zbirk ter rešitvam za iskanje optimalnih govorcev. prisotnostjo druge osebe med snemanjem ali pa poskušamo v snemalni postopek vgraditi določeno samodejno preverjanje. 8. ZAHVALA Govor snemamo preko mikrofona v digitalni obliki, po navadi kar Operacijo CityVOICE sofinancirata Republika Slovenija in na namenski računalnik v studiu. Potrebujemo še ustrezno mešalno Evropska unija iz Evropskega sklada za regionalni razvoj, in sicer mizo, zaslon in slušalke (preko katerih govorec prejema navodila v okviru »Operativnega programa za izvajanje evropske kohezijske in posluša povratni govor). Posnetke govora shranimo v digitalni politike v obdobju 2014-2020«. obliki na trajne računalniške pomnilniške medije. Med posameznimi sejami nastavitev ne spreminjamo (uporabljamo 9. LITERATURA IN VIRI vnapej preddefinirane nastavitve opreme). Ustreznost govornih [1] Gorjanc, V. 2005. Uvod v korpusno jezikoslovje. Govor in parametrov lahko preverjamo z ustreznimi merilci (npr. Bruel & jezikovne tehnologije, januar 2014. Kjaer 2230 Sound Level Meter). [2] Zemljarič Miklavčič, J. 2008. Govorni korpusi. Univerza v Govor se snema z več različnimi mikrofoni in pod različnimi koti Ljubljani, Filozofska fakulteta. (npr. Studio Projects T3 Dual Triode). Pred mikrofoni je primeren filter (angl. »anti [3] Žganec Gros, J., Mihelič, F. in Dobrišek, S. 2003. Govorne -pop filter«), ki zaduši razne poke, tleske ipd. tehnologije: Pridobivanje in pregled govornih zbirk za Zaželeno je, da imamo nameščeno tudi spletno kamero in ogledalo. slovenski jezik. Jezik in slovstvo IIL/3–4 (2003), 47–59. S tem se govorcu pomaga, da ohranja konstantno razdaljo do mikrofona (tudi med različnimi seansami snemanja). [4] Žganec Gros, J., Vesnicer, B.,Rozman, S., Holozan, P., Šef, T. 2016. Sintetizator govora za slovenščino eBralec. Zbornik Oseba, ki snemanje nadzoruje, lahko preveri položaj govorca pred konference Jezikovne tehnologije in digitalna humanistika, vsako snemalno sejo in jo primerja s položaji v predhodnih sejah. Filozofska fakulteta, Univerza v Ljubljani, 2016. Majhno ogledalo (na steni) govorcu omogoča ohranjati primerno [5] Šef, T., Romih, M. 2011. Zasnova govorne zbirke za razdaljo tekom same snemalne seje. Govorce se zaprosi, da periodično preverjajo položaj svojega obraza sintetizator slovenskega govora Amebis Govorec, v ogledalu [8]. Informacijska družba IS 2011. Priporočamo frekvenco snemanja 44,1 kHz in 24-bitno vzorčenje [6] Chevelu, J., Lolive, D. 2015. Do Not Build Your TTS Training govornega signala. Corpus Randomly. Proc. EUSIPCO 2015, Francija. Ker je pri snemanju pomembno, da govorec ni preveč utrujen, [7] Dobrišek S. 2001. Analiza in razpoznavanje glasov v predlagamo 10-minutno snemanje znotraj pol urnega intervala. Posamezna seja naj traja dve uri; znotraj tega časa je možno posneti govornem signalu. Doktorska disertacija, Fakulteta za elektrotehniko, Univerza v Ljubljani. 40 minut govornega materiala (v štirih sejah). Vsak govorec naj opravi le eno dveurno sejo na dan. Za 50 ur posnetkov tako [8] Oliveira, L.C., Paulo, S., Figueira, L., Mendes, C., Nunes, A., potrebujemo 75 sej oz. 150 ur časa. Godinho, J. 2008. Methodologies for Designing and Razlike pri pogojih snemanja med različnimi sejami je možno Recording Speech Databases for Corpus Based Synthesis. detektirati s primerjavo povprečnih vrednosti Proceedings of the Sixth International Language Resources MFCC parametrov and Evaluation (LREC 2008), Marrakech, Morocco. posameznih izgovarjav [9]. [9] Richmond, K., Strom, V., Clark, R. A. J., Yamagishi, J., Fitt, Med posameznimi sejami preverimo predhodne posnetke. Morebitne napake zaradi neenakih snemalnih pogojev, napačnih S. 2007. Festival multisyn voices for the 2007 blizzard challenge. Proc. Blizzard Challenge Workshop (Proc. SSW6), ISCA. 423 Telemetric ECG monitoring during physical activity in field tests Boris Širaiy Roman Trobec Institut “Jožef Stefan” Institut “Jožef Stefan” Jamova cesta 39 Jamova cesta 39 1000 Ljubljana 1000 Ljubljana +386 31 616 169 +386 41 708 994 boris.siraiy@hotmail.com roman.torbec@ijs.si ABSTRACT 2. MATERIAL AND METHODS In this paper we show how it is possible to measure ECG signal with telemetric ECG body-sensor during physical activity in 2.1 Study Population regular conditions. Based on previous studies we choose position In this study was included one participant, age 24 years. He was a and type of fixation for sensor. Participant made three different student at the Faculty of Sport and Physical Education at the tests, first test was shuttle run test where speed of running was University of Belgrade. He was healthy and without known increased every minute, second test was Cooper 2400 m test, and previous cardiac problems. Before the tests the study purpose, and third test was 100 m maximal sprint. We measure ECG signal in its protocol were explained to him, and he signed consent for all three tests and analyse it with special software for Holter participation in the study. analysis. 2.2 Experimental Setup Keywords ECG measurements during the field tests were made with wireless ECG body sensors, Shuttle run, Cooper 2400 m, 100 m sprint, ECG body sensors Savvy (Saving d.o.o., Ljubljana, Slovenia) Healthcare. [13], which is a certified medical device, described in detail in the study of Trobec et al. [14]. The body sensor is light and non- 1. INTRODUCTION obstructive for users, which allows long-term exercise ECG measurements. The sensor is fixed on the body via two self- Nowadays sudden cardiac death presents extremely high risk for adhesive skin electrodes. An Android application, MobECG, sudden death of professional sportsmen and recreational athletes. which runs on a smartphone, captures and displays the measured Cardiovascular disease causes more than 90 % of sudden deaths data and saves it in the smartphone’s memory for further during physical activity [1]. Two France studies reported a daily processing. incidence of 3 sudden deaths and 4 myocardial infarctions during physical activity in the general population [2,3]. Studies from USA shows sudden cardiac death incidence between sportsmen in high schools and colleagues is 1:200.000 per year [4,5,6], while European studies shows that incidence is even higher, 1- 1,6:100.000 [7,8]. Considering the fact that 55-80 % of the athletes who died of sudden cardiac death had no prior symptoms of heart disease [9], the question is what else should be done for the prevention of sudden cardiac death, and to detect the people who has increased risk of sudden cardiac death. Telemetric ECG body sensor used in this paper was already used in previous pilot studies made during light physical activity [10], and in maximal laboratory exercise stress test (EST) [11]. Since an EST with ECG monitoring is usually performed in laboratory conditions, our goal is to determine whether it is possible to measure an ECG during regular sport activities, because it will offer a significant advantage for the prevention of cardiovascular accidents [12]. Aim of this study was to evaluate body-sensor ECG signals during three different field tests, and to analyse the quality of the Figure 1. Sensor position and type of sensor fixation in the recorded ECG signal. Next aim was to determine if different types first test (left-hand) and in the second test (right-hand). of sensor fixation influenced the ECG signal. The use of a telemetric ECG during physical activity could be very beneficial for medical doctors and sports scientists, and the most of all for The sensor position should be close to the heart to obtain professional and recreational athletes. appropriate amplitude of ECG signal. In addition, its position should avoid large muscles, due to the signals from the electrical 424 muscular activity (EMG) that could disturb the ECG [14,15]. In this study was used position Left Inferior (LI), shown as most adequate in previous study [11]. In the LI position the sensor electrodes are at the positions V1 and V2 of standard precordial leads, and the sensor is translated by approximately 10 cm, below the xiphoid, where the influence of muscular disturbances is expected to be minimal (see Figure 1). The ECG electrodes were positioned 5 cm apart [16]. Before the positioning, we cleaned the skin of the subjects with diluted ethanol. The participant made six tests, two shuttle run, two Cooper 2400 m, and two 100 m sprint test. In the first test the ECG body sensor were fixed with self-adhesive medical Omniplast 2,5 cm tape (Paul Hartmann AG, Heidenheim, Germany), specially Figure 2. An example of adequate quality signal. designed to fix Holter electrodes. Both parts of the sensors were fixed together with one, approximately 40-cm-long strip of tape. In the second test sensor were fixed with standard Polar belt made An example of screenshot of a signal which is not of for measurements of the heart rate (HR) with Polar system. adequate quality for interpretation and for further analysis is shown in Figure 3. The measured ECG data were continuously stored in the mobile- phone memory and transferred to the personal computer. The ECG analysis were made with medically certified Holter interpretation software QuickReader® AFT-1000 (Holter Supplies, Paris, France). 2.3 Measurement Protocol Every test was made in a separate day, except 100 m sprints where both tests were made in the same day, with 30 min pause in between tests. Before performing the tests, the electrodes and sensors were positioned at the LI position and fastened with self- adhesive tape. After that the participant sat down and waited for 5 minutes, while the ECG was recorded. After 5 minutes the participant started with the test. When the test was finished the participant sat down and rested for 5 minutes, while the ECG was Figure 3. An example of a signal which cannot be interpreted. still recording. The next day the participant took part in a second test. In the second test sensors were fixed with Polar belt. The The computerized heart-rate analysis was successful for most of ECG recording protocol was the same as on the previous day. The the time during both shuttle run tests and during Cooper and 100 next type of test, under the same protocol, were performed in the m tests when sensor was fixed with Polar belt. During the visual next 2 days. The study was conducted in accordance with the inspection we also found erroneously detected QRS complexes, ethical standards of the Faculty of Sport and Physical Education using the Holter interpretation software, often because of the (IRB approval No. 02-1359/18-2), University of Belgrade, and the excessive artefacts that come from the intense activity (right-hand Helsinki Declaration. part of Figure 4). 3. RESULTS In the first shuttle run test participant successfully made 11 minutes of the test (11/2) with the maximal heart rate (HR) 196 beats per minute (bpm), while in second test he successfully made 10 minutes of the test (10/10) with the maximal HR 192 bpm. In the first Cooper test he run 11:49 minute, with the maximal HR 192 bpm, and in the second test he run 13:10 min, with the maximal HR 193 bpm. In the 100 m sprint test he run 13:45 sec in the first test, with maximal HR 172 bpm, while in second test he run 13:84 sec, with the maximal HR 179 bpm. Analyse of the signal show that ECG signal was of adequate quality in both shuttle run tests, while in Cooper and in 100 m Figure 4. Visualization of the detected heart beats using the sprint test signal was of adequate quality only in second test when Holter interpretation software QR. The correctly interpreted sensor was fixed with a Polar belt. An example of screenshot of a QRS complexes are shown in the left-hand panel. An example typical HR signal in bpm, and correct QRS complex detection of a of the erroneously interpreted QRS complexes is shown in the signal which is of adequate quality for interpretation and for right-hand panel. further analysis is shown in Figure 2. 425 4. DISCUSSION [4] Maron, B.J., Gardin, J.M., Flack, J.M., Gidding, S.S., This paper show that a wireless ECG body sensor can be used for Kurosaki, T.T., Bild, D.E. 1995. Prevalence of hypertrophic non-obstructive measurements of an ECG during some regular cardiomyopathy in a general population of young adults: physical activities. The aim of this study was to check if wireless echocardiographic analysis of 4111 subjects in the CARDIA ECG body sensors can be used during field tests. The results show study. Circulation, 92, 785-789. that the type of fixation and the type of test influence the results. [5] Maron, B.J., Shirani, J., Poliac, L.C., Mathenge, R., Roberts, When sensor was fixed with Polar belt signal was of adequate W.C., Mueller, F.O. 1996 . Sudden death in young quality during entire test, but when sensor was fixed with competitive athletes: clinical, demographic, and pathological Omniplast tape the signal was of adequate quality only in shuttle profiles. JAMA, 276, 199-204. run test, when the speed of running starts with 8.5 km/h and [6] Pedoe, D.T. 2000. Sudden cardiac death in sport-spectre or increases 0.5 km/h every minute, while in tests when speed of preventable risk? Br. J. Sports Med., 34, 137-140. running was constantly high the signal was not acceptable. In that case maximal speed of running in shuttle run test was between [7] Epstein, S.E., Maron, B.J. 1986 . Sudden death and the 13.5 and 14.0 km/h, and this was very similar to previous competitive athlete: perspectives on preparticipation laboratory study which confirm that ECG signal on a treadmill screening studies. Journal of the American College of was of adequate quality for speeds of running up to 13.5 km/h. Cardiology, 7, 220-230. [8] Pigozzi, F., Spataro, A., Fagnani, F., Maffulli, N. 2003 . The diagnostic ability of the ECG body sensor, used in this study, Preparticipation screening for the detection of cardiovascular has been compared in various previous pilot studies with other abnormalities that may cause sudden death in competitive similar devices [14] and with standard 12-lead ECG [14,17]. athletes. British Journal of Sports Medicine, 37, 4-5. Previous studies shown that ECG body sensor, even so simple, can also detect most of the arrhythmic events, e.g., atrial or [9] Drezner, J.A., Courson, R.W., Roberts, W.O., Mosesso, ventricular fibrillation, exstrasystole, tachycardias, bradycardias, V.N., Link, M.S., Maron, B.J. 2007. Inter-association task etc. The analysed measurement methodology can provide basic force recommendations on emergency preparedness and information about the heart rhythm’s status. In the case of any management of sudden cardiac arrest in high school and detected abnormality the users can be directed to further college athletic programs: a consensus statement,” J. Athl. diagnostics with a standard stress test and medical personnel. Train., 42, 143-158. [10] Širaiy, B., Stanič, U.J., Poplas-Sušič, A., Katkič, Z. 2018. 5. CONCLUSION Impact assessment of the mornint gymnastics “1000 The presented results are a motivation for further study, where movements” via ECG and sport tests. 41st Int Conven. on more participants will be included. With such a telemetric Inform. and Communic. Technol., Electron. and approach it might be possible, to the best of our knowledge for the Microelectron. (MIPRO), 284-288. first time, to routinely measure ECG signals in real conditions, [11] Širaiy, B., Ilić, V., Trobec, R. 2018 . Evaluating telemetric e.g., when users are running in nature, during a significant ECG body sensor signal in exercise stress test: Pilot study. activity. In the presented paper an appropriate method of sensor 41st Int. Conven. on Inform. and Communic. Technol., fixation was found; however, to confirm this conclusion study Electron. and Microelectron. , 289-294. should be extended, and heterogenous group of participants, e.g., [12] Casa, D.J., Guskiewicz, K.M., Anderson, S.A., Courson, male and female, should be included. R.W., Heck, J.F., et al. 2012. National Athletic Trainers' Association Position Statement. Preventing Sudden Death in 6. ACKNOWLEDGMENTS Sports. J. Athl. Train., 47(1), 96-118. This paper was supported by Slovenian Research Agency under [13] www.savvy.si, last time visited on 15. September, 2018. the Grant P2-0095. [14] Trobec, R., Tomašić, I., Rashkovska, A., Depolli M., Avbelj, 7. REFERENCES V. 2018. Body sensors and electrocardiography, SpringerBriefs in Applied Sciences and Technology. [1] Mazić, S., Ilić, V., Đelić, M., Arandjelović, A. 2011. Sudden cardiac death in young athletes. Srp. Arh. Celok. Lek., 139, [15] Clancy, E.A., Morin, E.L., Merletti, R. 2002. Sampling, 394-401. noise-reduction and amplitude estimation issues in surface electromyography. J. Electromyogr. Kinesiol., 12(1), 1-16. [2] Chevalier, L., Hajjar, M., Douard, H., Charief, A., Dindard, J.M., et al. 2009. Sports-related acute cardiovascular events [16] Kania, M., Rix, H., Fereniec, M., Fernandez, H.Z., Janusek, in a general population. A French prospective study. Eur. J. D., Mroczka, T., Stix G., Maniewski, R. 2014. The effect of Cardiovasc. Prev. Rehabil., 16, 365-370. precordial lead displacement on ECG morphology. Med. Biol. Eng. Comput., 52, 109-119. [3] Marijon, E., Uy-Evanado, A., Reinier, K., Teodorescu, C., Narayanan, K., Jouven, X., Gunson, K., Jui, J., Chugh, S.S. [17] Trobec, R., Avbelj, V., Rashkovska, A. 2014. Multi- 2015. Sudden cardiac arrest during sports activity in middle functionality of Wireless Body Sensors. Transac. Int. Res., age. Circulation, 131, 1384-1391. 10(1), 23-27. 426 Napredni pogovorni svetovalci Aleš Tavčar Matjaž Gams Institut "Jožef Stefan" Institut "Jožef Stefan" Jamova cesta 39, Jamova cesta 39, 1000 Ljubljana 1000 Ljubljana ales.tavcar@ijs.si matjaz.gams@ijs.si POVZETEK domenah. Posredovanje informacij o čakalnih dobah za Inteligentni pogovorni svetovalci olajšajo uporabnikom iskanje posamezne posege in poizvedovanje po platformi Think!EHR. informacij in komunikacijo z informacijskimi sistemi, saj omogočajo dostop do ogromne količine informacij na semantičnem spletu z uporabo naravnega jezika in s 2. ZASNOVA SISTEMA poizvedovanjem po strukturiranih podatkih, glede na Splošna arhitektura predlaganega sistema je sestavljena iz štirih uporabniške poizvedbe. Učinkovitost tovrstnih sistemov se je glavnih komponent: občutno povečala z razvojem semantičnega spleta, ki je v zadnjih - spletnega vmesnika za zajem uporabniških poizvedb, letih pripeljal do velike količine podatkov objavljenih na spletu na podlagi povezanih podatkovnih načel, kar olajša - pogovornega agenta znotraj sistema DialogFlow, avtomatsko - zalednega mehanizema, ki procesira zahtevke iz obdelavo podatkov. V tem prispevku predstavimo inteligentni sistem, ki je sposoben razumevanja vprašanj uporabn DialogFlow agenta, ikov in - univerzalnega vmesnika za dostop do podatkovnih lahko posreduje podatke ter informacije iz obstoječih spletnih zbirk. servisov ter podatkovnih virov. Ključne besede Spletni vmesnik je implementiran v obliki pogovornega okna pogovorni svetovalci, procesiranje naravnega jezika, think!EHR (angl. chat), kjer se vprašanja in odgovori izmenično prikazujejo 1. UVOD (glej Slika 4). Vmesniku so dodani tudi izbirni gumbi, kar olajša uporabniku interakcijo, saj mu ni potrebno pisati in lahko izbere Eno bolj dinamičnih področij raziskav je danes interakcija med želeno opcijo (glej Slika 4 človekom in računalnikom z razvojem inteligentnih ). Aplikacija je implementirana v html sistemov ali jeziku, z uporabo jquery komponent za dinamično spreminjanje aplikacij, ki so sposobni komuniciranja v naravnem jeziku in vsebine in prikazovanje kompleksnejših elementov. Zaledni del, upravljanja z obširnimi bazami znanj [1]. Take sisteme pogosto ki procesira vnose uporabnika pa je implementiran v spletnem imenujemo pogovorni svetovalci ali virtualni agenti. Takšni ogrodju Django. agenti so sposobni posredovati celovito znanje o določenih domenah, kar olajša dostop do ogromnih količin podatkov, ki Za učinkovito procesiranje naravnega jezika je bilo uporabljeno obstajajo na svetovnem spletu. Svetovalci so v zadnjih letih orodje DialogFlow. Gre za spletno storitev, ki jo podpira Google postali izredno uporabno orodje, ki uporabnikom omogoča in se izvaja na Googlovi oblačni infrastrukturi in omogoča iskanje informacij hitreje in natančneje. Podjetja, kot so Google, uporabo orodij za procesiranje in razumevanje vprašanj, Microsoft ali Apple, nudijo splošne pomočnike, ki uporabnikom postavljenih v naravnem jeziku. Vnos, ki se ga želi analizirati se pomagajo najti restavracije ali trgovine blizu njihove lokacije, iz zalednega dela posreduje DialogFlow servisu preko ustreznih upravljati svoj telefon ali nuditi računalniško podprto podporo za knjižnic. Za posamezno domensko področje (baza znanja programske pakete. V prihodnosti bodo programski paketi in svetovalca) se ustvari agent, ki ga s posredovanjem množice spletne platforme, v določeni obliki, vsebovale pogovorne učnih primerov naučimo razločevati med nameni uporabnika. Za svetovalce, ki bodo uporabnikom svetovale in pomagale pri vsak namen, ki ga definiramo, vnesemo večje število vprašanj in uporabi programske rešitve. prepustimo sistemu, da zgradi odločitveni model. Na podoben način je mogoče iz teksta, ki ga uporabnik vnese, izluščiti Na področju procesiranja naravnega jezike se je v zadnjem času vrednosti za različne parametre, ki jih predhodno definiramo razvilo precej orodij (spletnih storitev), ki omogočajo (datumi, kraji, imena ipd.). procesiranje tekstovnega vnosa, določanje namena uporabnika in tudi luščenje vrednosti določenih, v naprej definiranih Rezultat procesiranja v DialogFlow agentu je JSON objekt, ki vsebuje klasifikacijo uporabniškega vnosa v enega od definiranih parametrov. Izmed teh so najbolj poznane DialogFlow [2] in Rasa [3]. Prvi je Googlov sistem, ki deluje v oblaku in uporabniki razredov (angl. intents) in seznam morebitnih vrednosti dostopajo do funkcionalnosti preko REST vmesnika. Rasa pa je parametrov, ki jih je sistem izluščil. To prejme zaledni del odprtokodna rešitev, ki omogoča namestitev na lastnem (implementiran v Djangu) in na podlagi procesiranja JSON strežniškem sistemu. objekta določi nadaljnji potek. Ta lahko vsebuje že ustrezni odgovor, ki ga sistem nato vrne spletnemu vmesniku in ga Kombinacija spletnih ali mobilnih aplikacij, oblačnih sistemov prikaže v pogovornem oknu. Mehanizem pa lahko določi, da je za procesiranje naravnega jezika in zalednih funkcionalnosti, ki potrebno pridobiti odgovor ali podatke iz zunanjih virov ali omogočajo pridobivanje podatkov iz podatkovnih baz, podatkovnih baz. semantično anotiranih podatkov na spletu ali spletnih platform, Zahtevo se nato pošlje do ustreznega vmesnika, ki je sposoben omogoča razvoj uporabnih in naprednih aplikacij za komunicirati z zunanjim virom znanja. Ta pretvori JSON komunikacijo z uporabnikom. podatke v zahtevek, ki ga zunanja podatkovna zbirka, aplikacija V pričujočem prispevku predstavimo sistem za procesiranje ali platforma razume. Običajno se generira REST zahtevek. vprašanj uporabnikov in naprednega posredovanja informacij. Delovanje pogovornega svetovalca demonstriramo na dveh 427 3. ČAKALNE VRSTE zanima. To je realizirano s kombiniranjem algoritma za Definirani inteligentni sistem smo najprej aplicirali na domeno indeksiranje in klasifikacijo z DialogFlow agentom. V primeru, svetovanja o čakalnih vrstah za zdravstvene storitve. ko svetovalec ni prepričan o posegu, uporabniku vrne množico elementov, za katere je določil najvišjo verjetnost ujemanja. Ti 3.1 Implementacija so prikazani v obliki gumbov, katere lahko uporabnik izbere (glej Aplikacija uporablja generičnega svetovalca, ki smo ga Slika 4). predstavili v prejšnjem razdelku. Ustvariti smo morali DialogFlow agenta in ga naučiti razpoznavati ustrezne zahtevke uporabnikov. Iz vprašanj uporabnika agent poskuša pridobiti tri vrste podatkov: poseg, nujnost napotnice in regijo posega. Agenta smo naučili tako, da smo mu podali večje število primerov vprašanj za različne kombinacije parametrov. Uporabnik lahko povpraša po posegih na različne načine in v različnih sklonih. Zato smo pri iskanju ustreznih posegov uporabili tudi algoritem za indeksiranje nizov. V primeru, da DialogFlow agent ne najde natančnega zadetka za poseg, se izvede delno iskanje po indeksiranih posegih in metoda vrne seznam posegov, z izračunanimi verjetnostmi. Nekaj najverjetnejših posegov se ponudi uporabniku v izbiro preko izbirnih gumbov. Primer uporabe sistema je zaporedje vnosov, ki podajo informacijo o čakalnih dobah za določen poseg v določeni regiji. Uporabnik najprej poda poseg za katerega se zanima. Svetovalec nato poskuša pridobiti manjkajoče podatke, ki jih potrebuje za pridobitev informacij o čakalnih dobah. Najprej uporabniku prikaže izbirne gumbe z nujnostjo napotnice. V naslednjem koraku uporabnik izbere regijo, kjer želi izvesti poseg. V primeru, da uporabnik že v osnovnem zahtevku poda vse potrebne informacije, se ta dva koraka izpustijo. Primer takega vprašanja je: »Pod nujno iščem zobarja na štajerskem«. Podatke o čakalnih dobah pa zaledni del pridobi iz spletnega servisa [4], kamor se pošlje http POST zahtevek, odgovor se Slika 2: Prikaz rezultatov poizvedovanja za zobozdravstveni procesira in informacije o ustanovah in ustreznih čakalnih dobah pregled. shrani v strukturiran JSON objekt. V naslednjem koraku uporabnik preko vnosnih gumbov izbere 3.2 Primeri uporabe nujnost napotnice in regijo, v kateri želi opraviti poseg. Sistem nato pošlje poizvedovanje na spletni servis za razbrane parametre V tej sekciji predstavimo nekaj primerov uporabe aplikacije za in pridobi čakalne dobe za različne zdravstvene ustanove. posredovanje informacije o čakalnih dobah. Slika 1 prikazuje Uporabniku se prikaže 5 ustanov, kjer je čakalna doba najkrajša. začetni pogled spletne aplikacije. Ob zagonu vmesnika se Vsaki ustanovi je pripisan naslov in kontaktni podatki. Primer prikažejo pozdravni nagovori, kjer se pogovorni svetovalec izpisa je prikazan na Sliki 2. predstavi in uporabnika obvesti, da z ključno besedo »pomoč« lahko pridobi več informacij in prejme dodatna navodila. 4. THINK!EHR SVETOVALEC Uporabnik vnaša vprašanja in zahteve v spodnje vnosno polje. Naslednja domena kjer smo preizkusili pogovornega svetovalca je poizvedovanje po Think!EHR platformi. V tej aplikaciji se mora uporabnik najprej prijaviti. Svetovalec loči med dvema vlogama: zdravnik in pacient. Na podlagi vloge svetovalec posreduje različne tipe in obseg podatkov iz Think!EHR platforme. 4.1 Implementacija Pri implementaciji aplikacije smo ponovno uporabili generičnega svetovalca, ki smo ga prilagodili glede na specifike domene. Think!EHR platforma je namenjena zdravstvenim delavcem in vsebuje različne podatke o pacientih: osebni podatki, relacije med pacienti, rezultati analiz ipd. Zdravnik lahko pregleduje zgolj podatke svojih pacientov, določena oseba pa zgolj svoje podatke in meritve. Zaradi te logične razdelitve smo spletnemu vmesniku dodali možnost prijave (glej Slika 3), kjer se lahko prijavijo zdravniki ali pacienti in tako se določi uporabniku vloga. Na podlagi tega se nato uporabniku omogoči različna Slika 1: Začetni pogled svetovalca za posredovanje informacij o čakalnih vrstah. poizvedovanja in prikaz različnih tipov in obsega podatkov. Za pregledovanje meritev smo spletnemu vmesniku dodali Odgovori sistema se prikažejo na desni strani in so osenčeni s sivo barvo. Vnosi uporabnika pa se prikažejo na levi strani in so grafični prikaz v ozadju pogovornega svetovalca (glej Slika 5). osenčeni s svetlo zeleno barvo. Ta omogoča prikaz poljubnih podatkov v meritvah na Svetovalec najprej poskuša preglednejši način. razbrati iz vnosov uporabnika za kateri zdravstveni poseg se 428 Na novo smo generirali DialogFlow agenta in ga naučili razpoznavati različne zahtevke, ki jih uporabnik lahko posreduje. Agent načeloma lahko iz teksta izlušči imena, različne nazive meritev in datume. Pridobivanje podatkov pa je realizirano preko REST vmesnika ehrscape platforme [5]. Določene podatke smo lahko pridobili preko osnovnih API klicev, za kompleksnejše poizvedbe pa smo uporabili AQL jezik, ki omogoča poizvedovanje po EHR zapisih. 4.2 Primeri uporabe Pogovornemu svetovalcu je dodano prijavno okno, kjer se mora uporabnik prijaviti, da lahko dostopa do podatkov v platformi (glej Slika 3). S prijavo se uporabniku določi ena od dveh možnih vlog: zdravnik ali pacient. Glede na to se prilagodi funkcionalnosti sistema in nabor možnih poizvedovanj po platformi. Slika 5: Grafični prikaz posamezne vrste meritev za vlogo pacienta Druga vloga, ki jo lahko uporabnik zavzame je zdravnik. V tem primeru lahko prijavljeni uporabnik dostopa do seznama svojih pacientov (glej Slika 6) in pregleduje osebne podatke o posameznem pacientu (glej Slika 7). Vnosnemu polju je dodana tudi funkcionalnost iskanja in avtomatske dopolnitve znakovnega niza (angl. autocomplete), ki olajša vnašanje imen pacientov določenega zdravnika. Uporabniški vlogi zdravnik je omogočena polna funkcionalnost sistema in dostop do vseh različnih poizvedb, ki jih platforma podpira. Kot v prejšnjem primeru lahko pridobi osebne podatke o svojih pacientih, išče podatke o sorodstvenih vezeh ter njihove vnesene informacije in Slika 3: Prijavni obrazec svetovalca. S prijavo se določi vlogo pregleduje vse sklope meritev, laboratorijskih izvidov, in funkcionalnosti. napotnice, lekarniške recepte, alergije itd. Za posamezne sklope Uspešno prijavljena oseba (pacient) lahko pregleduje vse svoje meritve in laboratorijske izvide mu je omogočen grafičen prikaz zapise, ki so shranjeni v platformi Think!EHR. Slika 4 prikazuje posameznih komponent v času, kot je prikazano na Sliki 5. posredovani seznam zapisov, ki so zabeleženi za določeno osebo. S klikom na gumb se izbere posamezno meritev. Slika 6: Prikaz vseh pacientov, ki so dodeljeni prijavljenemu Slika 4: Pregled vseh vrst zapisov v Think!EHR za vlogo zdravniku. pacient. Posamezni sklopi meritev za izbrani vnos se prikažejo na strani v ozadju (glej Slika 5). Posamezne meritve so časovno urejene in grafično prikazane. Uporabnik lahko poljubno vklaplja ali izklaplja prikaz posameznih meritev znotraj sklopa. 429 prejema rezultate analize teksta, ki ga uporabnik vnese preko spletnega vmesnika in na podlagi pridobljenih vrednosti parametrov določi ustrezni vir za pridobivanje zahtevanega podatka. Zadnja komponenta sistema pa je množica vmesnikov za komunikacijo z zunanjimi viri podatkov. Vsi vmesniki prejemajo zahtevke iz zalednega dela na enoten način, njihova vloga pa je transformacija splošnega zahtevka v obliko, ki jo razume zunanji servis. V večini primerov gre za definiranje REST zahtevka, ki se nato posreduje ustreznemu cilju. Sistem smo demonstrirali na dveh domenah. Implementirali smo svetovalca za posredovanje informacij o čakalnih dobah za posamezne posege in za pridobivanje podatkov iz zdravstvene platforme Think!EHR. Predstavili smo primere uporabe za obe aplikaciji in različne vloge uporabnika. Predstavljeni sistem ima velik potencial za integracijo v različne produkte podjetij, saj olajša interakcijo in izboljša uporabniško izkušnjo pri uporabi produkta ali storitve podjetja. ZAHVALA Raziskave in razvoj so nastale v okviru programa EkoSMART in so delno sofinancirane s strani Ministrstva za izobraževanje, znanost in šport in Evropske unije iz Evropskega sklada za regionalni razvoj (ESRR). Slika 7: Prikaz podatkov o posameznem pacientu za vlogo 6. REFERENCE zdravnik. [1] Seron, F. J. in Bobed, C. 2016. VOX system: a semantic 5. ZAKLJUČEK embodied conversational agent exploiting linked data. V pričujočem prispevku smo predstavili koncept splošnega Multimedia Tools and Applications 75, 1, 381-404. sistema pogovornega svetovalca, ki omogoča komunikacijo v [2] https://dialogflow.com/, dostop 12.9.2018 naravnem jeziku in ga je mogoče uporabljati na različnih [3] https://rasa.com/, dostop 12.9.2018 domenah. Svetovalec lahko uporablja statično bazo znanja v obliki vprašanja–odgovor, ali želene informacije/podatke pridobi [4] https://cakalnedobe.ezdrav.si/, dostop 13.9.2018 iz zunanjih virov. To pa so lahko podatkovne baze, internetni [5] https://www.ehrscape.com/api-explorer.html, dostop viri, spletni servisi, platforme itd. Sistem je sestavljen iz štirih 13.9.2018 glavnih komponent. Spletni vmesnik omogoča interakcijo z uporabnikom preko tekstovnega vnosa ali izbirnih gumbov. Sistem za procesiranje naravnega jezika kombinira metode iskanja po indeksiranih tekstovnih nizih in razpoznavanje namena uporabnika z uporabo storitve DialogFlow. Zaledni del 430 The potential of Blockchain technology in health monitoring Aleksandar Tošić Michael Burnard Jernej Vičič University of Primorska InnoRenew CoE University of Primorska Glagoljaška 8, Koper, Livade 6, Izola, Slovenija Glagoljaška 8, Koper, Slovenija University of Primorska Slovenija InnoRenew CoE Glagoljaška 8, Koper, ISJFR, SRC SASA Livade 6, Izola, Slovenija Slovenija Novi Trg 2, Ljubljana, Slovenia aleksandar.tosic@upr.si mike.burnard@innorenew.eu jernej.vicic@upr.si ABSTRACT research for years. Previously, however, scientists, research This paper is an overview of available technologies for a prac- institutions and governments have relied solely on a system tical implementation of a system that would facilitate large of self-regulation based on shared ethical principles and gen- scale sensor networks with some unique required features. It erally accepted research practices to ensure integrity in the outlines some special features and benefits of decentralized research process. Due to an increase in provable reports of systems and explores the possibilities of applying them in private and public entities sharing and processing user data building and health monitoring. Arguably, the adoption of without consent, awareness amongst the general public has Internet of Things systems suffers from a trade-off between increased substantially. Regulation has tightened following privacy, usability and availability. While these systems can the data protection movement and increased public aware- solve a variety of problems by acquiring enormous amounts ness, which is reflected by recent developments of the GDPR of data, they pose security and privacy concerns. We identify in Europe and the battle for net neutrality internationally. these trade-offs and the unique properties needed to resolve While these regulatory frameworks are still in their infancy, them in an effort to find the most suitable technology for a overcoming the clear barrier of storing and processing data demonstration implementation. privately and securely must be carried out in a way that does not impact the progress of research and innovation. Categories and Subject Descriptors C.2.4 [Distributed Systems]: decentralized systems; E.3 Structural health monitoring of buildings is important in [DATA ENCRYPTION]: Public-private key encryption damage detection [6], [7] [3] and [4]. More specifically, dam- age to building components or systems may be observed as changes in materials or the geometrical properties of a sys- General Terms tem. Additionally, a building’s health can be assessed by Application monitoring and detecting damage over time. It is also im- portant to provide an explanation as to why the damage Keywords occurred. With the developments and progress in readily DLT, Blockchain, Sensor network, Structural health moni- available and affordable sensors, data collection has become toring more widespread. Storing and processing sensor data in a functioning building can also reveal sensitive and personal 1. INTRODUCTION information about building users. While this poses a privacy Storing and processing data has become very important. and security risk, it supports an emerging and discovering Technological advances in hardware have enabled high ca- field of research: discovering behavioral patterns in building pacity storage and data processing. In recent years, the usage and user health monitoring. Besides monitoring the ideology of an information society where information be- user’s health, there is a growing need to design and imple- comes a financial instrument has matured substantially in ment smart solutions for buildings, focused on elderly peo- both the private and public sectors. Small and large com- ple. These are just examples a few amongst many use cases panies have built business models monetizing information where storing and processing building and user data can be obtained from processing data. The ability to extrapolate a beneficial. deeper understanding of data has been a focus in academic Logging data-points from a large quantity of sensors and storing it safely is predominately and engineering task. How- ever, exploring architectural patterns and design principles is important to study the security and availability of data. Generally, such systems are implemented as centralized data- bases handling data integrity and storage, while peripheral devices are connected to the database directly through any medium of digital communication. However, centralized sys- tems inherently represent a single point of failure. Another 431 downside is that these systems require users to trust a cen- The absence of a central server or entity requires a peer-to- tral entity by losing control over their data and transparency. peer network of nodes that form a distributed system. In A broader view of these problems lies in the nature of digi- general, each node keeps the current state of the ledger. Any tal information. If digital information is passed to someone, change in the state of the ledger is propagated to all nodes. the owner loses control as the information can be copied or To ensure replication, a consensus algorithm is needed. The altered. In other words, as soon as access is granted, the algorithm’s properties are predominantly inherited by the data can be copied, processed and shared without the user’s specific implementation. However, they in general address consent or knowledge. Public-private key encryption tech- the following problems. niques solve this but again require users to trust a central authority issuing certificates (certificate authority) and the underlying implementation of software that facilitates the • All nodes controlling an instance of the ledger need to service. Ideally, a system for storing and processing data come to an agreement on the order of transactions and would be transparent, trustless and could uniquely identify commit the transactions to the ledger in that order. digital information. • Decide on the state of the ledger when two or more con- flicting states are being propagated through the net- Designing a system where data can be safely stored and pro- work cessed without the possibility of copying or sharing (without consent) is challenging. In this paper we explore the poten- • Secure the protocol against deliberate or accidental tial of distributed ledger technology (DLT) with a specifi- malicious actors. cally designed architecture that aims to solve the aforemen- tioned problems of centralized systems while still maintain- ing the same functionality, availability and security. One of the most well-known protocols that has addressed those issues is Bitcoin, a type of DLT called a Blockchain. There are now many variations of the protocol experiment- 2. REAL WORLD IMPLEMENTATION ing with different consensus algorithms that provide interest- This review and research of existing DLTs is inspired by a ing properties. A Blockchain is simply a distributed and de- project at the InnoRenew CoE. The research institute fo- centralized database that has mathematically provable prop- cuses on sustainable building with renewable materials, hu- erties addressing the problems mentioned above. These prop- man health in the built environment and ICT integration erties make it suitable for maintaining a digital currency or buildings, manufacturing and products. A new building will value transfer. The Bitcoin protocol enables participants in be constructed to house the InnoRenew CoE staff and re- the network to transact with one another using Bitcoin as a search facilities. In addition to its primary function, it is digital currency. The consensus algorithm, usually refereed also a demonstration project for sustainable building with to as Proof of Work, solves the aforementioned problems. wood and it will include various sensors that will give more Private-public key encryption is used for security and it has insight into how materials and components perform, how some interesting and unique properties, which we address spaces are used and the overall health and performance of later. Another successful implementation of a Blockchain is the building. A sensor network of this scale requires careful Ethereum, which added support for Smart Contracts (SC). examination to identify the appropriate technology and im- SCs are in essence programs that can be executed in a dis- plementation constraints. One of the key attributes of sensor tributed system. The result of the computation is decided networks is to store sensitive data, including user data, while through the consensus protocol. These programs have the having the ability to process and analyse data for research ability to send transactions and store data. Lately, most re- purposes. The unique properties of DLTs and distributed search and innovation related to consensus algorithms and computing could meet those requirements and at the same distributed system shifted focus to explore Blockchain tech- time maintain a high level of security, availability and scal- nology. Arguably, the interest spiked due to an abundance ability. In chapter 3 we identify the properties required and of investments and business opportunities in this emerging review existing platforms that would potentially be suitable technology. Due to the rapid expansion of DLT, different for real-world implementation. protocol implementations and use cases, a careful review is needed to choose a suitable platform. 3. REVIEW In this chapter we review various existing DLTs that enable Our review includes DLT systems specifically designed to storing, analyzing and processing data in a completely trust- tackle the Internet of Things (IoT) systems. The desired less way. We structure this chapter so that it provides some functionalities required are: background on DLTs. We then explore some important fea- tures of Blockchain technology, a type of DLT, describe some • Storing large amounts of uniquely identifiable data in a of their unique features and explain the functionality they trustless way. Successful implementations of DLT such provide to our system [1], [2]. as Bitcoin have the ability to uniquely identify digital data by unique fingerprints. This is achieved through 3.1 DLTs and Blockchain public private key encryption, which has been used for Interest in distributed ledger technology has spiked signif- years. However, in a completely trustless distributed icantly in recent years. This is mostly due to a few suc- and decentralized system such as Bitcoin, this can be cessful implementations. DLTs are systems that maintain a achieved without trusting a centralized entity. A Bit- distributed database. In such a system, there is no central coin must and can be uniquely identified, otherwise entity that administers or controls the state of the network. participants in the network could create new Bitcoin 432 by simply copying the digital fingerprint of another. connecting the current transactions with two others [5]. Ar- This can be translated to any data which would in guably, the choice of transactions to confirm is very impor- turn prevent copying and manipulating data. tant to prevent transaction starvation and avoid latency. With this data structure, IOTA eliminates the need for min- • Security is an important issue in centralized systems. ers and special nodes that confirm transactions. We do not Storing sensitive data in a central server inherently cre- go into details about the cryptography and transaction con- ates a single point of failure. A compromised central firmation but one important aspect is that it does not require database can be fatal for any centralized system. De- a lot of resources and can be done by most devices. This en- centralized implementation of DLT can solve this by ables the IOTA network to process transactions without fees having all nodes in the system keeping a copy of the as there is no need to incentivise specially designed nodes to current state of the database. In such systems, there is provide security to the network. Additionally, the scalabil- no central entity that malicious actors could attack, in- ity of the network is ensured as having more transactions on stead the whole system must be attacked. This would the network consequently means more transactions are con- require malicious actors to convince participants in the firmed. This makes IOTA the perfect candidate for creating network to change the state of the database through a decentralized sensor network where sensors log data in a consensus. The details of the different types of attack form of a transaction. However, IOTA has received some vectors is outside the scope of this paper, but has been criticism about the possibility of centralizing the network studied extensively. which has not been addressed yet. Another downside is the lack of SCs support, which prevents execution of programs • Data processing is the most important functionality in a distributed and trustless system. the system must provide. DLT implementations have achieved this by introducing SCs. SCs are computer 3.3 Ethereum programs that can be executed by nodes in a dis- tributed system. Through consensus protocols, the Ethereum is a unique platform which enables developers to network comes to an agreement of what the output implement their own protocol on top of Ethereum’s protocol of the program is. The output can be included in the with the use of Turing-complete SCs. The network is a vari- ledger. This important feature can facilitate programs ation of Bitcoin’s network using proof-of-work as to reach that process data without the need for the author to consensus [8]. Miners race to solve a problem, which in- ever access data. Instead, a SC is granted access to volves guessing the result of a problem with some accuracy. the data and writes the output to the ledger. In general the actual problem and the solution is not im- portant. The properties of such problems are. In a proof of • Transparency has been an ever growing problem in in- work model, the problem being solved has to be hard, while formation systems. Even though some systems claim checking if the found solution is correct is trivial. Since the to be transparent about how and which data they pro- only way to find the result is to check all possibilities in a cess, policing this is infeasible due to the nature of dig- brute-force manner, finding the result is also proof that a ital data. A distributed system can provide maximum lot of computational power (work) was put into finding the transparency. All transactions and SCs included in a result. Every time such the result is found by a miner, the distributed ledger can be transparent as all nodes keep network checks if the result is correct. If a winning node is a copy of the current state of the ledger. Any instance found, it gets to include the next block in the chain of blocks. of a ledger can be queried for information about the There are a lot of specifics to each implementation of Proof current state of the system and the entire history of of Work, but in general, the system is secure because trying transactions and SC executions (computation results to cheat it would potentially result in a huge financial loss are stored as a transaction). (electricity and equipment used to mine). Ethereum builds on Bitcoins idea by adding SC support. The ability for ac- tors within a trustless network to execute programs opens In the following subsections we review popular existing im- up a world of possibilities which are still being explored. plementations of DLT and identify subsets of the required However, due to the Poof of Work based consensus, blocks properties they provide. We focus on public permission- get added to the network realistically slowly. Additionally, less Blockchains that at least have a semi-working product. transactions and SC executions are not free, as miners have There are many other projects working on DLTs to support to be payed a reward for executing them. Hence, large scale smart buildings, health monitoring and sensor networks but sensor networks cannot be built on top of Ethereum due to are currently not ready for production use. the financial impact and slow transaction speed. Ethereum plans to update in the coming years, migrating from the 3.2 IOTA PoW consensus to a proof of stake, where instead of min- IOTA is one of the most popular projects aiming to solve ing, validator nodes will validate transactions. To ensure problems in IoT and create a machine to machine economy. security of the network, validators will have to stake coins, However, unlike many other DLT implementation, IOTA which can be taken from them by the network, should they is not a Blockchain. Consequently, the consensus protocol provably act against the network. Eliminating the need for is somewhat unique. IOTA is implemented as a directed miners would decrease block-time and allow the network to acyclic graph refereed to as the Tangle. The construction overcome it’s current scalability issues. of the graph is done by having transactions be vertexes and edges be confirmations of transactions. In a Tangle, each 3.4 VeChain new, unconfirmed transaction must confirm two other un- VeChain is a Blockchain based DLT that digitalizes real as- confirmed transactions. The confirmation creates an edge sets that can be uniquely identified and tracked. The most 433 promising application is arguably supply chain management 6. REFERENCES and verifiability of real assets. VeChain also supports autho- [1] K. Biswas and V. Muthukkumarasamy. Securing smart rization based digital assets, which simply means ownership cities using blockchain technology. In High Performance of digital assets (and, consequently, real assets) can be repre- Computing and Communications; IEEE 14th sented. This is done with the help of SCs that link a digital International Conference on Smart City; IEEE 2nd asset to an account. Whoever owns the private key to the International Conference on Data Science and Systems account, owns the digital asset. The processing of transac- (HPCC/SmartCity/DSS), 2016 IEEE 18th tions and execution of SCs is done by delegates. VeChain International Conference on, pages 1392–1393. IEEE, is built as a delegated proof of stake (DPOS) system where 2016. delegates using the network are granted the right to run a [2] A. Dorri, S. S. Kanhere, and R. Jurdak. Towards an node. The consensus algorithm, named Proof of Authority, optimized blockchain for iot. In Proceedings of the grants stakeholders the right to run a node if they maintain Second International Conference on Internet-of-Things a minimum balance of VET tokens (VeChain’s native cur- Design and Implementation, pages 173–178. ACM, rency) to engage in protocol governance. The network is not 2017. completely trustless as users and applications must trust the [3] C. R. Farrar and K. Worden. An introduction to majority of delegates to act in the favor of the network. In structural health monitoring. Philosophical general, this is solved by choosing delegates that would op- Transactions of the Royal Society of London A: pose one another rather than work together. The details of Mathematical, Physical and Engineering Sciences, SC support are still not known, which makes VeChain un- 365(1851):303–315, 2007. suitable for real life implementation at the time of writing. [4] V. Giurgiutiu. Structural Health Monitoring with Piezoelectric Wafer Active Sensors: with Piezoelectric 4. CONCLUSION Wafer Active Sensors. Elsevier, 2007. The paper presents an overview of the existing DLT tech- [5] B. Kusmierz. The first glance at the simulation of the nologies that could be potentially used as an aggregation tangle: discrete model, 2017. platform for personally identifiable data in the context of [6] A. Mita. Emerging needs in japan for health monitoring building monitoring and ambient assisted living. The chal- technologies in civil and building structures. In Proc. lenges that we face and try to address with the platform Second International Workshop on Structural Health are: Monitoring, pages 56–67, 1999. [7] H. Sohn, C. R. Farrar, F. M. Hemez, D. D. Shunk, D. W. Stinemates, B. R. Nadler, and J. J. Czarnecki. A • fault tolerance (the platform should detect faulty sen- review of structural health monitoring literature: sors), 1996–2001. Los Alamos National Laboratory, USA, • 2003. security issues detection and prevention, [8] G. Wood. Ethereum: A secure decentralised generalised • data security (platform should act as a safe storage transaction ledger. Ethereum project yellow paper, network), 151:1–32, 2014. • data anonymity and privacy • data availability and transparency Selected technologies will be used in a real-life pilot imple- mentation: the new Innorenew CoE building that will incor- porate sensors monitoring building parameters and health parameters of the employees. The pilot represents a true en- gineering challenge (mounting thousands of specialized and multi-purpose sensors with the accompanying equipment), but the true challenge lies in devising, implementing and testing a platform based on DLT that will address the mon- itoring and analytical needs as well as the security and pri- vacy concerns. The building is in the last preparation stages and is scheduled to be built in the next year and a half, the sensor network will be partly built into the building so we expect the full implementation of the presented setting to be fully functional in the months following the inauguration of the building. 5. ACKNOWLEDGEMENTS Authors Tošić and Burnard gratefully acknowledge the Eu- ropean Commission for funding the InnoRenew CoE project (Grant Agreement No. 739574) under the H2020 Widespread- Teaming programme. 434 Prenos projekta “Asistent IJS” na Python 3.6 Jakob Valič Institut “Jožef Stefan” Jamova cesta 39 Ljubljana, Slovenija jakob.valic@gmail.com POVZETEK 3. PRENOS PROJEKTA V prispevku si bomo ogledali, kako smo prenesli kodo projekta Prenos kode asistenta na verzijo programskega jezika Python “Asistent IJS” (asistent) iz programskega jezika Python 2.7 v 3.6 je potreben, ker bo v letu 2020 ukinjena podpora za Python Python 3.6 in opravili nekaj drugih popravkov. Prenos v novo 2.7, projekt asistent pa je dolgoročnejši projekt. Naloge smo se verzijo je bil potreben zaradi ohranitve tehnične podpore lotili s pomočjo knjižnice Py2to3, ki pregleda datoteke s kodo programskemu jeziku. in avtomatično vnese popravke. Naštejmo nekaj razlik med verzijama programskega jezika, ki jih knjižnica Py2to3 Ključne besede upošteva. V Python 3.6 asistent, avtomatično odgovarjanje, Py2to3  je ukinjena uporaba značke 'u' za unicode nize,  branje vrednosti slovarja vrne generator namesto seznama, 1. UVOD  imamo drugačno notacijo za prestrezanje napak, Asistent je projekt, katerega začetki segajo v leto 2013. Dostop  se je knjižnica urllib razdelila na dve knjižnici: do uporabniškega vmesnika projekta se nahaja na spletni strani urllib.request in urllib.error. https://www.ijs.si. Asistent je orodje za izbiro najustreznejšega odgovora na zastavljeno vprašanje v zvezi z Institutom “Jožef Med asistentovimi moduli ni neposredne komunikacije, zato Stefan” (IJS). Projekt je podrobno opisan v [1]. Trenutno smo lahko prenašali vsak modul posebej. Do komentiranih delujoča različica asistenta je napisana v programskem jeziku delov kode in blokov kode, ki so bili shranjeni v podatkovnih Python 2.7. Ker je projekt dolgoročne narave, smo se odločili, bazah asistenta, knjižnica Py2to3 ni imela dostopa, zato smo jih da opravimo prenos kode projekta na verzijo Python 3.6, kateri morali prenesti ročno. je zagotovljena nadaljna tehnična podpora. 4. DRUGE SPREMEMBE Izkoristili smo priložnost, da uvedemo še nekaj drugih 2. STRUKTURA PROJEKTA Asistent je spletna aplikacija, razdeljena na jedro in več med sprememb asistentove kode: seboj neodvisnih modulov. Vsak od modulov je odgovoren za  popravek prijave administratorja, vprašanja z določenega področja ali za določen vidik delovanja  dodajanje dokumentacijskih komentarjev, asistenta. Navzven je asistent spletni servis, ki prejme vprašanje  pisanje testnih primerov. uporabnika in vrne odgovor. Poleg te osnovne funkcije nudi asistent še uporabniški vmesnik za uporabnike in 5. ZAKLJUČEK administratorje. Na projektu asistent ostaja veliko možnosti za stilistično izboljšavo kode in za izboljšanje načina izbire pravega 2.1 Moduli odgovora. Morda je najbolj očitna pomanjkljivost v statičnosti Glavna uporaba asistenta je, da uporabnik vnese vprašanje in izbire odgovorov. Vzorci za iskanje odgovorov so določeni, pritisne tipko Enter. Ob tem se vsak modul odzove s klicem funkcije, ki poišče najboljši odgovor v modulu in ga doda izven njih pa asistent nima dobrega odgovora. v izbor vseh odgovorov. Odgovori so sestavljeni iz besedila Prenesena koda se v času pisanja tega prispevka nahaja v odgovora, uteži odgovora, morebitne povezave na spletno stran ločenem repozitoriju. V bližnji prihodnosti bo postavljena v v ozadju, informacije o tem, kateremu modulu pripadajo in produkcijo. morebitne pripadajoče geste asistenta. Utež odgovora je merilo njegove kvalitete. Primeri modulov so: 6. REFERENCE  modul static_answer_kb za statična vprašanja iz baze [1] D. Kužnar, A. Tavčar, in J. Zupančič. Virtual assistant platform. Informatica, 40(3):285-289, sep 2016. podatkov (pripona kb je okrajšava za ang. knowledge base),  modul ijs_contacts_kb za informacije o zaposlenih na IJS,  modul slovenia_info_kb za glavne informacije o Sloveniji,  modul dont_know_kb za nedefinirane odgovore,  modul qa za izbiro najboljšega odgovora izmed vseh odgovorov, ki so jih v izbor dodali ostali moduli. 435 Časovna sinhronizacija brezžičnih EKG senzorjev Andrej Vilhar Matjaž Depol i Institut »Jožef Stefan« Institut »Jožef Stefan« Jamova 39 Jamova 39 1000 Ljubljana 1000 Ljubljana +386 1 477 3132 +386 1 477 3135 andrej.vilhar@ijs.si matjaz.depolli@ijs.si POVZETEK tudi z vzporedno uporabo več neodvisnih brezžičnih senzorjev V prispevku opišemo in ovrednotimo na Institutu »Jožef Stefan« hkrati. Motivacija temelji na že predlaganih metodologijah, ki razvito metodo za sinhronizacijo brezžičnih EKG senzorjev. potrjujejo, da je sinteza iz treh vzporednih diferencialnih meritev Sinhronizacija je nujen pogoj, če želimo iz istočasnih meritev več EKG mogoča [8]. senzorjev, opravljenih na eni osebi, sintetizirati standardni 12- Cilj tega prispevka je na kratko predstaviti novo kanalni EKG. Rezultati kažejo, da je metoda uspešna in torej metodologijo, ki smo jo razvili za dosego natančne sinhronizacije omogoča tovrstno sintezo. več EKG signalov, merjenih z brezžičnimi senzorji. Natančna sinhronizacija predstavlja nujen pogoj za uspešno sintezo 12- Ključne besede kanalnega EKG. Čeprav je bila metoda razvita s specifičnim EKG senzor, sinhronizacija, linearna regresija, 12-kanalni EKG. namenom sinhronizacije senzorjev EKG pa deluje v splošnem za vse vrste meritev z visoko vzorčno frekvenco, kjer imajo posamezne merilne naprave samostojne nesinhronizirane ure. 1. UVOD Merjenje EKG signalov se je od začetka 20. stoletja do danes močno razvilo in se je uveljavilo v obliki standardnega 2. SINHRONIZACIJSKI ALGORITEM 12-kanalnega EKG. Poleg klasičnih stacionarnih večkanalnih Predlagani algoritem poskuša sinhronizirati prejete vzorce EKG sistemov [1] so v uporabi tudi prenosni monitorji (tipični tako, da doseže pravilno reprodukcijo časovnih trenutkov, pri predstavnik je Holterjev monitor), ki že od leta 1960 omogočajo katerih so bili vzorci dejansko vzeti. Ura vozlišča, t.j. pametnega dolgotrajno (okvirno en cel dan) spremljanje EKG [2]. telefona, služi kot referenca. Tehnološki napredek je pozneje omogočil, da so prenosne Na vhodu algoritem dobi vzorce, označene s številko števca naprave postajale vse manjše in so za snemanje in časovnim žigom. Števec je monotono naraščajoče število, ki visokokakovostnih EKG signalov uporabile manjše, a še ga vodi senzor. Predpostavljamo, da so števci pravilni in da so zadostno število elektrod. Prenos posnetkov lahko poteka preko bili predhodno že opravljeni morebitni popravki preliva (angl. brezžičnega kanala do vozlišča z dostopom do Interneta. Opisana overflow), ki so nujni zaradi omejene preciznosti števil, s konfiguracija omogoča zagotavljanje širokega spektra mobilnih katerimi števce shranjujemo oziroma prenašamo. Časovne žige zdravstvenih storitev, od spremljanja bolnikov v bolnišnicah [3], zapiše vozlišče v času sprejema na aplikacijskem nivoju (ker se oddaljenega spremljanja bolnikov [4][5] in oddaljene medicinske le-to ne zgodi na nivoju gonilnikov, do katerih uporabniške podpore do športa, rekreacije in zabave. aplikacije nimajo dostopa). Na voljo so torej naslednji podatki: cnt_sensor (n) in t_hub (n) za n ε [1, N], kjer je N število poslanih Na Institutu »Jožef Stefan« je bila razvita večfunkcionalna podatkovnih paketov. senzorska naprava EKG, ki je del mHealth platforme [6]. Naprava obsega analogni EKG senzor, mikrokontroler in radijski Na izhodu algoritem določi nove, popravljene časovne žige, oddajnik Bluetooth Low Energy (BLE). Ima zelo nizko označene kot t_new (n). procesorsko zmogljivost in nima zmogljivosti shranjevanja, Algoritem se izvaja v štirih korakih: zaradi česar je majhna in energetsko učinkovita. Nosi se jo na prsih, kjer meri EKG in ga v realnem času brezžično pošilja na 1. Razčlemba meritev. pametni telefon. 2. Linearna regresija s filtriranjem paketov. Enostavnost naprave po eni strani omogoča, da je majhna, nemoteča in kot taka lahko nosljiva v vsakodnevnih dejavnostih, 3. Odprava nezveznosti. po drugi strani pa zaradi svoje enostavnosti prelaga več 4. Določitev časa vzorčenja. tehnoloških izzivov na stran pametnega telefona oziroma druge naprave, ki služijo urejanju prejetih podatkov v koherentno Korak 1: Razčlemba meritev. V tem koraku želimo razdeliti meritve v dobre in slabe bloke, kjer imajo slednji težko meritev. Glavni izziv je celovitost podatkov, ki je lahko ogrožena zaradi netočne ure na strani senzorja, naključnih zakasnitev v nadomestljive pomanjkljivosti, kot so večja izguba paketov, prenosu, ki so posledica brezžičnega protokola, in po naših daljši premor v prenosu paketov ali močno zapozneli paketi. Vse izkušnjah tudi zaradi nihanja urnega takta na pametnem telefonu. od naštetih pomanjkljivosti je mogoče prepoznati le z analizo časov prejetih paketov, to je iz daljše časovne razlike med Izziv je potrebno obravnavati že na ravni enega samega zaporednimi prejetimi paketi. EKG senzorja [7]. Naša motivacija pa je, da sistem razvijemo do te stopnje, da pravilno združimo meritve iz več EKG senzorjev, Druga naloga tega koraka je frekvenčna razčlemba, ki dobre ki merijo srčno aktivnost istega človeka in so priključeni na isti bloke nadalje razdeli na manjše bloke vnaprej določene največje dolžine. Motivacija je izboljšati natančnost ocenjevanja frekvenc pametni telefon. Cilj takega združevanja je pridobiti zmožnost sinteze klasičnega 12-kanalnega EKG, ki bi bil tako dosegljiv 436 vzorčenja, ki bo potekala v naslednjem koraku, saj lahko le-ta Vsak od signalov, prejetih na pametnem telefonu, je med samo meritvijo opazno niha. podvržen procesu sinhronizacije. Rezultat sinhronizacije nato primerjamo medsebojno, glede na absolutne časovne razlike, Korak 2. Linearna regresija s filtriranjem paketov. Na izračunane z iskanjem vrhov treh signalov in primerjavo njihovih vsakem posameznem dobrem bloku opravi algoritem postopek časov. Vrhove signalov poiščemo z uporabo kvadratne enostavne linearne regresije. Postopek je iterativen in ob vsaki interpolacije, kar zmanjša napako algoritma za iskanje vrhov. iteraciji je po postopku linearne regresije določena linearna Dobljeni rezultat je funkcija časa in služi kot metrika za oceno funkcija, ki ima najmanjšo vsoto kvadratnih razlik do uspešnosti sinhronizacije. pripadajočih vzorcev. Na koncu vsake ponovitve se izvaja filtriranje paketov. Njegov namen je odstraniti manj zanesljive pakete z veliko zamudo. Vsi paketi, katerih vrednost je nižja od vrednosti določene z regresijsko funkcijo, se odstranijo pred izvedbo naslednjega ponovitvenega koraka. Po končanem iterativnem procesu določitve linearne funkcije se opravi zadnja naloga tega koraka – določanje začetne vrednosti. Ta je določena glede na vzorec z največjo (pozitivno) Slika 1. Shema istočasnega merjenja umetnega signala napako med preostalimi vzorci. Z drugimi besedami, začetna EKG s 3 senzorji. vrednost je nastavljena po najzgodnejšem vzorcu glede na linearno funkcijo. Postopek zagotavlja, da so ocenjeni časovni 3.1 Rezultati žigi nižji ali enaki izmerjenim časovnim žigom, saj vemo, da . Rezultati analize kažejo, da v primeru, ko meritve niso izmerjeni časi ne morejo biti nižji od dejanskega časa vzorčenja. sinh ronizirane, povprečna časovna razlika merjenih vrhov znaša 12,7 ms s standardnim odklonom 10,2 ms. Vrednosti se znatno Korak 3. Odprava nezveznosti. Da bi odpravili znižajo, ko se uporabi sinhronizacijski algoritem. Srednja nezveznosti med deli funkcije, ki jih lahko med bloki dobimo po vrednost je v tem primeru le 0,49 ms, standardni odmik pa 2. koraku, v 3. koraku konsolidiramo meje blokov. 0,39 ms. Glede na postopek 2. koraka slabi bloki vedno ležijo le ob Za ponazoritev so prikazani povprečni primeri izmerjenih dobrih blokih, dobri bloki pa so lahko bodisi zraven dobrih bodisi generiranih signalov EKG na sliki 2. Časovne razlike med vrhom zraven slabih blokov. Zato je meja med dvema sosednjima v nesinhroniziranem primeru so 12,9 ms in 13,3 ms za zeleno- blokoma, t.j. zadnjim paketom enega bloka in hkrati prvim modri in zeleno-rdeči par, medtem ko je zakasnitev med vrhovi paketom drugega bloka, lahko bodisi med dobrim in slabim za modro-rdeči par 26,2 ms, kar presega napako treh vzorčnih blokom bodisi med dvema dobrima blokoma. obdobij. Po drugi strani so pari sinhroniziranih signalov vizualno Konsolidacija poteka tako, da za vse možne primere velja skoraj popolnoma poravnani, njihova časovna razlika je med eno od dveh pravil, kot sledi. 0,2 ms in 0,6 ms. - Pravilo 1. Če leva stran nima ocenjenega časa (slab blok), se zadnji točki v njenem območju dodeli vrednost prve točke desnega območja. - Pravilo 2. Če desna stran nima ocenjenega časa (slab blok), potem se prvi točki v njenem območju dodeli vrednost zadnje točke levega območja. Če imata obe strani čas ocenjen (dva dobra bloka), se uporablja pravilo 2. Korak 4. Določitev časa vzorčenja. V zadnjem koraku se ocenjene frekvence vzorčenja, določene na vsakem od signalnih blokov, uporabijo za izračun vzorčnega časa vseh paketov. Za vsak paket je čas vzorčenja določen kot: 𝑐𝑛𝑡 (𝑛)−𝑐𝑛𝑡 Slika 2. Del generiranega EKG signala, merjenega s 𝑡 0 𝑛𝑒𝑤 (𝑛) = , 𝑓(𝑡) tremi senzorji. Levi graf: nesinhroniziran primer. Desni graf: sinhroniziran primer. kjer je cnt(n) vrednost števca za ta paket, medtem ko sta cnt0 in f(t) začetna vrednost in nagib (smerni koeficient) paketnega 4. ZAKLJUČEK bloka. Postopek sinhronizacije bre . zžičnih senzorjev mora biti siste matičen in mora ustrezno rešiti dve glavni vprašanji, (i) 3. VREDNOTENJE ALGORITMA negotovost in zakasnitve na komunikacijskem kanalu ter (ii) nihanje ure (takta) senzorjev in pametnega telefona. Algoritem smo ovrednotili v laboratorijskem okolju z umetnim signalom EKG, generiranim z napravo, izvedeno na V članku predlagana metoda problem rešuje učinkovito in razvojnem modulu Arduino. Simulirani signal (stilizirani QRS natančno. Rezultati kažejo na to, da bi s takšno sinhronizacijo kompleks iz realnega EKGja) ima obliko, podobno eni periodi zagotovili vse pogoje za potrebe pravilne sinteze 12-kanalnega sinusnega vala, ki ji sledi enosmerni signal. Opisani vzorec se EKG iz signalov, merjenih vzporedno z več samostojnimi ponavlja periodično enkrat na sekundo. V eksperimentu brezžičnimi senzorji. uporabljeni senzorji EKG so merili signal s približno frekvenco vzorčenja 125 Hz. Vsi senzorji so povezani z enim samim 5. REFERENCE vozliščem (Android-napravo) prek radijskega protokola [1] Trobec, R. 2003. Computer analysis of multichannel ECG . Bluetooth LE. Konfiguracija je prikazana na sliki 1. Comput. Biol. Med. , 33, 3, 215–226. 437 [2] Janousek, J. 1989. Various systems of long-term ECG [6] Depolli, M., Avbelj, V., Trobec, R., Kalisnik, J. M., (Holter) monitoring in clinical practice. Cas. Lek. Cesk., Korosec, T., Poplas Susic, A., Stanic, U. and Semeja, A. 128, 14, 429-43. 2016. PCARD platform for mhealth monitoring. [3] Bifulco, P., Cesarelli, M., Fratini, A., Ruffo, M., Informatica. 40, 1, 117–124. Pasquariello, G., and Gargiulo. G. 2011. A wearable device [7] Vilhar, A. and Depolli, M. 2017. Synchronization of time in for recording of biopotentials and body movements In wireles ECG measurement. In Proceedings of the 40th Proceedings of the IEEE International Workshop on Jubilee International Convention MIPRO (Opatija, Croatia, Medical Measurements and Applications (MeMeA), 469– May 2017), 311–315. 472. [8] Tomašić, I. and Trobec, R. 2014. Electrocardiographic [4] Pantelopoulos, A. and Bourbakis, N. G. 2010. A Survey on systems with reduced numbers of leadssynthesis of the 12- Wearable Sensor-Based Systems for Health Monitoring and lead ECG. In IEEE Reviews in Biomedical Engineering. 7, Prognosis IEEE Transaction on Systems, 40, 1, 1–12. 126–142. [5] Lindén, M. and Björkman, M. 2014. Embedded sensor systems for health - providing the tools in future healthcare Stud. Health Technol. Inform. 200, 161–163. 438 Integration of Oncology Information System with Proton Therapy Software and Think!EHR Platform M. Vitorovič, Ž. Kroflič, J. Bobnar, A. Smokvina R. Savadkouhi Marand d.o.o. Cosylab d.d. Koprska ulica 100 Gerbičeva ulica 64 SI-1000 Ljubljana, Slovenia SI-1000 Ljubljana, Slovenia +386 1 470 31 00 +386 1 477 66 76 info@marand.si info@cosylab.com ABSTRACT Think!EHR platform. The role of an OIS is to delegate and Proton therapy uses a beam of high-energy protons accelerated in manage the workflow of a PT system. This is done through task a particle accelerator to treat cancer. Treatment plans are created assignments and progress reporting. Within the study of the OIS for each patient individually and are uploaded to the Oncology system architecture we laid out a basic treatment plan with a Information System (OIS). The OIS forwards this information to specified data structure, defined the patient registration procedure the Proton Therapy (PT) software system. When the treatment is and created the final treatment report. complete, a treatment report is generated detailing treatment We defined the acquisition, processing and final display of patient specific information. The PT system uploads this report to the data in a graphical interface. The processing and structure of OIS. To connect the OIS to an external platform such as the imaging data for medical software is described by the Think!EHR [2] it is critical for the PT system to be able to international DICOM standard [1] (Figure 1). communicate with the external API of the platform. This article describes the design of a Oncology Information System simulator and the integration of the Think!EHR platform, into a PT software system using the EHR service software component. Keywords oncology information system, proton therapy, DICOM, OIS simulator, Think!EHR, EHR service 1. INTRODUCTION Cosylab provides system integration and customer-adapted products and solutions covering the complete area of control systems and instrumentation. We specialize in accelerators both for scientific research and particle therapy, especially in the area of Proton Therapy machines. A complete Proton Therapy software system is composed of a Motion Control System (MCS), a Treatment Control System (TCS) and a System of Safety (SOS). The MCS controls movements of the patient couch, the gantry (for directing the beam to the patient), the X-ray imaging system (for patient scanning) and the translation slides (for inserting patient specific devices). The TCS controls the accelerator that accelerates protons to a certain energy, it checks the delivered dose and Figure 1: General DICOM communication model [1] manages the data storage that stores patient and machine related data. The SOS controls a set of PLCs (programmable logic controllers) connected to multiple PT hardware components and 3. OIS SIMULATOR room sensors. The SOS regularly checks the status of these An important research field in our work is the architectural design components and reports issues to the TCS. of an OIS simulator, which is adaptable and can be used to mimic an OIS system. We laid out the design of OIS simulator (OISSIM) and specified the compliance of the simulator to the DICOM 2. ONCOLOGY INFORMATION SYSTEM standard. The primary goal of OISSIM is to simulate an OIS that Cosylab is a partner in the Ekosmart project in the domain of IoT is being used at a hospital and provide a controlled environment health platforms. We analysed the architecture, data structure and where OIS-dependent applications such as the software used in communication standards of an Oncology Information System proton treatment can be tested. The client can request a list of (OIS) and its integration with a proton therapy system and the tasks, decide to perform one or several of them and report back on 439 the status of the performed tasks (Figure 2). With such a pass data from the proton treatment system to the Thin!EHR simulator, proton therapy software can already be tested during its platform based on the OpenEHR standard. With this component development phase. we establish a data flow from the OIS to the Think!EHR platform [2] via a PT software system. OISSIM supported functionalities: In collaboration with Marand we analysed the data structure and • worklist retrieval: an AE can retrieve a worklist, i.e. communication protocol of the Think!EHR platform and a list of UPSs, by sending a C-FIND request to the specific OpenEHR data model was created to accommodate data simulator. The OIS simulator then processes the request generated by the OIS. We analysed the communication standard and returns the UPSs specified by the query. the platform uses and took this into account in the development of • claim UPS: before the AE can perform work on behalf the EHR service. The service reads data from the external control of a UPS it has to claim it, i.e. the AE commits to system (ECS), which handles communication between the PT performing the work specified by the UPS. This is done software system and externally connected systems, e.g. the OIS. by sending an N-ACTION request to move the UPS The Data Distribution Service (DDS) protocol is used in all state from “SCHEDULED” to “IN PROGRESS”. communication. Once the EHR service connects to the ECS service, algorithms for filtering out patient data are used to • update UPS: the AE can modify the attributes of the retrieve targeted information. We also defined a template data set UPS by sending an N-SET request with new values. that should be uploaded to the Think!EHR platform. A basic • close UPS: after work on the UPS is completed, the AE workflow of the EHR service is shown in Figure 3. has to close the UPS. This is done by sending an N- ACTION request to move the state of the UPS from “IN PROGRESS” to “COMPLETED” or “CANCELED”. Figure 3: 4-step EHR workflow 5. ACKNOWLEDGMENTS The work was carried out in the framework of the EkoSmart programme, which is partially financed by the Republic of Slovenia – Ministry of Education, Science and Sport and the European Union – European Regional Development Fund. Figure 2: OISSIM application data flow 6. REFERENCES 4. EHR SERVICE [1] International DICOM standard: http://dicom.nema.org As part of the RRP6 Ekosmart project section (testing and [2] Think!EHR platfrom: http://www.ehrscape.com validation of prototypes) we are developing the EHR service component that integrates into a PT software system and is able to 440 Training of precise movements in 3D or 2D for persons with Parkinson’s disease? Imre Cikajlo Zlatko Matjačić Helena Burger Karmen Peterlin Potisk University rehabilitation University rehabilitation University rehabilitation University rehabilitation institute institute institute institute Linhartova 51 Linhartova 51 Linhartova 51 Linhartova 51 SI-1000 Ljubljana SI-1000 Ljubljana SI-1000 Ljubljana SI-1000 Ljubljana +386 1 475 8 150 +386 1 475 8 150 +386 1 475 8 150 +386 1 475 8 150 imre.cikajlo@ir-rs.si zlatko.matjacic@ir-rs.si helena.burger@ir-rs.si karmen.potisk@ir-rs.si ABSTRACT designed the VE for the use with LCD screen/laptop and with The 10Cubes for training and assessment of precise movements the 3D goggles (Oculus Rift CV1, Oculus VR, LCC, with upper extremities has been developed for 3D virtual USA). reality. The system comprises of an infrared stereo camera for hand/finger movements detection and a 3D head mounted device. In the small scale study 13 persons with Parkinson’s disease participated. The participants were randomized into 2 groups; one using a laptop and the other using a 3D head mounted device. The 2-week protocol with 10 sessions, each lasting for 30 min revealed that participants improved some functions by clinical means besides the successful game performance. The kinematics of the hand movements and the outcomes of the clinical test Box & blocks improved for both groups. 1. INTRODUCTION Parkinson’s disease (PD) is a progressive degenerative disease of the extrapyramidal system [1]. The disease often affects Figure 1. 10Cubes 3D: What are the advantages of the people at the age between 35 and 60 years. The following 3D virtual exergaming for persons with PD at home disorders can be observed: rigidity of muscles, slowness of based physiotherapy or telerehabilitation? movements (bradykinesia), tremor and various abnormal uncontrolled movements (dyskinesia). Physiotherapy at all stages may help to maintain certain level of quality of life In the study 13 persons with Parkinson’s disease were involved without changing the dosage of medicine. However, only some (67y, 7y after the first symptoms). The participants were reports support the statements on successfulness of randomized into 2 groups; one using the Oculus Rift 3D (5 physiotherapy [2] [3]. males, 2 females) and the other using a laptop (4 males, 2 females). The study was approved by local ethics committee In the study we hypothesized that persons with PD can improve and all participants gave a written consent. We examined the their fine motor skills using the 10Cubes3D application kinematics of the hand movement [4] and the changes in the regardless of the type of the visual equipment. clinical test Box & blocks. 2. METHODOLOGY 3. RESULTS Virtual environment (VE) for pinch and grip was created in The participants demonstrated higher score at Box & Blocks Unity3D (Unity Technologies, CA, USA). The dedicated space clinical test in both groups (4/2, 4/3): with simulated grass floor was limited with hidden walls and a model of a treasure box. In the middle of the dedicated space 10 BBT affected non-affected cubes with the same physical model, size, virtual weight, bounce stiffness, material, etc., were placed. The goal of the LCD 4/2/0 5/1/0 task was to grab and put all the cubes into the treasure box, one- by-one with the virtual hand. The virtual hand was a VR avatar 3D 4/3/0 4/1/2 presentation of the participant’s hand that was tracked in real- time by a mini camera (Leap Motion Controller, Leap Motion Inc., CA, USA). The camera tracked hand and fingers position, both required for construction of the 3D VR hand motion (Fig. 1) and the participant can view his/her hand in the VE. We designed an environment for left and right handed participants; the right handed grabbed the cube with the right hand and put it in the box on the left side of the VE and vice-versa for the left handed participants. The software for the assessment and control was written in C# using Leap Motion libraries. We 441 5. ACKNOWLEDGMENTS The authors would like to acknowledge the financial support to the Republic of Slovenia and the European Union under the European Regional Development Fund (EkoSMART) and Slovenian Research Agency (project MIO-A J2-7357, program P2-0228). 6. REFERENCES [1] Melnik M.E. 1995. Basal ganglija disorders. In : Umphred DA ed. Neurological rehabilitation. 3rd ed. St. Louis: Mosby, 606-636 [2] Clarke CE, Patel S, Ives N, Rick CE, Dowling F, Woolley Figure 2. The mean Box & blocks score improved (more R, et al. Physiotherapy and Occupational Therapy vs No boxes collected, more points) for both groups of Therapy in Mild to Moderate Parkinson Disease. JAMA participants for affected and non-affected hand. Neurol [Internet]. 2016 Mar;73(3):291. Available from: http://www.ncbi.nlm.nih.gov/pubmed/26785394 [3] Cikajlo I, Hukić A, Dolinšek I, Zajc D, Vesel M, The kinematic analysis demonstrated that the laptop group Krizmanič T, et al. Can telerehabilitation games lead to gradually performed faster, more efficient (inserted more cubes) functional improvement of upper extremities in individuals with less tremor after the training and the 3D group was indeed with Parkinson’s disease? Int J Rehabil Res [Internet]. more successful in speed performance and efficiency, but with 2018 May [cited 2018 Sep 26];41(3):1. Available from: significantly more attempts and higher tremor. http://www.ncbi.nlm.nih.gov/pubmed/29757774 4. DISCUSSION [4] Barry G., Galna B. and Rochester L. 2014. The Role of Exergaming in Parkinson's Disease Rehabilitation: A+A In the preliminary study with only few participants with Systematic Review of the Evidence . J Neuroeng Rehabil. Parkinson’s disease we cannot confirm any major differences 7, 11- 33. between the two applied approaches. Both groups performed well, improved their functional pick and place tasks in the [5] Fisher A.G. 2003. Assessment of Motor and Process Skills: virtual environment and in the real environment as Volume I – Development, Standardization, and demonstrated by the clinical test. During the task we recorded Administration Manual Fifth Edition. Three Star Press, several unsuccessful trials like misplacement of the cube, cubes Inc.-Fort Collins, Colorado USA falling out of the hand, causing tremendous hand tremor and [6] Goršič M, Cikajlo I, Novak D. Competitive and other measurable components supported by literature [6]. cooperative arm rehabilitation games played by a patient Even if we did not confirm the superiority of the 3D technology and unimpaired person: Effects on motivation and exercise over 2D, we would assume that the virtual cube game has intensity . J Neuroeng Rehabil [Internet]. 2017 Dec 23 enormously increased motivation and thus the participation of [cited 2017 Apr 11];14(1):submitted for publication. the subjects. The motivation of the participant can play an Available from: important role [6]. http://www.ncbi.nlm.nih.gov/pubmed/28330504 442 Forecasting the physical fitness and all-cause mortality based of schoolchildren’s fitness measurements Matej Cigale Anton Gradišek Miha Mlakar Jožef Stefan Institute Jožef Stefan Institute Jožef Stefan Institute Jamova cesta 39 Jamova cesta 39 Jamova cesta 39 Ljubljana, Slovenija Ljubljana, Slovenija Ljubljana, Slovenija matej.cigale@ijs.si anton.gradisek@ijs.si miha.mlakar@ijs.si Mitja Luštrek Jožef Stefan Institute Jamova cesta 39 Ljubljana, Slovenija mitja.lustrek@ijs.si ABSTRACT prediction of fitness for children based on previous years is The focus of medicine is steadily shifting from curing the sick quite hard. Several approaches were evaluated, but at this to preventive measures. In order to assist the policy makers time linear regression seems to provide the best results, al- in making the right decisions that would lead to a healthier though research on creating better models is still ongoing. population, there is an increasing need to develop models Since data on the health risks for the subjects in the SLOFit that can forecast the state of the population in the future, dataset is not available, we use risk calculation based on the check what measures are effective and what policies synchro- literature to calculate general mortality models based on nize. In order to track these changes, predict the state of the certain fitness indicators. As the data stored in the system population in the future, and thus make informed decisions, and its applications is complex, the standard used must en- the CrowdHEALTH platform is developed. The SLOfit use able flexible storage of information. The CH infrastructure case takes the information collected from a large population uses Fast Healthcare Interoperability Resources (FHIR) [3] of school children and aggregates this to provide informa- as the standard for data storage, meaning that all data can tion on the future health of the population that is just now be queried in a similar manner, and if the appropriate infor- finishing school. mation is available, compared and forecasts generated. Keywords The rest of the paper is organized as follows. Section 2 pro- vides an overview of the SLOFit data set. Section 3 provides obesity, fitness, exercise, machine learning the information on the architecture of the Forecasting An- alytical tool and places it in the context of the CH system. 1. INTRODUCTION Section 4 discusses the forecasting algorithm. Section 5 pro- The focus of medicine is steadily shifting from curing the sick vides the outline of the Risk assessment and finally Section to preventive measures[8]. With people’s growing desire to 6 discusses the results. increase their lifespan and health, there is an ever greater push for the policy makers to provide ways for people to increase or maintain their fitness. In order to forecast what 2. SLOFIT DATASET the population health will be in the years to come, research SLOfit is a massive cohort study of physical fitness of Slove- looks to the machine learning algorithms that can generate nian schoolchildren. Every April, almost all elementary and models predicting the trends in specific populations. The high school students undergo measurements of 3 anthropo- focus of the CrowdHEALTH (CH) project is to gather this metric tests (height, weight, triceps skinfold) and 8 motoric kind of information in a consistent way across multiple data tests, aimed at monitoring different components of physical sources and generate models that can be used to predict fitness (such as cardiorespiratory fitness, muscular fitness, what the effects of implementing health policies will be on explosivity, agility, coordination, etc.) The SLOfit study has a population. been ongoing on the national level since 1987 and serves as the scientific backbone for most of policies related to physi- SLOFit is a large study on physical fitness that includes data cal education in schools and enhancing of physical fitness in collected from Slovenian schoolchildren for over 40 years, schoolchildren. To date, the SLOfit database includes over and is used to chart global health trends in the population. 7 million sets of measurements for over 1 million children, This data set was used as the basis for our modeling so that being one of the largest cross-sectional and cohort databases we could predict the state of the population in the future of physical and motor development in the world. (when they are grown) and calculate what the associated risks for mortality will be in the future. In the course of In our study on forecasting of physical fitness, a subset of the the project we investigated several models that can be used SLOfit data was used, encompassing the data from aproxi- to predict the state of the population and, as expected, the mately 2000 children from the age of 6 to 18. In the analysis, 443 the data was anonymized, retaining only the municipality- level data in order to be able to create policies on regional level. When assessing risks, we focus on a subset of SLOfit parameters that are directly connected to the risks we are interested in. Height and weight are used to calculate the body-mass-index (BMI), which is used to determine whether a person is overweight (obesity) or underweight. 30 s sit-up results are used as a proxy for muscular fitness (MF), while the 600 m run results are indicative of cardiorespiratory fit- ness (CRF). In the risk analysis, we are currently focusing on all-case early mortality risks while risks for developing cardiovascular diseases (CVD) or diabetes are planned to be looked at in future. 3. ARCHITECTURE OVERVIEW The data in the CH project is stored following an extension of the FHIR standard, where each measurement is stored as an observation that includes all the meta-data of the mea- surement, such as when it was taken, by whom, what are the units of the measurement, etc. The current architecture of the CH system is demonstrated in Figure 1. This enables the overall system to be extended in the future with custom tools. The data is stored in LeanXcale (LXC) [1], a flexible, ultra- scalable database with analytical capabilities. In order that the information of different types can be stored, a special- ized schema was developed. The part that is pertinent to Figure 1: The architecture and flow of the applica- our work was the addition of a new Person class - Student, tion. to differentiate it from Patient that is the general subject in FHIR. Additionally, the metadata for schools, municipalities and regions were added. The 11 standard anthropometric As the starting point, we defined two baseline forecasting ap- and motoric tests were also codified in the system so that proaches. The first one, called baseline percentiles, uses the they can be easily accessed. In order to speed up the queries, percentile method: if an individual is in the n-th percentile the Forecasting module we developed includes a small inter- at the age of 13, we assume he would be in the same per- nal database that caches the data. This is facilitated by centile at the age of 18. The second baseline model, called SQLite with schema that mimics the data stored in LXC baseline average growth, uses the current value and adds the system - i.e., the region, municipality, school, student and average growth values for each year until the age of 18. observation classes that include most of the data stored in the LXC system. This provides faster look-up times and More advanced approaches use machine learning. To im- simplifies some filtering, as SQLite can be tightly coupled prove the prediction accuracy, we generated additional fea- with the Django service. tures, such as average, maximum and minimum year growth, standard deviations, data percentiles and peak height veloc- Django is a framework that enables the creation of web APIs ity - PHV (the year with the maximum growth). Since PHV in Python. It consists of three main parts. Django Models was not notated in data, we had to estimate it. We manu- are mapped directly to supported databases, allowing for ally annotated it for the small amount of children and then fast and efficient filtering and querying of the system. The trained a prediction model for PHV on this data. We used developer can specify the DB schema and provide rules to this model to predict PHV on all other data. These pre- check if the values are correct, serialize the data and link dictions are not 100 % accurate, but this information as an the tables in several ways. Djanogo Models are specification input for predicting e.g. height improves predictions as will that maps directly to the DB schema, that the Framework be seen in the results. actually creates for itself, and also handles creation of queries to the system. The Django Views are where the processing Next, we built a model for each year up to which we have of the data happens. Each request can be handled here and available data. For example, the model for the age of 13 responded to accordingly. Django Templates are the presen- takes the measurements from ages 6 to 13 and forecasts the tation layer of the system, but are not used in the current value at the age of 18. Since we have data from 6 to 18 implementation as this is handled by outside systems. years, we build 12 models for each SLOfit parameter. 4. FORECASTING Additionally, we enriched each SLOfit parameter data with The task of the forecasting algorithm is to predict a partic- additional data from another parameter. For predicting the ular SLOfit parameter (height, weight, sit-ups, 600 m run) height, we also used weight, for predicting weight, we also at the age of 18, based on the data from previous years and used height, for predicting sit-ups we also used results from knowing the general population trends. the 600 m run, and when predicting the 600 m run, we also 444 used data from the 60 m run. Several machine-learning algorithms were tested on the data set of 2000 children introduced in Section 2. To evaluate and compared them, the average absolute error was calculated for each years’ predictions and then the average error over all the years. This average error over all the years for predicting the height is presented in Table 1. Table 1: Comparing algorithms when predicting height. Method Average error [mm] Figure 4: Prediction error for each year when pre- Baseline percentiles 36.0 dicting time running 600 meters at the age of 18. Baseline average growth 34.3 Linear regression 27.5 Decision tree 38.9 Logistic regression 41.2 SVM 52.3 As we see, the best results were obtained using a linear re- gression model. Very similar results were obtained also when predicting other SLOfit parameters. The average errors for each year for linear regression and baseline models are shown in Figures 2–5. We see that predicting the values at the age of 18 is a hard problem, so the errors start to decline just a few years before this age. Figure 5: Prediction error for each year when pre- dicting sit-ups at the age of 18. taken from the database and using an appropriate model, predictions at the age of 18 – we assume the person will be fully developed by this age – are generated. Models are generated for each SLOfit parameter and as such need to be run separately for each prediction we want to generate. As the overall goal of the system is to generate the predic- tions for a group of people, not just for an individual, the result should be a cross-section of the population based on a filter that is applied. The filter is usually the mean for the population, but other options are also available, for in- Figure 2: Prediction error for each year when pre- stance quartiles or median. The system automatically takes dicting height at the age of 18. the information for the available children based on the re- gion where they are from, and generates forecasts for each child. This can require the system to generate multiple fore- casts for each individual, for instance height and weight if the desired outcome of the analysis is the BMI. Due to the nature of the system the result of this operation is stored in the database and must be retrieved from there. The aggre- gator then takes this information and generates reports that can be visualized by the CH systems. 5. RISK ASSESSMENT In order to asses the risks for mortality, a stochastic model was generated that describes how BMI, CRF (approximated Figure 3: Prediction error for each year when pre- by 600 m run) and MF (approximated by sit-ups in 30 s) dicting weight at the age of 18. influence mortality. The influence is based on several pub- lished studies [2, 4, 5, 7] that relate fitness indicators to all- cause mortality. Table 4 shows how less-than-ideal values 4.1 Forecasting the population based on col- of different parameters increase the probability of mortality lected measurements [6]. In the case of BMI, it is not surprising that this happens Forecasting of SLOfit parameters takes place at the level of if the individual is overweight. But low BMI is also a risk as an individual. The available measurements of a person are it signals other difficulties of the person. The risk for obese 445 Table 2: The risk increases for certain calculated metrics. BMI (kg/m2) 15-18.5 18.5-20 20-22.5 22.5-25 25-27.5 Risk increase(%) 82 44 2 ref. 7 BMI (kg/m2) 27.5-30 30-35 35-40 40-60 Risk increase(%) 27 66 166 335 CRF (600m run ) Q1 (high) Q2 Q3 Q4 Q5 (low) Risk increase (%) ref. 28 59 78 85 MF (30s sit-u ps) Q1 (high) Q2 Q3 Q4 (low) Risk increase (%) ref. 61 32 172 people rises quite drastically, since increased weight prevents jana, email: Bojan.Leskosek@fsp.uni-lj.si) and Gregor a person from exercising, further decreasing fitness and in- Jurak (Faculty of Sports, University of Ljubljana, email: creasing the risk for comorbidities of physical or psycholog- Gregor.Jurak@fsp.uni-lj.si). ical nature. Low CRF and MF have similar consequences, increasing the risk to one’s health. While these factors are 9. REFERENCES certainly correlated there is at this time no quantitative data [1] A. Azqueta-Alzuaz, M. Patino-Martinez, I. Brondino, to what extent the correlation should be taken into account. and R. Jimenez-Peris. Massive data load on distributed There is also no concrete information how fitness at the end database systems over HBase. Proceedings - 2017 17th of schooling predicts the fitness of individuals during the rest IEEE/ACM International Symposium on Cluster, of their life, as they can at any time decide to change their Cloud and Grid Computing, CCGRID 2017, pages lifestyle. However, since the change can be for the better or 776–779, 2017. worse, we assume it stays the same, which is probably not [2] C. E. Barlow, L. F. DeFina, N. B. Radford, J. D. Berry, far from the truth for the whole population. K. H. Cooper, W. L. Haskell, L. W. Jones, and S. G. Lakoski. Cardiorespiratory fitness and long-term 6. CONCLUSIONS AND FURTHER WORK survival in ”low-risk” adults. Journal of the American Predicting the state of the population and the associated Heart Association, 1(4):e001354, 2012. risks for them in the future is an important goal if we want [3] D. Bender and K. Sartipi. Hl7 fhir: An agile and restful to provide good advice to individuals and people that are approach to healthcare information exchange. In directly or indirectly given charge over them. While chil- Computer-Based Medical Systems (CBMS), 2013 IEEE dren are the focus of the current work, the implications are 26th International Symposium on, pages 326–331. broader. The same approaches could be used on the adult IEEE, 2013. population, predicting their physical fitness and assessing [4] N. R. F. Collaboration et al. Trends in adult body-mass their risks during their lifetime. index in 200 countries from 1975 to 2014: a pooled analysis of 1698 population-based measurement studies In the future work of the project we plan to increase the pre- with 19· 2 million participants. The Lancet, dictive power of the models by using more data and more 387(10026):1377–1396, 2016. advanced machine-learning methods. Risk assessment will [5] E. Di Angelantonio, S. N. Bhupathiraju, D. Wormser, be augmented by additional studies from the literature. We P. Gao, S. Kaptoge, A. B. de Gonzalez, B. J. Cairns, would also like to base it on our own data, but it is doubt- R. Huxley, C. L. Jackson, G. Joshy, et al. Body-mass ful we will be able to obtain appropriate data, since most index and all-cause mortality: schoolchildren in the SLOfit dataset do not yet suffer from individual-participant-data meta-analysis of 239 many serious health problems, and relating their fitness with prospective studies in four continents. The Lancet, medical data is problematic for privacy reasons. 388(10046):776–786, 2016. [6] A. Gradišek, M. Mlakar, M. Cigale, L. Lajovic, Perhaps the greater advancement will be achieved by model- M. Luštrek, M. Sorić, G. Starc, B. Leskošek, and ing the impact of various health policies and interventions – G. Jurak. Physical Fitness Forecasting and Risk for instance, what happens if an additional hour of physical Estimation in Slovenian Schoolchildren. Studies in education is instituted at a school. health technology and informatics, 251:125–128, 2018. [7] P. T. Katzmarzyk and C. L. Craig. Musculoskeletal 7. ACKNOWLEDGMENTS fitness and risk of mortality. Medicine and science in Funding: This work was supported by the European Union’s sports and exercise, 34(5):740–744, 2002. Horizon 2020 research and innovation program [grant agree- [8] G. Miller, C. Roehrig, P. Hughes-Cromwick, and ment No 727560 (CrowdHEALTH)]. C. Lake. Quantifying national spending on wellness and prevention. In Beyond Health Insurance: Public Policy 8. ADDITIONAL AUTHORS to Improve Health, pages 1–24. Emerald Group Additional authors: Maroje Sorić (Faculty of Sports, Uni- Publishing Limited, 2008. versity of Ljubljana, email: Maroje.Soric@fsp.uni-lj.si) and Gregor Starc (Faculty of Sports, University of Ljubl- jana, email: Gregor.Starc@fsp.uni-lj.si) and Bojan Leskovšek (Faculty of Sports, University of Ljubl- 446 Zapestnica za pomo č starejšim Tomaž Kompara Elgoline d.o.o. Podskrajnik 34 1380 Cerknica Slovenija tomaz.kompara@elgoline.si POVZETEK Da bi starejšim približali sodobno tehnologijo je to potrebno, prilagoditi njihovim potrebam in željam ter jo narediti in- tuitivno za uporabo. Intuitivnost uporabe lahko dosežemo tako, da uorabnik ne potrebuje nobene dodatne akcije za izvedbo novih funkcionalnosti. Primer takšne funkcional- nosti je detekcija padcev, pri čemer uporabnik ne potrebuje ročno prožiti alarma, ampak se ta samodejno proži ob padcu. Kljub enostavnosti uporabe pa je potrebno zagotoviti, da bo zapestnica vedno na zapestju, s čimer omogočimo večjo varnost uporabnika. V tem delu so predstavljeni rezultati testov in analiz. Ključne besede detekcija padcev, zapestnica, starejši Slika 1: Razpoložjivost baterije. 1. UVOD Da bo razvita zapestnica primerna za končne uporabnike, mora izpolnjevati uporabnikova pričakovanja, saj lahko v na- 2.2 Polnjenje sprotnem primeru določena pomanjkljivost privede do slabe Vprašanim smo predstavili dva načina polnjenja: žično in uporabniške izkušnje in posledično do neuporabe zapestnice brezžično. Vsi uporabniki so bili že seznanjeni z žičnim pol- ter nezadovoljstva uporabnika. Da bi dosegli čim boljšo upo- njenjem, medtem ko so le nekateri (20%) poznali brezžično rabniško izkušnjo, smo izvedli raziskave na področjih, kot so: polnjenje. Vsi vprašani so bili mnenja, da je brezžično pol- avtonomija, polnjenje, ohišje, detekcija padcev in lokaliza- njenje enostavnejše in uporabniku bolj prijazno, medtem ko cija. V nadaljevanju so predstavljene ugotovitve raziskav. je žično polnjenje lahko zelo zahtevno za starejše. Težave, ki so jih navedli pri žičnem polnjenju je majhnost in krhkost 2. ANALIZE priključkov, ki jih je težko videti, kaj šele pravilno priklju- Analize so bile izvedene s pomočjo intervjujev 10 ljudi pov- čiti. Pri brezžičnem polnjenju uporabniki niso imeli težav, prečne starosti 63 let, pri čemer nekateri samostojno pre- čeprav jo je večina vprašanih prvič videla. Nekateri (20%) bivajo v lastnih domovih, drugi prebivajo skupaj z družino bi bili pripravljeni tudi pogosteje polniti zapestnico, ki bi ali v varovanih domovih. V nadaljevanju so predstavljeni omogočala brezžično polnjenje. rezultati analiz. 2.1 Avtonomija Zaradi narave naprave je potrebno v napravi ohraniti do- volj energije, da je ob klicu na pomoč mogoče izvesti klic ter omogočiti vsaj 5-minutni pogovor z klicanim. V nasprotnem primeru se lahko zgodi, da klica ni mogoče izvesti, zaradi česar v času, ko je akumulator skoraj prazen, uporabnik ni varovan, čeprav se tega morda ne zaveda. V analizi smo že- leli izvedeti, kolikšna je minimalna oz. željena avtonomnost naprave, v kateri je uporabnik varovan. Mnenja vprašanih so se precej razlikovala, in sicer od enega tedna do enega meseca. Čim daljša avtonomija za uporabnika pomeni manj skrbi in več svobode, medtem ko krajša avtonomija pomeni dodatno nalogo, ki jo lahko pozabijo. Nekateri (40%) so polnjenje povezovali tudi z mobilnim telefonom, za katerega Slika 2: Primeri brezžičnega polnjenja. pravijo, da ga velikokrat pozabijo napolniti. 447 2.3 Ohišje Ohišje naprave je lahko velikokrat ključni faktor nakupa po- samezne naprave, pri čemer ni pomembna le oblika, temveč tudi funkcionalnost ohišja [1]. Po vprašanju kakšno ohišje bi si želeli, so nekateri odgovorili, da bi bilo enako uri, ki jo nosijo, drugi so želeli čim manjšo in takšno, ki ne bode pre- več v oči. Več kot polovica vprašanih (60%) bi imela rajši zapestnico, ki ima tudi ekran z uro ali celo uro z kazalci. So se pa skoraj vsi vprašani (90%) strinjali s tem, da je zelo dobro, če bi bila zapestnica vodoodporna, s čimer jim ne bi bilo potrebno skrbeti, kdaj jo nosijo ter je poleg tega ne bi mogli pozabiti namestiti na roko. 2.4 Ročno proženje alarma Razvita zapestnica omogoča klic na pomoč ob kliku na gumb (na trgu obstajajo podobni produkti [2, 3, 4, 5, 6]), kar omo- goča uporabniku, da je vedno varovan in lahko pomoč hitro in enostavno pokliče, ne glede na situacijo. Vprašani so se strinjali, da je to uporabna funkcionalnost ter bi jo vsi upo- rabljali. Udeležene v intervjuju smo povprašali po dogodkih, ob katerih bi to funkcionalnost uporabili in ali se jim je kdaj pripetilo, da pomoči ne bi mogli sami poklicati s pomočjo Slika 4: Samodejna detekcija padcev. zapestnice. Navedli so nekaj primerov, v katerih bi bilo kli- canje na pomoč oteženo oz. zakasnjeno (npr. ukleščena roka je posredovanje lokacije, ki omogoča hitrejše posredovanje pod telesom, šok zaradi padca, izguba zavesti), v nekaterih koristna in jih ne bi motila. okoliščinah mogoče celo nemogoče (npr. možganska kap). 3. ZAKLJU ČEK Pridobljeni podatki o potencialnih uporabnikovih željah in pričakovanjih ter o njihovih izkušnjah so pomembni vhodni podatki pri načrtovanju zapestnice. Na podlagi pridobljenih podatkov lahko sklenemo, da je zapestnica zelo zaželena ter bo olajšala življenja uporabnikov, kar je glavni cilj ravzoja. Trenutna omejitev je le detekcija padcev, ki še ne ustreza pri- čakovanjem vprašanih. Drugi izziv, s katerim se v razvoju še nismo soočili je razvoj ohišja, ki bo tako po funkcionalnosti kot tudi po izgledu prilagojen starejšim. 4. ZAHVALE Raziskava je bila izvedena v okviru projekta ”Ekosistem Pa- metnega mesta (EkoSmart)” in je sofinancirana s strani Re- publike Slovenije in Ministrstva za izobraževanje, znanost in šport ter Evropske unije iz Evropskega sklada za regionalni Slika 3: Ročno proženje alarma. razvoj (ESRR). 2.5 Detekcija padcev 5. VIRI Pomembna funkcionalnost zapestnice je samodejna detekcija [1] Korir Loice. Effect of buyer-supplier relationships on padcev. Želeli smo izvedeti, ali je ta funkcionalnost zanimiva procurement performance: Evidence from kenyan za potencialne uporabnike. Vsi vprašani so bili mnenja, da supermarket. ESJ, 1(Special edition vol. 1), 2015. je to zelo uporabna funkcija. V nadaljevanju smo želeli ugo- [2] Safeguardian, carecallersTM. Dostopno na toviti, koliko udobja so uporabniki pripravljeni žrtvovati za https://safeguardian.com/, september 2017. to funkcionalnost. Tukaj so se mnenja uporabnikov zelo raz- [3] Life call, fallalertTM system. Dostopno na likovala. Nekaterim (30%) ne bi bilo moteče, če bi detekcija http://lifecall.com/products/, september 2017. enkrat na dan napačno zaznala padec, v kolikor bi v primeru [4] Medical guardian, premium guardian. Dostopno na pravega padca ta bil pravilno zaznan. Drugi vprašani (70%) https://www.medicalguardian.com/product/, bi želeli bolj natančen algoritem, drugače te funkcionalnosti september 2017. najverjetneje ne bi uporabljali. [5] Alert 1, mobile medical alert system. Dostopno na https://www.alert-1.com/, september 2017. 2.6 Lokalizacija [6] Philips, gosafe. Dostopno na Potencialne uporabnike smo spraševali, ali so pripravljeni https://www.lifeline.philips.com/, september 2017. deliti svojo lokacijo za namene hitrejšega posredovanja v pri- meru klica na pomoč. Vsi vprašani so bili istega mnenja, da 448 A Protocol for Joint Acquisition of Heterogeneous Resources Gregor Grasselli Jožef Stefan Institute and Jožef Stefan International Postgraduate School Jamova 39 1000 Ljubljana, Slovenia gregor.grasselli@ijs.si ABSTRACT for satisfying each of those levels of satisfaction. A possi- This article presents a protocol for multiple agents to jointly ble setting where the levels of satisfaction correspond to the acquire a number of heterogeneous resources. The proto- number and kind of home appliances to be turned on, which col allows the agents to negotiate the amount of resources in turn corresponds to the agent’s electricity consumption is bought based on their individual needs and budget con- presented in [3]. In fact the protocol presented here builds straints. It also ensures that the price of the purchased upon the protocol described in [3] without some of the limi- resources is fairy distributed among all the buyers, and that tations introduced in that article: the cost function need not risk averse agents will not try to gain an advantage by being be convex, and it does not have to be a weighted sum of the untruthful. costs of individual resources. This has its drawbacks, as our protocol needs more information from each of the participat- Keywords ing smart home agents and also does not have a mechanism for rewarding agents with a smaller consumption. smart city, protocol, heterogeneous resources, negotiations 2. NOTATION CONVENTIONS AND DEFI- 1. INTRODUCTION As buying in bulk is usually more advantageous then buying NITIONS smaller amounts of resources, it is natural that agents in a The actors in the protocol are the consumer agents and a smart city would organise together to acquire resources as a coordinator agent whose role is to collect the needs of the group in larger quantities to benefit from that. This however consumers and then inform them of the prices associated produces a need for an established protocol that is capable with their purchases. Let us use n to denote the total num- of determining the amounts of resources to order, as well ber of consumer agents, and N to denote the set of all con- as a way to fairly distribute the price tag for the purchase sumer agents. We will use the letter c to refer to a consumer among the buyers. This article presents a protocol that aims agent; in particular let ci, i = 1, . . . , n denote the i-th con- at solving both of these issues. Finally, by producing a pro- sumer agent. We will describe the resources being bought tocol that is fully automatic and capable of providing real using the notation ri, r = 1, . . . , m where m is the total num- time decisions on the bought resource bundles, this proto- ber of available resources, and denote the set of all resources col also makes it possible for the agents to strategize when by M . Finally we will also need a way to express the cost they buy different resources, based on their current prices. of a bundle. While the language of the article will be limited to a setting where smart houses jointly acquire resources, the presented protocol is applicable in other areas within and outside of Definition 1. Let R+ be the set of all non negative real numbers. A function C : m the smart city setting. R+ → R+ is a cost function when it is continuous, increasing in the sense, that for any two a, b ∈ m The protocol presented here is applicable in a scenario where R+ such that a o(k)} Thus C(q) is in the image of h of every q ∈ m R+ . Because h is for every k ∈ N and Q increasing it is injective. We have thus proven the following 0 = 0 and Vk [0] is the first element of the bundles vector belonging to consumer theorem: ck. When a consumer ci has no resource bundle in EC (Vk[0]), the value of pi(Vk[0]), is produced by find- Theorem 1. Let pf be an increasing unbounded path. Then ing the intersection between EC (Vk[0]) and an unbounded for any q ∈ m R+ , p(f ) intersects EC (q) exactly once. increasing path that is initially constructed for each user based on the list of bundles as reported in step 2. The method of its construction is described in subsec- We will use the notation pi(q) to mean the intersection be- tion 3.1. tween the unbounded path constructed from the list of bun- dles corresponding to the levels of satisfaction of user ci and 4. The coordinator informs each consumer c the set E i of their price C (q). We will use o : N → N to denote the ordering x of users based on the price of their current consumption, in i for their current comfort level. other words o(ci) < o(cj ) means that C(qi) < C(qj ) where 5. Those consumers who find the price acceptable send we use qj to denote the bundle requested by consumer cj a confirmation to the coordinator. If that covers all users, the process terminates. 3. PROTOCOL DESCRIPTION The protocol starts when the coordinator agent sends a re- 6. For every consumer ci who did not agree with the price, quest to all of the participating consumer agents. Each of the coordinator pops the first element from Vi, and the the consumer agents is required to respond with a list of protocol restarts from point 3. resource bundles corresponding to their satisfaction levels and ordered by their values. From here on, the protocol is a 3.1 Constructing Paths series of rounds where the coordinator agent computes the The formula used to compute the individual payments for price to be paid by each of the consumer agents for their the consumer agents is the path serial rule as described in [2]. most expensive bundle. The consumers then respond with a As its name suggests, it uses the serial cost sharing rule from yes or no, based on whether they find the price acceptable. [1] to produce a fair cost allocation between the consumer If the price is not acceptable, the most expensive bundle agents. The serial cost sharing rule works by saying that for the consumer is removed from its list and a new round the consumer agent with the smallest demand should pay a is started. If a consumer’s list is empty, it is understood proportion of the cost incurred by his demand. That is that the agent is happy with paying 0 for receiving 0 re- sources. The protocol terminates when all consumer agents C(nq1) agree with the price they are paying (obviously, it does ter- , n minate, since for every round in which some agents do not where we are using the same conventions as before, except agree to the price, the total length of all the bundle lists is that now q is a number and the users are already sorted. shortened). The second user has to pay his proportionate share of the cost incurred by the smallest demand as well as his propor- A full description of the algorithm is given by: tional share of the cost assigned to the difference between his demand and that of the first user. And so on for the rest 1. The coordinator requests a list of resource bundles of them, each covering their proportional share of all the from each consumer. cost differences up to the level of their own demand. The 450 last user covers the remaining difference alone. So each user 4.1 Proofs of the Path Serial Rule Properties has to pay Proving the first two properties, (E) and (S) is as simple as observing that the sum from equation (1) is exactly the j X C ((n − k)qk ) − C ((n − k + 1)qk−1) same for all users whose demands are equally costly, which xj = . n − k + 1 proves (E) and that since only intersections of paths with the k=1 EC (q) subspaces are present in the terms of the sum from equation (1), when referring to costlier demands than that In order to generalise this to multiple dimensions it is nec- of the consumer whose payment we are computing, chang- essary to find a way to represent the amounts requested by ing the size of those costlier demands does indeed have no other consumers on the same price level as that of the cur- bearing on the price payed by the given user, which proves rent consumer, so that a fair proportion can be found. It is (S). shown in [2] that in order to achieve that in a manner that preserves the nice properties of the serial rule, it is enough In order to show property (I), let us call the linear trans- for each consumer to be assigned any increasing path from 0 formation used to change the scales f . Because it is lin- to their bundle vector and then use the intersection of that ear it has an inverse f −1. Let C0 be the equivalent cost path with the price level. function of C on the transformed space. The function C0 must assign the same value to the same actual amount of We have decided to use paths that are piecewise linear. They resources as C. So we must have that for any q in the orig- start at 0, and then go in straight lines from one bundle to inal space C0(f (q)) = C(q). If a path intersects EC (q) in the next. Since we never need to find an intersection of a a point s, then f (s) is on the transformed path, but it is consumer’s path with a price level above his most pricey also in EC0 (f (q)), meaning that f (s) is the intersection of bundle, the path from there on does not really matter. In the transformed path with EC0 (f (q)). This means that the order to make it unbounded in the sense of definition 3 we terms in the sum of equation (1) all stay exactly the same can set ti to a straight line going in the direction of a vector and (I) holds as well. whose coordinates are all 1. Theorem 1 then guarantees the existence of all the required pi(qk) for computing the price 5. CONCLUSIONS allocations for individual consumer agents. We have shown a possible protocol for joint acquisition of heterogeneous resources by any number of consumer agents. 4. PROPERTIES OF THE PROTOCOL It inherits the properties that make it fair from the path se- rial rule. We believe that it might be an interesting exercise Since the final prices payed by the consumers are computed to try finding path constructions, that would add additional using the “path serial rule” that is described in [2] they have properties to the protocol, such as guaranteeing that a price all of the properties brought by the rule. Those properties once agreed upon never changes in the future, or that con- are: sumers with cheaper demands would enjoy additional bene- fits. If they exist, that is. 1. Equal treatment of equivalent demands (E) 6. ACKNOWLEDGMENTS The research presented in this article was done as part of 2. The serial principle (S) the Ekosmart project, which is financed by the Slovenian Ministry of Education, Science and Sport, and the European Regional Development Fund. The first of these properties means that for any two con- sumers ci, cj whose demands qi, qj in the final round of the 7. REFERENCES protocol were equally priced, pay the same amount of money. [1] H. Moulin and S. Shenker. Serial cost sharing. Property 2 means that the amount payed by consumer ci is Econometrica, 60(5):1009–1037, September 1992. the same no matter how big the demand of any other con- [2] C. Téjédo and M. Truchon. Serial cost sharing in sumer cj as long as C(qj ) > C(qi) and that qj stays on the multidimensional contexts. Mathematical Social same path. Sciences, April 2002. The original path serial rule has a third property called or- [3] J. Zupančič and M. Gams. Dynamic protocol for the dinality which states that the payments allotted to the con- demand management of heterogeneous resources with sumer agents do not change under arbitrary change of units convex cost functions. Informatica, 41(1):121–128, for measuring resource quantities, as long as the paths used March 2017. to compute the payments are transformed along with the rest. This would not do here, because the paths are always piecewise linear functions that go between the points in the consumers’ demand lists. However, if we restrict ourselves to linear bijective transformations, that will map line seg- ments to line segments, this property is still true. Since changing measurement units is usually a linear transforma- tion this property that is usually called scale invariance (I) covers most if not all cases where ordinality would be needed in an actual application of the protocol. 451 Sistem za podporo odlo čanju zdravniku dr. Andrej Brodnik Nal Lukšič Janez Žibert Univerza v Ljubljani, Fakulteta Univerza v Ljubljani, Fakulteta Univerza v Ljubljani, Fakulteta za računalništvo in informatiko za računalništvo in informatiko za računalništvo in informatiko Univerza na Primorskem, ter Zdravstvena fakulteta Oddelek za informacijske znanosti in tehnologije Nenad Živković Andrej Rus Parsek d.o.o. Univerza v Ljubljani, Fakulteta za računalništvo in informatiko ABSTRACT • Podatki v zdravstvenih ustanovah so večinoma otoki Sistemi za podporo pri odločanju zahtevajo nekoliko speci- informacij brez možnosti izmenjave le-teh z drugimi fično arhitekturo, ki vključuje poseben gradnik. Le-ta ima ustanovami. običajno dodatno zalogo podatkov, ki mu omogočajo, da ob • Informacije so še vedno v nestandardni, tiskani in ne- upoštevanju ostalih zdravstvenih podatkov pripravi zdravniku strukturirani obliki ter pogosto niso na voljo, ko se jih ostale podatke kot na primer statistične podatke o stanju potrebuje. pacienta ali o klinični poti zdravljene bolezni. • Zdravstveni strokovnjaki z uporabo neoptimalnih in neprijaznih rešitev dragocen čas posvečajo adminis- V prisoevku opisujemo takšno arhitekturo, ki bo črpala po- traciji namesto pacientom. datke iz glavnega medicinskega repozitorija (na primer sis- • Pacient kot subjekt in uporabnik zdravstvenih storitev tema fhir ali sistema z opisi kliničnih poti in podobno). nima možnosti dostopa do optimalnih zdravstvenih sto- Obdelane podatke gradnik lokalno obdela in njihov izvleček ritev ter prav tako nima možnosti enostavnega in pri- hrani ter ga predstavi zdravniku, ko slednji obranava pa- jaznega dostopa do svojih informacij. cienta. Podjetje Parsek si je zadalo ob tem dva cilja – interoperabil- Arhitekturo smo prototipno implementirali v okviru sistema nost ter uporabniku prijazni in smiselni digitalni uporabniški Vitaly. vmesniki z dobro premišljenimi delovnimi postopki (work- flow ). Interoperabilnost dosega z uporabo standardov v zdravstveni informatiki [1]. V njihovem sistemu se podatki Categories and Subject Descriptors izmenjujejo z HL7 sporočili in dokumenti, delovni postopki C [e]: lostna oskrba v bolnišnici pa z realizacijo specifikacij profilov IHE. Uporabniški vmes- niki so rezultat Human-centered design (HCD) procesa načr- Keywords tovanja in izvedbe, pri kateri so sodelovali zdravstveni stro- kovnjaki s posameznih področjih. Rešitve so načrtovane za pomoč pri odločitvah v zdravstvu, CDS, CDSS, FHIR, Vi- končne uporabnike za njihove potrebe. S tovrstnim načinom taly razvoja je omogočen velikemu številu ljudi dostop do opti- malne zdravstvene storitve. V tem digitalno spremenjenem 1. UVOD okolju se informacije zlahka delijo med strokovnjaki in bol- Poslanstvo sistemov, ki jih poznamo pod skupnim imenom niki ter vsakemu posamezniku nudijo najboljšo razpoložljivo e-zdravje, je izboljšati zdravje ljudi z uporabniku prijazn- zdravstveno storitev. imi programskimi rešitvami, ki močno poenostavljajo sode- lovanje na področju zdravstvenih storitev. 1.1 Sistem Vitaly Zdravstvo je ena od redkih panog, v kateri tehnološke rešitve Platforma Vitaly in rešitve zgrajene nad njo prinašajo celovit še vedno ne prinašajo optimalnih rezultatov. Razlogov je vpogled v zdravstveno stanje pacienta, omogočajo digital- več: izacijo zdravstvenih storitev ter sodelovanje med zdravstven- imi strokovnjaki in pacienti. Poleg tega prinašajo kontekst, ki je zelo pomemben, in predstavlja dodano vrednost v nas- protju z enostavnim deljenjem in prikazom podatkov. Vitaly rešitve, povezane na zdravstvene sisteme z uporabo interoperabilnih rešitev, ponujajo zdravstvenim ustanovam možnost, da pacientu omogočijo enostaven dostop do zdrav- stvenega kartona (Electronic Health Record ) in zdravstvenih dokumentov ter omogočajo ponujanje zdravstvenih storitev. Po drugi strani lahko pacient upravlja z osebnim zdrav- 452 stvenim zapisom, ki ga deli z zdravstvenimi strokovnjaki, 2.1 Predpisovanje zdravil na klinični poti in dostopa do storitev na enostaven način. Na ta način Klinična pot predstavlja natančen, vnaprej zapisan protokol vsi prispevajo k celovitejši sliki o pacientu, delujejo pre- vseh aktivnosti med obravnavo določenega zdravstvenega ventivno in skrajšajo čas, potreben za zdravstveno oskrbo. stanja [5]. Med te aktivnosti spada tudi predpisovanje zdravil. Podatki so na voljo tudi, ko je pacient na videz zdrav in Ko zdravnik ve, katero zdravilo želi predpisati pacientu, ne samo, ko potrebuje zdravstveno ukrepanje. Ko imajo mora pri tem preveriti, če morebiti obstajajo kontraindikacije. deležniki enkrat dostop do zdravstvenih podatkov, digital- To so kriteriji za odložitev medicinskih ukrepov (uporabe izacija procesov v zdravstvu postaja uresničljiva oziroma določenega zdravila, medicinskega pripomočka ali postopka bolj optimalna. Digitalizacija procesov znotraj zdravstvene zdravljenja) zaradi stanja bolnika, dejavnikov ali drugih oko- ustanove, zdravstvena oskrba na daljavo, sodelovanje odd- liščin. Kontraindikacijo za uporabo določenega medicinskega aljenih zdravstvenih strokovnjakov z uporabo ustreznih teh- ukrepa lahko na primer predstavljajo sočasne bolezni, alergije, nologij tako zagotavlja pravočasno, optimalno in najboljšo določen genotip, predhodni neželeni učinki zdravila ali skupine storitev za pacienta. zdravil, starost, spol, predispozicije [8]. Da zmanjšamo število napačno predpisanih zdravil, lahko zdravniku ponudimo po- V nadaljevanju prispevka najprej opišemo arhitekturo in v moč v obliki modula ekspertnega sistema, ki bo skrbel za njej implementiranih dveh različnih rešitev. Prva je pod- preverjanje kontraindikacij pri predpisu zdravila pacientu. pora pri predpisovanju zdravil v okviru klinične poti in druga predstavalja podporo pri diagnosticiranju pacienta. Sistem je zgrajen po principu, ki temelji na predhodnem znanju in ne vključuje umetne inteligence, ampak upošteva vnaprej podana pravila za odločanje klinične poti. Bazo 2. ARHITEKTURA znanja predstavlja podatkovna baza, kjer so navedene kon- Arhitektura sistema, opisanega v nadaljevanju, je primer traindikacije posameznih zdravilnih učinkovin. razširitve že obstoječega sistema Vitaly. Arhitektura razšir- itve je sestavljena iz štirih delov, kakor je prikazano na sliki 1. Ko želi zdravnik pacientu predpisati zdravilo, se v sistem Jedro predstavlja modularni ekspertni sistem, ki vsebuje ra- pošljejo podatki o pacientovi identifikacijski številki strežnika fhir ter šifra zdravila, ki je bilo predpisano. S pomočjo teh dveh podatkov sistem pridobi vse kontraindikacije zdravilnih učinkovin, ki jih zdravilo vsebuje, iz baze znanja ter pacien- tove medicinske podatke s strežnika fhir. Zgodi se primer- java teh dveh skupin podatkov in če so najdene skupne točke, zdravnik dobi obvestilo, da pri predpisu tega zdravila ob- staja nevarnost, da pride do zdravstvenih zapletov. Gre torej za funkcionalnost, ki zdravniku ne predlaga določenih rešitev, temveč preprečuje morebitne napake. Pri implementaciji testnega sistema smo se omejili na eno zdravilno učinkovino, in sicer klozapin, ki se uporablja pri zdravljenju psihoz. Izkazalo se je, da se kontraindikacije med seboj precej razlikujejo in da če primerjamo na primer kon- traindikaciji bolezen paralitični ileus in nezmožnost opravl- janja rednih preiskav krvi vidimo, da pri prvi lahko pri pre- verjanju kontraindikacij preprosto iščemo po imenu bolezen- Slika 1: Arhitektura razširitev sistema Vitaly. skega stanja oziroma še boljše po njegovi šifri, ki ni vezana na jezik, ki ga uporabljamo. V drugem primeru ne gre za bolezensko stanje ali zdravilno učinkovino in je prever- zlične funkcionalnosti. Tako sistem Vitaly, kot ekspertni janje po pacientovem zdravstvenem kartonu bolj zapleteno. sistem črpata podatke s strežnika fhir [6]. Ta se uporablja Testni sistem, ki smo ga postavili, deluje za prvi primer in za vnašanje in dostopanje do medicinskih podatkov. Tretja izkazalo se je, da v primeru najdenih kontraindikacij do- komponenta sistema so baze podatkov, ki jih ekspertni sis- bimo ustrezno obvestilo. Če bi želeli sistem nadgraditi ter tem potrebuje za svoje delovanje. Zaradi različnih funkci- ga uporabljati v praksi, bi bilo potrebno povezati naš sistem onalnosti sistema, dostopa do različnih podatkovnih baz. z eno izmed že obstoječih baz kontraindikacij, kjer bi imeli Primer take baze je baza opisov kliničnih poti vključno s podatke za vse zdravilne učinkovine, obenem pa bi morali kontraindikacijami posameznih zdravilnih učinkovin, ali po imeti za vse tipe kontraindikacij šifre, po katerih bi lahko boleznih razvrščene vrednosti meritev telesnih značilnosti. iskali znotraj elektronskega zdravstvenega kartona. Zadnji del sistema predstavlja uporabnik, ki do sistema dos- topa preko uporabniškega vmesnika, katerega vključuje že V Sloveniji za enkrat v državnih bolnicah zdravniki nimajo obstoječ sistem Vitaly, in skrbi za vnos novih podatkov in na voljo tovrstnega sistema, se je pa pojavil korak naprej uporabo funkcionalnosti sistema. pri farmacevtih, ki pred izdajanjem zdravila preverijo, če obstajajo kontraindikacije med zdravili, ki so bili pacientu V obstoječo arhitekturo lahko vključimo različne module predpisani. ekspertnega sistema. Dva primera takšnih razširitev sta podpora zdravniku pri predpisovanju zdravil na klinični poti in sistem za podporo zdravniku pri diagnosticiranju bol- nikov. 453 2.2 Diagnosticiranje bolnikov 3. ZAKLJUčEK Drugi primer modularnega ekspertnega sistema temelji na V prispevku smo predstavili prototip arhitekture, ki omo- osnovi statistične obdelave medicinskih podatkov, ki s po- goča nadgradnjo zdravstvenega sistema z dodatnimi moduli močjo znanj iz velike količine podatkov pomaga zdravstvenim za pomoč pri odločanju zdravniku. Namen implementirane uslužbencem pri diagnosticiranju bolnikov in boljšemu ra- izvedbe je zgolj prikaz prototipnega delovanja. V nasled- zumevanju delovanja bolezni. V medicini obstaja že kar njem koraku je potrebno uporabiti višjo stopnjio abstrak- nekaj podjetij, ki se ukvarjajo z razvojem tovrstnih siste- cije arhitekture, ki bo omogočala po eni strani preprostejšo mov. Nekatera med njimi so Lumiata [3], Infermedica [9] razširitev na različne kliničnih poti, oziroma na upiorabo in Health Catalyst [10]. Vsako podjetje ima svojevrstno različnih podatkov pri diagnosticiranju pacientov. Po drugi arhitekturo sistema in tako svoje prednosti. Mi smo se po strani pa je potrebno samo arhitekturo abstrahirati tako, da drugi strani osredotočili na preprostost uporabe, pregled- se za modeliranje uporabi jezik BPML, kot je opisano v [2]. nost, povečljivost in odzivnost. S pomočjo takšne abstrakcije bo sistem preprosteje razširljiv in nadgradljiv. Modul ekspertnega sistema je sestavljen iz dveh glavnih de- lov, in sicer strukturiranja medicinskih podatkov v obliko 4. REFERENCES pripravljeno na hitro obdelavo in statistična analiza. Sistem [1] M. Beštek and A. Brodnik. Interoperability and črpa medicinske podatke o boleznih in bolnikih iz strežnikov mhealth – precondition for successful eCare. In fhir. Strežnike fhir smo izbrali zato, ker zagotavljajo struk- S. Adibi, editor, Mobile Health, chapter 5. Springer, turirano shranjevanje medicinskih podatkov. Po prenosu po- 2015. datke preoblikujemo, da so pripravljeni za nadaljnjo statis- [2] M. Beštek and A. Brodnik. Pogoji za uspešno tično obdelavo. eOskrbo. Informatica Medica Slovenica, 20(1–2):17–29, 2015. Drugi del modula ekspertnega sistema je statistična obdelava [3] R. Bhardwaj, A. R. Nambiar, and D. Dutta. A study podatkov. V sistem je mogoče vgraditi različne algoritme of machine learning in healthcare. In Computer statistične obdelave. V našem primeru smo uporabili ROC Software and Applications Conference (COMPSAC), analizo in analizo iskanja pomembnih telesnih značilnosti pri 2017 IEEE 41st Annual, volume 2, pages 236–241. nastanku bolezni s pomočjo p-vrednosti. IEEE, 2017. [4] K. Hajian-Tilaki. Receiver Operating Characteristic ROC analizo smo uporabili na tipih podatkov, ki imajo (ROC) Curve Analysis for Medical Diagnostic Test neomejeno zalogo vrednosti [4]. Z njegovo pomočjo lahko Evaluation. Caspian Journal of Internal Medicine, uporabnik najde tiste telesne značilnosti, ki so pri določeni 4(2):627–635, 2013. bolezni najbolj izrazite. Rezultati analize so v uporabniškem vmesniku prikazani v pregledni tabeli. Telesne značilnosti [5] B. Hajnrih. Priročnik za oblikovanje kliničnih poti. na levi so urejene po vrednostih AUC na desni, od največje Ministrstvo za zdravje, 2009. do najmanjše, kot je prikazano na primeru na sliki 2. [6] M. Hussain A., S. G. Langer, and M. Kohli. Learning HL7 FHIR Using the HAPI FHIR Server and Its Use Drugi primer statistične analize je iskanje p-vrednosti [7], in Medical Imaging with the SIIM Dataset. Journal of do katere smo prišli s pomočjo različnih testov. Podatke Digital Imaging, 31(3):334–340, 2018. o bolezni smo glede na tip telesne značilnosti razvrstili na [7] D. C. Montgomery and G. C. Runger. Applied kategorične in skalarne. Na vrednostih skalarnih telesnih statistics and probability for engineers. John Wiley & značilnostih, katerih porazdelitev je normalna smo izvedli T- Sons, 2010. test, sicer pa Mann Whitneyev U test. Na kategoričnih spre- [8] A. Obreza et al. Farmacevtski terminološki slovar. menljivkah smo izvedli Chi kvadrat test, razen v primeru, ko Založba ZRC, 2011. je imela telesna značilnost le 2 različne možni vrednosti smo [9] K. O. Okokpujie, A. Orimogunje, E. Noma-Osaghae, uporabili Fisherjev natančni test. Primer takšne značilnosti and O. Alashiri. An intelligent online diagnostic bi bil spol, katerega vrednost sta lahko moški ali ženska. Vsi system with epidemic alert. An Intelligent Online štirje testi se uporabljajo za računanje p-vrednosti, ki nam Diagnostic System With Epidemic Alert, 2(9), 2017. pove pomembnost telesne značilnosti pri diagnosticiranju [10] D. Sanders, D. A. Burton, and D. Protti. The bolezni. Telesne značilnosti, katerih p-vrednost je manjša, healthcare analytics adoption model: A framework so pri diagnosticiranju bolezni bolj pomembni. Rezultati and roadmap. Health Catalyst, 2013. analize so v uporabniškem vmesniku razvrščeni naraščajoče po p-vrednostih na desni, kot je razvidno iz primera na sliki 3. Opisana primera statistične obdelave podatkov sta prikaz zmožnosti delovanja arhitekture. Tovrstni ekspertni sistem lahko uporabniku omogoči lažje razumevanje bolezni in posledično diagnosticiranje pacienta, s pomočjo že ob- stoječih medicinskih podatkov. Njegov cilj je uporaba v zdravstvenih institucijah, kot pomočnik zdravniku pri di- agnosticiranju bolnika. Zdravnik lahko za bolezen, katero sumi da jo bolnik ima, vnese v sistem. Ta mu prikaže telesne značilnosti, ki so bile pri bolnikih z enako boleznijo izrazite. S pomočjo sistema torej lahko zdravnik preveri, če je bila njegova diagnoza pravilna. 454 Slika 2: Tabela z rezultati ROC analize. Slika 3: Tabela analize p-vrednosti. 455 Razvoj intervencij za srčne bolnike in bolnike z motnjami gibanja Iztok Cukjati Jure Bon Jurij Matija Kališnik Klinični oddelek za bolezni živčevja Klinični oddelek za bolezni živčevja Klin. oddelek za kirurgijo srca in ožilja Univerzitetni klinični center Ljubljana Univerzitetni klinični center Ljubljana Univerzitetni klinični center Ljubljana Zaloška 2, 1000 Ljubljana Zaloška 2, 1000 Ljubljana Zaloška 2, 1000 Ljubljana +38640186268 jure.bon@kclj.si jmkalisnik@gmail.com iztok.cukjati@upr.si Janez Žibert Zvezdan Pirtošek Univerza v Ljubljani Klinični oddelek za bolezni živčevja Zdravstvena fakulteta Univerzitetni klinični center Ljubljana Zdravstvena pot 5, 1000 Ljubljana Zaloška 2, 1000 Ljubljana janez.zibert@zf.uni-lj.si zvezdan.pirtosek@kclj.si POVZETEK in respiratorne bolezni (KOPB, astma) povzročitelj več kot 80% Države razvitega sveta se soočajo s spremembami v demografski t.i. prezgodnjih smrti v starosti od 30 do 69 let [3]. sliki z izrazitim staranjem prebivalcev ob hkratni nizki rodnosti. Dolgotrajna finančna vzdržnost zdravstvenega sistema z Tako se povečuje število starejših prebivalcev, sočasno pa se obvladovanjem stroškov NKB bo v prihodnje ključna za zmanjšuje število delovno aktivnega prebivalstva. S staranjem se učinkovito zdravljenje. K temu lahko ključno pripomore povečuje delež ljudi z nenalezljivi kroničnimi boleznimi (NKB), vpeljava IKT v procese zdravstvene oskrbe. ki so razlog za 71% vseh smrti [1]. Prav dolgotrajno zdravljenje bolnikov z NKB predstavlja največji delež sredstev V okviru projekta EkoSmart se razvijajo nove intervencijena zdravstvenega sistema in se zaradi omenjenih razlogov izrazito področjih postoperativnega spremljanja srčnih bolnikov in oseb povečuje. Preventivno delovanje, zgodnje odkrivanje in cenovno z motnjami gibanja. Del sistemov bo tudi avtomatizirano učinkovito zdravljenje pacientov z NKB bo ključno za pošiljanje podatkov v nacionalno informacijsko hrbtenico in obvladovanje stroškov pri tem pa so lahko IKT v veliko pomoč. posledično dostopnost avtoriziranim uporabnikom. Aktivno vključevanje pacientov v proces zdravstvene oskrbe skupaj z uporabo sodobnih tehnologij omogoča učinkovitejše 1.1 Tehnologija modele zdravstvene oskrbe. V slovenskem zdravstvenem sistemu se še vedno večinoma V projektu EkoSmart se bo v kliničnem okolju testiralo spletno- uporablja ročno opravljanje diagnostičnih meritev z vpisovanjem mobilne rešitve za vnos in spremljanje podatkov bolnikov za podatkov v fizične zdravstvene kartone v posameznih potrebe znanstveno-raziskovalnega dela. Tako bomo v zdravstvenih ustanovah. Tako ni (avtomatiziranega) prenosa sodelovanju Univerzitetnega kliničnega centra Ljubljana administrativnih in kliničnih podatkov o posameznem pacientu (UKCL), Medicinske fakultete (MF) in Fakultete za med različnimi zdravstvenimi ustanovami oz. delavci. računalništvo in informatiko (FRI) Univerze v Ljubljani (UL) Tehnologija že dlje časa omogoča (avtomatizirano) elektronsko razvili pilotni sistem za spremljanje kliničnih parametrov pri beleženje podatkov, kar je časovno bistveno hitrejše in cenovno srčnih bolnikih v postoperativni fazi. Sisem bo predvidoma učinkovito, podatki pa so na zahtevo avtorizirane osebe dostopni omogočal medsebojno primerjavo različnih senzorjev in s tem praktično kjerkoli in kadarkoli. Smiselna uporaba obstoječe njihovo klinično validacijo. tehnologije v (prilagojenih) kliničih poteh bi lahko pomembno vplivala na boljšo učinkovitost zdravstvene oskrbe. Ključne besede EkoSmart, e-zdravje, Parkinsonova bolezen, srčno-žilne bolezni, 1.2 Izmenljivost informacij nenalezljive kronične bolezni, interoperabilnost Uporaba IKT orodij omogoča hitro in zanesljivo (avtomatizirano) pridobivanje kliničnih in ostalih podatkov oz. informacij ključnih za boljšo (indiviualizirano) zdravstveno obravnavo pacientov. Čeprav so posamezne informacije o 1. IZHODIŠČA pacientu s pomočjo omenjene tehnologije za diagnostiko (in Staranje prebivalstva je v t.i. zahodnem svetu izrazito prisotno v spremljanje terapije) uporabne obstaja t.i. »ozko grlo« zaradi zadnjem desetletju in je posledica nizke rodnost in daljše lokalne hrambe podatkov, ki niso na voljo ostalim deležnikom; življenjske dobe. Delež starejših od 65 let naj bi se do leta 2040 pri tem pa je prav interoperabilnost ključna za učinkovitost IKT povečal sedanjih 19% na 27% celotnega prebivalstva [1]. Ob podprtih intervencij [4]. povečevanju števila starejših prebivalcev se sočasno povečuje tudi število bolnikov z NKB, ki že danes predstavljajo kar 71% V projektu EkoSmart bomo posamezne podatke pridobljene v vseh smrti [2]. Pri tem je pomembno dejstvo, da so srčno-žilne procesu zdravstvene oskrbe pošiljali v nacionalno bazo pri NIJZ; bolezni (srčni infarkt, možganska kap), rakava obolenja, diabetes s čimer bodo podatki na voljo pacientom in ostalim zaintereiranim/avtoriziranim uporabnikom. 456 1.3 Boljša zdravstvena oskrba bolezenijo od 2010 do leta 2040 podvojilo, pri tem pa so ocenjeni Osnovni namen novih IKT storitev je v boljši zdravstveni oskrbi letni stroški pacienta približno 20.000 Eur [9]. pacientov. Njihova uporaba naj zdravstvenim delavcem Pri projektu PAR se izdeluje tako nove, kot tudi dopoljuje že omogoča natančnejši in hitrejši vpogled v zdravstveno stanje obstoječe gradnike OpenEHR. Za izvedbo projekta se bo posameznih pacientov in uspešnost posameznih metod uporabilo periferni senzor motoričnih simptomov parkinsonove zdravljenja. bolezni, ki bo vključen v klinično intervenco (sensor gibanja Pri tem je ključno zadovoljevanje pričakovanj in potreb proizvajalca PKG). V okviru projekta se opravlja tudi zdravnikov, medicinskih sester, bolnikov in njihovih svojce, kar raziskovalno delo na implementaciji principov zaprte terapevtske bi moralo biti vodilo pri izdelavi tovrstnih sistemov. zanke, pri kateri meritve trenutnega stanja centralnega ali perifernega živčevja v realnem času vplivajo na modulacijo 1.4 Finančna učinkovitost terapije. Med rezultati omenjenega raziskovalnega dela so Cilj, ki ga zasleduje IKT podprta zdravstvena oskrba je deloma že razvite naslednje programske aplikacije: učinkovitejše in cenejše zdravljenje bolnikov. Hitro naraščujoči - palMEP, C++ knjižnica in GUI za avtomatsko in ročno stroški zdravljenja bolnikov z NKB tako predstavljajo poglavitni preprocesiranje in analizo TMS-EMG podatkov; razlog za pravilno in finančno učinkovito načrtovanje razvoja in vpeljave spremenjenih (IKT podprtih) zdravstvenih intervencij. - palEEG, optimizirana C++/Matlab knjižnica in GUI za časovno-frekvenčno dekompozicijo, analizo in vizualno 1.5 Izkušnje iz slovenskega prostora reprezentacijo EEG podatkov, z delno implementacijo računsko V Sloveniji se je v preteklih letih že testno izvajala vpeljava zahtevnih algoritmov v NVIDIA CUDA arhitekturi za časovno novih intervencij z uporabo IKT storitev na področju oskrbe občutljive analize v realnem času pri eksperimentih z uporabo bolnikov z NKB. Tako je bila npr. razvita in klinično validirana zaprtih zank; IKT podprta oskrba za paciente z depresijo [5], kasneje pa na - palPULSER, C++ knjižnica in GUI za mikrosekundno enotni informacijski platformi za diabetike [6], astmatike [7] in osebe s prekomerno telesno težo. natančnost kontrole trigerjev pri CED 1401 družini AD Ob tem je bila pretvornikov v TMS-EEG okolju; 4) palNAV, aplikacija za MRI predstavljena/dokazana tudi finančna učinkovitost tovrstnih vodeno nevronavigacijo v realnem času v TMS okolju, z delno intervencij na primeru depresije [8]. C++ implementacijo 2. RAZVOJ IN VALIDACIJA NOVIH 3. REFERENCE INTERVENCIJ [1] http://ec.europa.eu/eurostat/statistics- 2.1 Orodje za zdravnike-raziskovalce explained/index.php/Population_structure_and_ageing Spletno-mobilni aplikaciji sta v fazi razvoja oz. beta testiranja. [2] http://www.who.int/mediacentre/factsheets/fs355/en/ in izvedbi kliničnih študij na področjih obravnave novonastale [3] http://www.who.int/news-room/fact- atrijske fibrilacije po operacijah na srcu, izdelave aktivnih sheets/detail/noncommunicable-diseases registrov za zdravljenje atrijskih fibrilacij in za obravnavo bolnikov z motnjami gibanja, ki so vključeni v kontinuirane [4] Beštek, M., Brodnik, A. Interoperability and mHealth – oblike zdravljenja. Delo je razdeljeno v dva sklopa, obravnava precondition for successful eCare. Mobile Health: A atrijske fibrilacije (AF) in motenj gibanja (PAR). Oba projekta Technology Road Map, 2015: Chapter 16; Springer. sta primera izvedbe aktivnega registra s pripadajočo klinično [5] Meglic M, Furlan M, Kuzmanic M, Kozel D, Baraga študijo. D,Kuhar I, Kosir B, Iljaz R, Novak Sarotar B, Dernovsek MZ, Marusic A, Eysenbach G. Brodnik. A. Feasibility of an 2.2 Postoperativno spremljanje srčnih eHealth Service to Support Collaborative Depression Care: bolnikov (AF) Results of a Pilot Study. J Med Internet Res, 12(5):e63, Spremljanje srčnih bolnikov v postoperativni fazi je izrazito 2010. pomembno zaradi pojava morebitnih zapletov, med drugim tudi [6] Iljaž R, Brodnik A, Zrimec T, Cukjati I. E-healthcare for atrijske fibrilacije. Z uporabo prenosne EKG naprav, ki je Diabetes Mellitus Type 2 Patients - A Randomised cenovno relativno ugodna, se lahko doseže učinkovitejši nadzor Controlled Trial in Slovenia. Zdr Varst. 2017 May nad zdravstvenim stanjem bolniki in pravočasno prepreči 26;56(3):150-157. postoperativne, pogostno smrtno nevarne, zaplete. [7] Nemanic T, Sarc I, Skrgat S, Flezar M, Cukjati I, Marc Pri projektu AF skupina zdravnikov v sodelovanju z razvojno Malovrh M. Telemonitoring in asthma control: a ekipo na Fakulteti za računalništvo in informatiko skrbi za randomized controlled trial. J Asthma. 2018 Sep 5:1-9. doi: tehnične vidike razvoja spletno mobilnega okolja (web 10.1080/02770903.2018.1493599. aplikacija). Sočasno je v procesu izdelave tudi nabor potrebnih arhetipov v formatu OpenEHR za potrebna področja. Pripravila [8] Naveršnik K, Mrhar A. Routine real-time cost-effectiveness se je klinična monitoring of a web-based depression intervention: a risk- intervencija za (post)operativno spremljanje sharing proposal. J Med Internet Res. 2014 Feb bolnikov z atrijsko fibrilacijo, že izvedla pa se je pilotna klinična študij 27;16(2):e67. a glede uporabnosti vključitve senzorja Savvy EKG kot telemonitoringa v okviru klinične intervencije. V študijo je bilo [9] Kowal SL, Dall TM, Chakrabarti R, Storm MV, Jain A. The v pilotnem delu vključenih okoli 40 bolnikov z Kliničnega current and projected economic burden of Parkinson's oddelka za kardiovaskularno kirurgijo UKCL. disease in the United States. Mov Disord. 2013. Mar;28(3):311-8 2.3 Vodenje bolnikov z motnjami gibanja (PAR) Število bolnikov z nevrodegenerativnimi boleznimi je v izrazitem porastu, tako naj bi se število bolnikov s Parkinsonovo 457 458 Indeks avtorjev / Author index Ambrožič Borut .......................................................................................................................................................................... 398 Andova Andrejaana ........................................................................................................................................................................ 9 Avbelj Viktor ..................................................................................................................................................................... 381, 409 Banda Marko ................................................................................................................................................................................ 82 Bartz-Beielstein Thomas ............................................................................................................................................ 213, 217, 245 Beranič Tina ............................................................................................................................................................................... 335 Bizjak Jani .............................................................................................................................................................................. 29, 37 Black Michaela ........................................................................................................................................................................... 181 Blažun Vošner Helena ................................................................................................................................................................ 411 Bobnar J. .................................................................................................................................................................................... 439 Bohanec Marko ............................................................................................................................................................................ 17 Bokal Drago ................................................................................................................................................................................. 21 Bon Jure ..................................................................................................................................................................................... 456 Bozic Josip ................................................................................................................................................................................. 275 Breiderhoff Beate ....................................................................................................................................................................... 217 Bresjanac Mara ........................................................................................................................................................................... 134 Brodnik Andrej ........................................................................................................................................................................... 452 Budna Borut ............................................................................................................................................................................... 406 Burger Helena ............................................................................................................................................................................ 441 Burnard Michael ......................................................................................................................................................................... 431 Butala Peter .................................................................................................................................................................................. 13 Cankar Matija ............................................................................................................................................................................. 414 Carlin Paul .................................................................................................................................................................................. 181 Čarman Dominik ........................................................................................................................................................................ 381 Čepar Drago ............................................................................................................................................................................... 285 Čerin Matej ................................................................................................................................................................................. 185 Cheron Nicolas ............................................................................................................................................................................... 5 Chuchurski Martin...................................................................................................................................................................... 347 Cigale Matej ....................................................................................................................................................................... 107, 443 Cikajlo Imre ............................................................................................................................................................................... 441 Colnarič Matjaž .......................................................................................................................................................................... 411 Cserép Máté ............................................................................................................................................................................... 363 Cukjati Iztok ............................................................................................................................................................................... 456 Cvetkovič Tom ........................................................................................................................................................................... 381 Debeljak Marko ................................................................................................................................................................ 41, 45, 49 Depolli Matjaž ............................................................................................................................................................................ 436 Dergan Tanja .......................................................................................................................................................................... 45, 49 Detela Jernej ............................................................................................................................................................................... 411 Dovgan Erik ........................................................................................................................................................................... 9, 229 Drnovšek Mateja ........................................................................................................................................................ 385, 389, 394 Dujič Darko ................................................................................................................................................................................ 177 Džeroski Sašo ......................................................................................................................................................... 41, 57, 189, 193 Elersič Kristian ........................................................................................................................................................................... 119 Ellaia Rachid .............................................................................................................................................................................. 225 Engelhardt Meike ....................................................................................................................................................................... 107 Enova Dušan .............................................................................................................................................................................. 398 Enova Tej ................................................................................................................................................................................... 398 Fatur Katarina ............................................................................................................................................................................. 122 Fekete Anett ............................................................................................................................................................................... 363 Filipič Bogdan ............................................................................................................................................................ 217, 229, 245 Filipovič Hrast Maša .................................................................................................................................................................. 297 Fuart Flavio ................................................................................................................................................................................ 181 Fukumoto Hiroaki ...................................................................................................................................................................... 237 Gabrić Petar .................................................................................................................................................................................. 82 Galen Candace ............................................................................................................................................................................... 5 459 Gams Matjaž .................................................................................... 13, 21, 37, 103, 107, 257, 288, 385, 389, 394, 402, 406, 427 Gjoreski Hristijan ......................................................................................................................................................................... 25 Gjoreski Martin .................................................................................................................................................................... 25, 406 Glavač Aljaž ............................................................................................................................................................................... 402 Gojo David ................................................................................................................................................................................. 177 Gorenc Vujović Ana ................................................................................................................................................................... 303 Gostečnik Christian .................................................................................................................................................................... 116 Grad Janez ...................................................................................................................................................................................... 5 Gradišek Anton .............................................................................................................................................................. 5, 406, 443 Grasselli Gregor ......................................................................................................................................... 262, 266, 272, 389, 449 Grebenšek Tjaša ......................................................................................................................................................................... 310 Grobelnik Marko ........................................................................................................................................................ 157, 161, 181 Heise David .................................................................................................................................................................................... 5 Heričko Marjan .......................................................................................................................................................................... 331 Heričko Tjaša ............................................................................................................................................................................. 343 Janko Vito .................................................................................................................................................................................... 29 Jenko Miha ................................................................................................................................................................................. 153 Jerebic Sara .................................................................................................................................................................................. 75 Jovanoski Viktor ........................................................................................................................................................................ 173 Jurič Simon................................................................................................................................................................................. 411 Kalan Aleš .................................................................................................................................................................................. 418 Kališnik Jurij Matija ........................................................................................................................................................... 381, 456 Kamišalić Aida ........................................................................................................................................................................... 331 Karakatič Sašo .................................................................................................................................................................... 339, 343 Karavanić Ivor .............................................................................................................................................................................. 82 Kasesnik Karin ........................................................................................................................................................................... 293 Katrašnik Marko ........................................................................................................................................................................... 25 Kenda Klemen ............................................................................................................................................................................ 185 Kerbler Boštjan .......................................................................................................................................................................... 297 Kikaj Adem .................................................................................................................................................................................. 17 Klauser Florian ........................................................................................................................................................................... 145 Kocjančič Staša .......................................................................................................................................................................... 409 Kokol Peter ................................................................................................................................................................................. 411 Kolenik Tine ....................................................................................................................................................................... 130, 394 Kompara Tomaž ......................................................................................................................................................................... 447 Koprivec Filip ............................................................................................................................................................................ 185 Kordeš Urban ..................................................................................................................................................................... 141, 145 Korošec Peter ..................................................................................................................................................................... 241, 249 Kostovska Ana ........................................................................................................................................................................... 189 Kous Katja .................................................................................................................................................................................. 335 Kozjek Dominik ........................................................................................................................................................................... 13 Krämer Torsten .......................................................................................................................................................................... 107 Kranjc Jakša Urška ..................................................................................................................................................................... 300 Krész Miklós .............................................................................................................................................................................. 197 Kroflič Ž. .................................................................................................................................................................................... 439 Kuhar Saša ................................................................................................................................................................................. 327 Kukar Matjaž ................................................................................................................................................................................ 62 Kuzmanovski Vladimir ................................................................................................................................................................ 41 Leppäniemi Jari .......................................................................................................................................................................... 319 Lipič Viktorija ............................................................................................................................................................................ 141 London András ........................................................................................................................................................................... 197 Lukan Junoš ................................................................................................................................................................................. 25 Lukšič Nal .................................................................................................................................................................................. 452 Luštrek Mitja .............................................................................................................................................................................. 107 Luštrek Mitja ........................................................................................................................................................ 9, 25, 33, 53, 443 Luzar Anže ................................................................................................................................................................................. 414 Mahnič Blaž ............................................................................................................................................................................... 266 Malus Andreja .............................................................................................................................................................................. 13 Massri M. Besher ....................................................................................................................................................................... 161 Matjačić Zlatko .......................................................................................................................................................................... 441 460 Mesarič Katja ............................................................................................................................................................................. 398 Mladenić Dunja .......................................................................................................................................................................... 169 Mlakar Miha ......................................................................................................................................................................... 66, 443 Mlakar Nejc .................................................................................................................................................................................. 29 Moharić Metka ........................................................................................................................................................................... 134 Monteiro Wellington Rodrigo .................................................................................................................................................... 221 Moravs Stanislav ........................................................................................................................................................................ 411 Mravljak Andoljšek Veronika .................................................................................................................................................... 303 Nadásdy Zoltán ............................................................................................................................................................................ 91 Nastran Jurij ................................................................................................................................................................................. 21 Naujoks Boris ............................................................................................................................................................................. 217 Németh József ............................................................................................................................................................................ 197 Novak Blaž ................................................................................................................................................................................. 165 Novak Erik ................................................................................................................................................................................. 153 Novaković Dušanka ................................................................................................................................................................... 134 Novalija Inna ...................................................................................................................................................................... 157, 161 Orgulan Mojca ........................................................................................................................................................................... 347 Oyama Akira .............................................................................................................................................................................. 237 Panov Panče ....................................................................................................................................................................... 189, 193 Papa Gregor ........................................................................................................................................................................ 241, 249 Pataki Norbert .................................................................................................................................................................... 355, 359 Pavliha Denis ............................................................................................................................................................................. 418 Peev Gjorgi ................................................................................................................................................................................... 57 Pejović Veljko ............................................................................................................................................................................ 169 Petelin Gašper ............................................................................................................................................................................ 241 Peterlin Potisk Karmen ............................................................................................................................................................... 441 Pirtošek Zvezdan ........................................................................................................................................................................ 456 Pita Costa Joao ........................................................................................................................................................................... 181 Planinc Nataša ............................................................................................................................................................................ 418 Podgorelec Blaž.......................................................................................................................................................................... 351 Podgorelec Vili ................................................................................................................................................................... 339, 343 Poglajen Gregor ......................................................................................................................................................................... 406 Polančič Gregor .......................................................................................................................................................................... 327 Popović Bojan ............................................................................................................................................................................ 371 Porkoláb Zoltán .......................................................................................................................................................................... 367 Puh Urška ................................................................................................................................................................................... 134 Rahimi Mohammad .................................................................................................................................................................... 233 Rajšp Alen .................................................................................................................................................................................. 335 Rakić Gordana ............................................................................................................................................................................ 371 Ramovš Ana ............................................................................................................................................................................... 303 Ramovš Jože ....................................................................................................................................................................... 303, 310 Ramovš Ksenija ......................................................................................................................................................................... 303 Ramovš Marta ............................................................................................................................................................................ 303 Rant Maja ................................................................................................................................................................................... 303 Rehbach Frederik ....................................................................................................................................................................... 213 Rek Patrik ................................................................................................................................................................................... 351 Repič Slavič Tanja ..................................................................................................................................................................... 111 Reščič Nina .................................................................................................................................................................................. 53 Révész Ádám ............................................................................................................................................................................. 355 Reynoso-Meza Gilberto ............................................................................................................................................................. 221 Rifel Janez .................................................................................................................................................................................. 308 Rola Tadej .......................................................................................................................................................................... 347, 351 Rupnik Jan .................................................................................................................................................................................. 173 Rupnik Rok ................................................................................................................................................................................ 323 Rus Andrej ................................................................................................................................................................................. 452 Santanam Raghu ......................................................................................................................................................................... 181 Savadkouhi R. ............................................................................................................................................................................ 439 Šef Tomaž .................................................................................................................................................................................. 420 Sendi Richard ............................................................................................................................................................................. 297 Serrar Jihane ............................................................................................................................................................................... 225 461 Sillberg Pekka ............................................................................................................................................................................ 319 Šimenko Samo ........................................................................................................................................................................... 339 Simidjievski Nikola ...................................................................................................................................................................... 57 Simjanoska Monika .................................................................................................................................................................... 406 Simonič Barbara ........................................................................................................................................................................... 87 Širaiy Boris................................................................................................................................................................................. 424 Šircelj Beno .................................................................................................................................................................................. 21 Siserman-Gray Cristina ................................................................................................................................................................ 99 Slana Ozimič Anka..................................................................................................................................................................... 137 Slapničar Gašper ............................................................................................................................................................................ 9 Slemnik Bojan ............................................................................................................................................................................ 411 Slevec Katarina .......................................................................................................................................................................... 398 Smerkol Maj ................................................................................................................................................................................. 33 Smokvina A. ............................................................................................................................................................................... 439 Sobel Scott ................................................................................................................................................................................... 66 Sodnik Jaka ................................................................................................................................................................................ 229 Soini Jari..................................................................................................................................................................................... 319 Stanovnik Sašo ........................................................................................................................................................................... 414 Stopar Luka ................................................................................................................................................................................ 181 Stork Jörg ........................................................................................................................................................................... 213, 245 Sun Chenlu ................................................................................................................................................................................. 181 Svetelšek Ajda ............................................................................................................................................................................ 303 Szalay Richárd ........................................................................................................................................................................... 367 Talbi El-Ghazali ................................................................................................................................................................. 225, 233 Tavčar Aleš ........................................................................................................................................................................ 389, 427 Tazl Oliver August ............................................................................................................................................................. 266, 270 Tišler Aljaž ................................................................................................................................................................................. 347 Tolovski Ilin ............................................................................................................................................................................... 193 Török Márk ................................................................................................................................................................................ 359 Tošić Aleksandar ........................................................................................................................................................................ 431 Tosser Veronique ......................................................................................................................................................................... 41 Trajanoska Marija ........................................................................................................................................................................ 25 Trajanov Aneta ................................................................................................................................................................. 41, 45, 49 Trobec Roman .................................................................................................................................................................... 398, 424 Turčin Marko ............................................................................................................................................................................. 411 Turkanović Muhamed ........................................................................................................................................................ 331, 347 Tušar Tea .................................................................................................................................................................................... 209 Unger Tea ................................................................................................................................................................................... 347 Urbančič Jasna ................................................................................................................................................................... 153, 169 Urbančič Maša ........................................................................................................................................................................... 126 Valič Jakob ................................................................................................................................................................................. 435 Vičič Jernej ................................................................................................................................................................................. 431 Vidmar Gaj ................................................................................................................................................................................. 134 Vilhar Andrej ............................................................................................................................................................................. 436 Vitorovič M. ............................................................................................................................................................................... 439 Vodeb Aljaž ............................................................................................................................................................................... 347 Vodopija Aljoša ......................................................................................................................................................................... 245 Vrabič Rok ................................................................................................................................................................................... 13 Vukčević Marković Maša ............................................................................................................................................................ 79 Wallace Jonathan........................................................................................................................................................................ 181 Welzer Tatjana ........................................................................................................................................................................... 331 Wotawa Franz .................................................................................................................................................................... 270, 275 Y. Fakhreddine Rawan ................................................................................................................................................................. 91 Zapušek Tjaša ............................................................................................................................................................................... 93 Završnik Jernej ........................................................................................................................................................................... 411 Završnik Tadej ........................................................................................................................................................................... 411 Zentel Peter ................................................................................................................................................................................ 107 Žibert Janez ................................................................................................................................................................ 381, 452, 456 Živković Nenad .......................................................................................................................................................................... 452 Žlahtič Gregor ............................................................................................................................................................................ 411 462 Žnidar Žan .................................................................................................................................................................................. 347 Zorman Milan ............................................................................................................................................................................. 411 Zupančič Jernej .................................................................................................................................................. 257, 262, 266, 402 Zupanič Katja ............................................................................................................................................................................... 91 463 464 Konferenca / Conference Uredili / Edited by Slovenska konferenca o umetni inteligenci Slovenian Conference on Artificial Intelligence Mitja Luštrek, Rok Piltaver, Matjaž Gams Kognitivna znanost Cognitive Science Toma Strle, Olga Markič Odkrivanje znanja in podatkovna skladišča - SiKDD Data Mining and Data Warehouses - SiKDD Dunja Mladenić, Marko Grobelnik Mednarodna konferenca o visokozmogljivi optimizaciji v industriji, HPOI 2018 International Conference on High-Performance Optimization in Industry, HPOI 2018 Bogdan Filipič, Thomas Bartz-Beielstei n Delavnica AS-IT-IC AS-IT-IC Workshop Matjaž Gams, Jernej Zupančič Soočanje z demografskimi izzivi Facing Demographic Challenges Janez Malačič, Matjaž Gams Sodelovanje, programska oprema in storitve v informacijski družbi Collaboration, Software and Services in Information Society Marjan Heričko Delavnica za elektronsko in mobilno zdravje ter pametna mesta Workshop Electronic and Mobile Health and Smart Cities Matjaž Gams, Aleš Tavčar Document Outline 02 - Naslovnica - notranja - All - brez H (ITTC) 03 - Kolofon - All - brez H (ITTC) 04 - 05 - IS2018 - Predgovor in odbori 07 - Kazalo - All - brez H (ITTC) Vsebina - A 01 - Naslovnica-sprednja-A 02 - Naslovnica - notranja - A 03 - Kolofon - A 04 - 05 - IS2018 - Skupni del 07 - Kazalo - A 08 - Naslovnica podkonference - A 09 - Predgovor podkonference - A 10 - Programski odbor podkonference - A 11 - Clanki - A 01 - Gradisek 02 - Slapnicar 03 - Malus - po recenziji 04 - Kikaj - po recenziji - ver3 05 - Nastran 06 - Junos Introduction Context-Aware Stress Detection Method Lab Stress Detection Activity Recognition Context-Based Stress Detection AWARE Implementation Empatica Data Streaming Plugin Stress Detection Plugin Further Work and Conclusions References 07 - Janko 08 - Smerkol 09 - Bizjak - po recenziji - 2ver 10 - Debeljak - po recenziji 11 - Trajanov - po recenziji 12 - Dergan - po recenziji - ver3 13 - Rescic - po recenziji - 2ver 14 - Peev - po recenziji - 2ver 15 - Kukar 16 - Mlakar 12 - Index - A 13 - Naslovnica-zadnja-A Blank Page Blank Page Blank Page Blank Page Blank Page Blank Page 04 - 05 - IS2018 - Predgovor in odbori.pdf 04 - IS2018 - Predgovor 05 - IS2018 - Konferencni odbori 04 - 05 - IS2018 - Predgovor in odbori.pdf 04 - IS2018 - Predgovor 05 - IS2018 - Konferencni odbori Vsebina - B 01 - Naslovnica-sprednja-B 02 - Naslovnica - notranja - B 03 - Kolofon - B 04 - 05 - IS2018 - Skupni del 07 - Kazalo - B 08 - Naslovnica podkonference - B 09 - Predgovor podkonference - B 10 - Programski odbor podkonference - B 11 - Clanki - B 01 - Jerebic-Razumevanje odraslih partnerskih odnosov v luči spolne zlorabe v otroštvu 02 - Vukčević Marković-Do healthy participants make advantageous decisions Evidence from the Iowa Gambling Task_final 03 - Gabrić, Banda & Karavanić- Palaeolithic toolmaking and the evolution of cognition and language 04 - Simonič-Psihonevrobiološka razvojna perspektiva razumevanja nasilja 05 - Zupanič et al-Emergence of Visual Consciousness in ADHD Children 06 - Zapušek-European Legal Regulation of Self-learning Systems 07 - Siserman Gray-Legal Practitioners’ Views on Neuroscientific Evidence in Court Proceedings 08 - Gams-From AI to our future I 09 - Cigale et al_The Quest for understanding-Helping people with PMLD to communicate with their caregivers 10 - Repič Slavič-IJS Kognitivna raven kot del rdt pri predelovanju spolnega nasilja 11 - Gostečnik-Medgeneracijski prenos nasilja 12 - Elersič-The other side of neurotechnology 13 - Fatur-Braincrafting 14 - Urbancic-In search of the authentic self 15 - Kolenik-Exploring Features of Cognitive Science as Natural Epistemology 16 - Novakovič et al.-Učinki telesne poze na prag termične bolečine za vroče 17 - Slana-The effect of distractors in lateralized change detection working memory task 18 - Kordeš, Lipič-Zmota luči v hladilniku in značilnosti opazovanja fenomenov z obrobja zavesti 19 - Klauser Kordeš zemljevid prepričanj 12 - Index - B 13 - Naslovnica-zadnja-B Blank Page Blank Page Blank Page Blank Page Blank Page 04 - 05 - IS2018 - Predgovor in odbori.pdf 04 - IS2018 - Predgovor 05 - IS2018 - Konferencni odbori Vsebina - C 01 - Naslovnica-sprednja-C 02 - Naslovnica - notranja - C 03 - Kolofon - C 04 - 05 - IS2018 - Skupni del 07 - Kazalo - C 08 - Naslovnica podkonference - C 09 - Predgovor podkonference - C 10 - Programski odbor podkonference - C 11 - Clanki - C 01 - NovakErik Abstract 1 Introduction 2 Related Work 3 Data Preprocessing 4 Recommender Engine 4.1 Recommendation Results 5 Future Work and Conclusion Acknowledgments References 02 - Novalija 1. INTRODUCTION 2. BACKGROUND The development of smart labour market statistics touches a number of issues from labour market policies area and would provide contributions to questions related to: - job creation, - education and training systems, - labour market segmentation, - improving skill supply and productivity. For instance, the analysis of the available job vacancies could offer an insight into what skills are required in the particular area. Effective trainings based on skills demand could be organized and that would lead into better labour market integrat... A number of stakeholder types will benefit from the development of smart labour market statistics. In particular, the targeted stakeholders are: 3. RELATED WORK The European Data Science Academy (EDSA) [1] was an H2020 EU project that ran between February 2015 and January 2018. The objective of the EDSA project was to deliver the learning tools that are crucially needed to close the skill gap in Data Science ... - Analyzed the sector specific skillsets for data analysts across Europe with results reflected at EDSA demand and supply dashboard; - Developed modular and adaptable curricula to meet these data science needs; and - Delivered training supported by multiplatform resources, introducing Learning pathway mechanism that enables effective online training. 4. PROBLEM DEFINITION 4.1 DATA SOURCES 4.2 CONCEPTUAL ARCHITECTURE 4.3 SCENARIOS 4.3.1 DEMAND ANALYSIS 4.3.2 SKILLS ONTOLOGY DEVELOPMENT 4.3.3 SKILLS ONTOLOGY EVOLUTION 5. STATISTICAL INDICATORS 6. CONCLUSION AND FUTURE WORK 7. ACKNOWLEDGMENTS 8. REFERENCES 03 - Massri 1. INTRODUCTION 2. RELATED WORK 3. DESCRIPTION OF DATA 4. METHODOLOGY 4.1 Clustering and Formatting Data 4.2 Choosing the Main Entities 4.3 Detecting the Characteristics of Relationship 5. VISUALIZING THE RESULTS 5.1 Characteristics of the Main Graph 5.2 Main Functionality 5.3 Displaying Relation Information 6. CONCLUSION AND FUTURE WORK 7. ACKNOWLEDGMENTS This work was supported by the euBusinessGraph (ICT-732003-IA) project [6]. 8. REFERENCES 04 - NovakBlaz 1. INTRODUCTION 2. EXPERIMENTAL SETUP 3. RESULTS 4. CONCLUSIONS AND FUTURE WORK 5. ACKNOWLEDGEMENTS 6. REFERENCES 05 - Urbancic Introduction Related work Proposed approach Results Conclusions Acknowledgments References 06 - Jovanoski 07 - Gojo 08 - PitaCosta 09 - Koprivec Introduction PerceptiveSentinel Platform Data Data Acquisition Data Preprocessing Methodology Sample Data Feature Vectors Experiment Results Conclusions Acknowledgments References 10 - Kostovska 11 - Tolovski 12 - London 12 - Index - C 13 - Naslovnica-zadnja-C Blank Page Blank Page Blank Page Blank Page Blank Page Blank Page 04 - 05 - IS2018 - Predgovor in odbori.pdf 04 - IS2018 - Predgovor 05 - IS2018 - Konferencni odbori Vsebina - D 02 - Naslovnica - notranja - D 03 - Kolofon - D 04 - IS2018 - Predgovor 05 - IS2018 - Konferencni odbori 07 - Kazalo - D 08 - Naslovnica podkonference - D 09 - Predgovor podkonference - D 10 - Programski odbor podkonference - D 11 - Clanki - D 01_HPOI_2018_paper_8 02_HPOI_2018_paper_3 03_HPOI_2018_paper_7 04_HPOI_2018_paper_4 05_HPOI_2018_paper_11 06_HPOI_2018_paper_2 07_HPOI_2018_paper_1 08_HPOI_2018_paper_5 09_HPOI_2018_paper_6 10_HPOI_2018_paper_10 11_HPOI_2018_paper_9 12 - Index - D Blank Page Blank Page Blank Page Blank Page Blank Page Blank Page Blank Page Blank Page 04 - 05 - IS2018 - Predgovor in odbori.pdf 04 - IS2018 - Predgovor 05 - IS2018 - Konferencni odbori Vsebina - E 01 - Naslovnica-sprednja-E 02 - Naslovnica - notranja - E 03 - Kolofon - E 04 - 05 - IS2018 - Skupni del 07 - Kazalo - E 08 - Naslovnica podkonference - E 09 - Predgovor podkonference - E 10 - Programski odbor podkonference - E 11 - Clanki - E 01_Zupancic-Gams_AS-IT-IC-report 02_Grasselli-Zupancic_Tourism-tools 03_Zupancic-Tazl-Mahnic-Grasselli_AS-IT-IC-databases.pdf 04_Tazl-Wotawa_Content-API.pdf 05_Grasselli_e-Tourist2 Introduction Why a New System Features Trip planning Recommendation Full text search e-Tourist2.0 architecture Conclusions Acknowledgments References 06_Bozic-Wotawa_Security-testing-chatbots 12 - Index - E 13 - Naslovnica-zadnja-E Blank Page Blank Page Blank Page Blank Page Blank Page 04 - 05 - IS2018 - Predgovor in odbori.pdf 04 - IS2018 - Predgovor 05 - IS2018 - Konferencni odbori Vsebina - F 01 - Naslovnica-sprednja-F 02 - Naslovnica - notranja - F 03 - Kolofon - F 04 - 05 - IS2018 - Predgovor in odbori 04 - IS2018 - Predgovor 05 - IS2018 - Konferencni odbori 07 - Kazalo - F 08 - Naslovnica - notranja - F 09 - Predgovor podkonference - F 10 - Programski odbor podkonference - F 11 - Clanki - F 01 - 2 - Čepar v2 02 - 3 - Gams popravljen ver2 03 - 5 - Kasesnik - 280918 – ZaOddajo 04 - 6 - Kerbler 05 - 7 - Kranjc Jaksa 06 - 10 - Ramovš_formatted 07 - 11 - Rifel 08 - 9 - Ramovš Jože et al 12 - Index - F 13 - Naslovnica-zadnja-F Blank Page Blank Page Blank Page Blank Page Blank Page Blank Page Blank Page Blank Page Vsebina - G 01 - Naslovnica-sprednja-G 02 - Naslovnica - notranja - G 03 - Kolofon - G 04 - 05 - IS2018 - Skupni del 07 - Kazalo - G 08 - Naslovnica podkonference - G 09 - Predgovor podkonference - G 10 - Programski odbor podkonference - G 11 - Clanki - G 01_Soini 02_Rupnik 03_Kuhar 04_Kamisalic Introduction Methodology Experimental framework Experimental instruments Results and discussion Knowledge perception Knowledge perception and notation Conclusions Acknowledgments References 05_Rajsp 06_Simenko 07_Hericko 08_Vodeb 09_Podgorelec 10_Revesz 11_Torok 12_Fekete 13_Szalay Abstract 1 Motivation 2 C++ Special Member Functions 2.1 Constructors 2.2 Destructor 2.3 Assignment operators 2.4 Members for move semantics 3 Implementation 3.1 Syntax transliteration 3.2 Special member overview 4 Conclusion Acknowledgments References 14_Popovic 12 - Index - G 13 - Naslovnica-zadnja-G Blank Page Blank Page Blank Page Blank Page Blank Page Blank Page Blank Page Blank Page 04 - 05 - IS2018 - Predgovor in odbori.pdf 04 - IS2018 - Predgovor 05 - IS2018 - Konferencni odbori 11 - Clanki - G.pdf 01_Soini 02_Rupnik 03_Kuhar 04_Kamisalic Introduction Methodology Experimental framework Experimental instruments Results and discussion Knowledge perception Knowledge perception and notation Conclusions Acknowledgments References 05_Rajsp 06_Simenko 07_Hericko 08_Vodeb 09_Podgorelec 10_Revesz 11_Torok 12_Fekete 13_Szalay Abstract 1 Motivation 2 C++ Special Member Functions 2.1 Constructors 2.2 Destructor 2.3 Assignment operators 2.4 Members for move semantics 3 Implementation 3.1 Syntax transliteration 3.2 Special member overview 4 Conclusion Acknowledgments References 14_Popovic Vsebina - I 02 - Naslovnica - notranja - I 03 - Kolofon - I 04 - 05 - IS2018 - Skupni del 07 - Kazalo - I 08 - Naslovnica podkonference - I 09 - Predgovor podkonference - I 10 - Programski odbor podkonference - I 11 - Clanki - I 01 - Carman Cvetkovic Avbelj 02 - Drnovsek Gams 03 - Drnovsek Gams 04 - Drnovsek Kolenik Gams 05 - Enova 06 - Glavac Zupancic Gams 07 - Gradisek 08 - Kocjancic 09 - Kokol 10 - Luzar Introduction Data storage in edge devices Sensors, edge devices and cloud storage The performance degradation issue Benefits of using relational or time series databases and their comparison Criteria for filtering the databases Relational database selection process Time series database selection process Experiment and results The test between time series and relational databases Measurement environments IoT characteristics for databases The experiment metrics The results Database setup times and resource consumption Conclusion Acknowledgments References 11 - Pavliha 12 - Šef 13 - Siraiy Trobec 14 - Tavcar 15 - !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Blank Page Blank Page Blank Page Blank Page 16 - Valic 17 - Vilhar Depolli 18 - Vitorovic Kroflic Bobnar CosyLab Marand 19 - Cikajlo Matjacic Burger PetelinPotisk 20 - Cigale Gradisek Mlakar Lustrek 21 - Kompara 22 - Grasselli 23 - Brodnik 24 - Cukjati 12 - Index - I Blank Page Blank Page Blank Page Blank Page Blank Page Blank Page 04 - 05 - IS2018 - Predgovor in odbori.pdf 04 - IS2018 - Predgovor 05 - IS2018 - Konferencni odbori 12 - Index - All - brez H (ITTC) Blank Page Blank Page Blank Page Blank Page Blank Page Blank Page