77 2591-2259 / This is an open access article under the CC-BY-NC-ND license https://creativecommons.org/licenses/by-nc-nd/4.0/ DOI: 10.17573/cepar.2023.2.04 1.01 Original scientific article Analysis of Research on Artificial Intelligence in Public Administration: Literature Review and Textual Analysis Nejc Lamovšek Educational Research Institute, Slovenia nejc.lamovsek@pei.si https://orcid.org/0000-0003-3528-0527 Received: 28. 8. 2023 Revised: 25. 10. 2023 Accepted: 20. 11. 2023 Published: 30. 11. 2023 ABSTRACT Purpose: This study aims to investigate how analysing academic research through digital tools can improve our understanding of the applications, functions, and challenges related to the use of advanced artificial tech- nologies (AI) in public administration. Methodology: The applied methodology relies on the use of digital tools, specifically Voyant-Tools and Chat Generative Pre-Trained Transformer (GPT-4), for text analysis in conjunction with a selection of scientific lit- erature on artificial intelligence and public administration. Findings: The results of our study show that researchers equally report advantages and disadvantages of using AI in public administration. More- over, the research highlights the benefits of using artificial intelligence while emphasising the importance of the ethical and appropriate regula- tion thereof. Practical implications: Our innovative approach of developing and using a combined methodology involving specialised digital tools to analyse scientific literature introduces a new dimension to the examination of scientific texts and has the potential to shape public policy in the field of public administration. Originality: The existing body of research on public administration and artificial intelligence is limited. Our study expands the scientific field by delving into the use of artificial intelligence in public administration. Keywords: digital tools, artificial intelligence, GPT-4, public administration, regu- lation JEL: Z18 Lamovšek, N. (2023). Analysis of Research on Artificial Intelligence in Public Administration: Literature Review and Textual Analysis. Central European Public Administration Review, 21(2), pp. 77–96 Central European Public Administration Review, Vol. 21, No. 2/202378 Nejc Lamovšek 1 Introduction In the age of quickly developing artificial intelligence (AI), it is imperative that the progress of science in this field is accompanied by the progress of AI in the domain of public administration. Xu et al. (2019) defined AI as the simula- tion of human intelligence with a specific system or machines with the aim of mimicking human thinking and behaviour. AI subareas defined by Vijayakumar and Sheshadri (2019) include expert systems, natural language processing, pattern recognition, robotics, and machine learning, among others. Ahn (2023) explained that the essence of AI lies in large language models (LLMs). LLMs are machine-learning models trained on vast text collections and operate by predicting the most likely word to follow a given sequence of text. With their help, processing similar to natural language can be performed, which includes tasks such as translation, analysis, summarization, and proof- reading. LLMs analyse and comprehend the connections between words and concepts, enabling them to follow a logical sequence of ideas. Moshirfar et al. (2023) noted that Chat Generative Pre-Trained Transformer (ChatGPT) is among the most well-known natural language processing (NLP) models that is trained on large language databases. The latest available version of Chat- GPT is Version 4 (ChatGPT-4), which has shown great efficiency and accuracy of results compared to previous versions, even from the perspective of un- derstanding the context of analysed texts. However, software applications available prior to ChatGPT also can perform a general analysis of texts. One such program is Voyant Tools. Alhudithi (2021) explained that Voyant Tools uses computer algorithms to obtain the required information from the text. The areas of use for the Voyant Tools program were defined by Gregory et al. (2022) as the identification of terms that most frequently appear in the text, the occurrence of other terms in connection with the most commonly used terms, visualization of results, and the occur- rence of terms that connote positively or negatively.1 With certain fine adjust- ments, we can also customize the obtained results. Gesk and Leyer (2022) noted the growing interest in utilizing AI-based soft- ware in the public sector, as well. However, research findings from the pri- vate sector cannot be directly applied to the public sector due to disparities in citizens’ perception of services. AI holds significant potential for enhancing public services, primarily in terms of efficiency and service quality, but con- cerns regarding its growth and application remain a significant obstacle to its adoption. Similarly, Androniceanu (2023) found that digitalization and the use of AI can visibly improve managerial and economic efficiency in public admin- istration. Štefanišinová et al. (2021) added that individual AI tools are still in the development phase but already offer substantial advantages in providing public services that will further improve in the future. However, in the use of 1 “Generally words can be used for positive or negative connotations depending on the contex- tual situation. The usage of words may be good or bad sense, impression, experience, feeling, etc. For example, politicians and advertisers may prefer words with positive connotations in view of expressing their message attractively. In case of unpleasant feeling, a word with neg- ative connotations may be used to describe them” (Rao, 2017). Central European Public Administration Review, Vol. 21, No. 2/2023 79 Analysis of Research on Artificial Intelligence in Public Administration: Literature Review and Textual Analysis advanced analytics, Simonofski et al. (2022) emphasized the importance of re- specting the protection of personal data and human verification of decisions made by AI technologies. The primary objective of the current study was to understand how analysing academic research through digital tools may improve our view into the ap- plications, functions, and issues related to the use of advanced artificial tech- nologies in public administration. We next provide a literature review of the analysed articles. Thereafter, we discuss the selection and advantages of our employed methodology and explain the analytical procedures we used. In the following Results section, we cover the outcomes of literature identification and selection as well as graphical and visual representations of the findings. We then further elabo- rate upon and compare these results with the outcomes of other comparable studies in the Discussion section. Finally, we provide brief conclusions into es- sential insights into the broader applicability of the results obtained, highlight the weaknesses of our research, and show opportunities for further develop- ment of the study. 2 Literature Review of Selected Articles In our brief review of the scientific literature, we primarily focused on the scientific contributions that were discussed and analysed in our research. A brief literature review is a collaborative work of the authors with ChatGPT-4, in which the authors sought a deeper meaning of the analyzed text in accord- ance with the goals of our research. This may, to a certain extent, differ from the intentions of the original authors of the analyzed texts. ChatGPT-4 can hal- lucinate, i.e., cosmetically re-interpret the actual state – a result that must be considered in further interpretations of literature summaries. For one, Wirtz and Müller (2019) discussed the use of AI in public administration through a conceptual study. In their research, they developed an integrated AI frame- work for public management that encompassed all crucial aspects (regula- tion, and ethical and political guidelines), goal of using AI is achieving greater efficiency in public management. Subsequently, Wirtz et al. (2021) conducted a systematic review of the literature in the field of AI in the public sector. Utilizing qualitative and quantitative approaches, they analysed 189 articles. They further performed a methodological classification of articles and ana- lysed the risks and benefits of using AI in the public sector. They discovered an extremely heterogeneous research area that is methodologically unbal- anced and thus proposed more empirical and in-depth studies on the use of AI in the public sector, they anticipate a larger number of empirical data and preposed more in-depth studies of the use of AI in public sector. Previously, Wirtz et al. (2020) had proposed an integrated AI governance framework by considering the interactions of AI challenges, previous regulations and public administration. We can conclude that when balancing risks with the benefits of AI regulation, all stakeholders’ viewpoints should be taken into account for optimal results for the society. Wirtz and Müller (2023) further expanded Central European Public Administration Review, Vol. 21, No. 2/202380 Nejc Lamovšek their research by questioning the development of modelling smart cities and technological interactions of its stakeholders and the use of technologies. They used a literature review, which was rather complex. In the study, they noted that technological city governance can improve efficiency in resource use and enhance the quality of life for citizens. However, they also noted that despite technological progress, traditional governance mechanisms will not become redundant and will be important to balance the weaknesses of smart technological governance. Moreover, Wirtz et al. (2019) explored the use of AI in the public sector and defined the possibilities and challenges in using AI. They conducted a literature review with a selection of keywords that defined the use of AI and challenges in the public sector. Their results identified 10 AI application areas in the public sector and defined four main dimensions of challenges in using AI, the primary of which were how to ensure regulation of AI, how to use it in an ethically acceptable way, and what the impact of AI on a society as a whole will be. In the specialized field of the public sector, namely in the provision of health and social services, Štefanišinová et al. (2021) investigated the use of AI. Utiliz- ing a comparative approach and case analysis, they acquired a realistic assess- ment of current AI technologies and anticipated levels of AI. They emphasized that using AI could both improve and challenge the way healthcare and social services are provided, but the main goal should be to make life better for people. Among the principal challenges of using AI are the utilization of data and the potential reduction of jobs due to task optimization. Simonofski et al. (2022) also scrutinized the legal requirements od data pro- tection in public administration in the area of fraud analytics with advanced technologies. They examined two case studies and summarized 15 different governance practices in this field. They accentuated the complexity of the in- tegration a legal requirements that would be in line with advanced analytics. A further challenge in employing AI technologies represents the protection of personal data and the application of administrative law. Furthermore, Busuioc (2021) researched the use of AI algorithms in the public sector in connection with ensuring accountability for decisions. The author employed conceptual analysis to diagnose and analyse the challenges and re- sponsibilities associated with the use of AI (accountability) in the public sector and underscored the importance of interpretable and transparent AI models. Additionally, McDonald et al. (2022) addressed the area of research and the future of research in public administration. They reflected on the state of pub- lic administration research and analysed the methodology used in that field. They noted that technology could significantly impact how governments re- spond to emerging changes, which could represent a further potential area of research. Vogl et al. (2020) also explored the use of smart technologies at the local level of public administration. In their study, they used questionnaires and inter- views with employees in local public administration. They found that the use Central European Public Administration Review, Vol. 21, No. 2/2023 81 Analysis of Research on Artificial Intelligence in Public Administration: Literature Review and Textual Analysis of smart technologies in local public administration was on the rise. Smart technologies are becoming part of the process in providing public services, which may result in certain changes. Moreover, Terzidou (2022) investigated the use of AI in the judiciary. Specifi- cally, she examined the transition from the use of information and communica- tion technologies (ICT) to the use of AI. Although she emphasized that the use of AI may present certain benefits, especially in terms of improved efficiency and better accessibility of judicial services, she also mentioned disadvantages that need to be addressed with the implantation of regulatory rules. The key risks are primarily related to the independence and impartiality of the courts. In another study, Bodó and Janssen (2022) explored the impact of private technological systems in the public sector on citizens’ trust in the govern- ment. They conducted a critical assessment and analysis of various aspects of technology and trust in the public sector. They found that when technology fails, it can significantly influence citizens’ trust in the state. In their study, AI was mainly understood in connection with the use of technology. Furthermore, Hartmann and Wenzelburger (2021) investigated the applica- tion of algorithms and computer models to support decision-making process- es within the U.S. public administration – criminal justice. They conducted a case study based on the review of primary sources and interviews with ex- perts. The results of the study showed that decision-making with the help of algorithmic methods was popular mainly because of the prior uncertainty of outcomes, the consequent dispersion of responsibility for negative con- sequences of decisions, because of the help of algorithmic methods, it is not such an important factor. Thus, the careful consideration of legal, social, and ethical aspects is important when using AI systems. Grimmelikhuijsen (2021) also explored the effect of algorithms on perceived trust in automated decision-making. Experimental testing of two scenarios showed that explainability of algorithmic decision-making is more important for trust. Similarly, Wenzelburger et al. (2022) examined how people accept algorithms used in the public sector and issue of context. They conducted two case studies with surveys completed by over 2,600 people from Germa- ny. Their research results also indicated that people accept algorithms more if they are solving a problem of personal importance and if they trust the or- ganization using them. Moreover, Giest and Klievink (2022) analysed two cases and explored the in- fluence of AI on bureaucrats roles within public administration in different organizational contexts by focusing mainly on the impact of AI on innovation in the public sector. They found that there was pronounced administrative- process innovation, in other words, a change in the organizational structure and tasks of employees. In one case, there was also conceptual innovation be- cause the AI system handled a specific task faster, more accurately, and more efficiently than a human could have. Central European Public Administration Review, Vol. 21, No. 2/202382 Nejc Lamovšek The interaction between humans and AI in decision-making in the public sec- tor was also studied by Alon-Barkat and Busuioc (2023) in three experimental studies. Their research primarily dealt with the aspect of automating bias and selective adherence to decisions and advice from AI or algorithms when they align with group stereotypes. They emphasised understanding the function- ing of algorithmic shortcomings when used to assist decision-making for al- ready vulnerable and disadvantaged citizens in the public sector. Additionally, Pencheva et al. (2020), through a review of scientific literature in the field of public administration, investigated the transformational impact of big data and AI on governance around the world. They observed a benefits of big data and AI on policy cycle, especially in terms of increasing accuracy, efficiency, and speed of the policy-making process due to Big Data – AI usage. Similarly, Castelnovo and Sorrentino (2021) addressed the impacts of big data and AI on government role in their research, which used a conceptual approach. They noted that big data can help to achieve significant improve- ments in policymaking and the provision of public services. However, we think that governments need to be careful and plan ahead when dealing with the issues of Big Data and AI. With this critical evaluation of existing literature, we can improve our under- standing of the gaps and shortcomings in this subject. 3 Methods We methodologically designed this study as an identification and selection of scientific literature in the field of AI in public administration, a literature review of analysed articles with an emphasis on the research area under in- vestigation (presented in the literature overview of selected articles), a deter- mination of the most frequently used terms in the corpus of articles, and an identification of the positive and negative connotations of the article’s texts. We focused on using the Web of Science (WoS) database because our aim was to demonstrate the utility of digital tools in analysing a limited set of scien- tific articles indexed in one of the major databases. We also decided to limit the data to the last 5 years, which enabled us to analyse the latest research trends and developments in the selected period. This approach ensured the relevance and timeliness of the acquired data while also allowing for the ef- fective use of digital tools for analysis and interpretation of the gathered in- formation. Articles that met all the criteria below (applied to the WoS data- base) and were freely accessible (accessed through the Educational Research Institute network) in the WoS database or from the publisher were used for further text analysis: – Keywords: artificial intelligence and public administration in all WoS data- bases (ALL); – Publication years: 2018–2022; Central European Public Administration Review, Vol. 21, No. 2/2023 83 Analysis of Research on Artificial Intelligence in Public Administration: Literature Review and Textual Analysis – Languages: English; – Countries/regions: EU; – Document types: Article or review article; – WoS categories: Public administration. There are various digital tools available for analyzing scientific texts, includ- ing fairly traditional ones such as VOSviewer (VOSviewer, 2023) and those used for classic quantitative bibliographic analysis. Our objective was to visu- alize text with a deep understanding of semantics and content analysis, so we sought tools that were generally accessible, free of charge, user friendly, and required less technical expertise compared to, for example, NVivo and Bibli- oshiny. Consequently, we determined Voyant Tools (Voyant Tools, 2023) and ChatGPT-4 (OpenAI, 2023b) to be suitable choices. The 19 scientific articles, which were freely accessible, were analysed using Voyant Tools. Voyant Tools is a freely accessible, web-based program for tex- tual analysis of text. With Voyant Tools, it is possible to analyse documents in different languages because it supports analysis in any language because it operates on character sequences (Voyant Tools, 2023). Our analysis encom- passed the entire content of each article. We first identified 25 terms that appeared most frequently across the entirety of the article corpus, namely, the keywords. During the analysis, we excluded irrelevant words, and, among the first 25 terms, we tried to combine words with the same roots. We also determined the frequency of occurrence of the first 25 individual terms that denoted positive and negative connotations. The frequency of occurrence of positive and negative connotations was determined using the word base of Voyant Tools. Determining the positive and negative connotations in texts is important because it can help identify the authors’ and the scientific commu- nity’s perspectives on a specific topic, for example, perception and potential receptiveness to AI technologies. In ChatGPT-4, using the appropriate and available PDF plugin (Ai PDF), we analysed-interact with 19 freely accessible scientific articles. Furthermore, we created and used prompts2 following the principles outlined in “Best Prac- tices for Prompt Engineering with OpenAI API” (OpenAI, 2023a). According to these guidelines, prompts should be concise, precise, and elaborative. The analysis using prompts was divided into two steps: 1. Searching for the most frequent terms in the corpus of scientific articles. Example prompt: Analyse the entire content of the provided scientific ar- ticles in PDF format and identify the five most frequently occurring terms, focusing on key terminology. Please ensure that the results are based sole- ly on the actual content of the articles, without conjecture or fabrication of data. 2. Using prompts that focused on determining the general connotation of the entire corpus of text. Example prompt: Determine the general text 2 As Reynolds and McDonell (2021) pointed out, a prompt is an instruction to the GPT chat on how to perform a specific task. Central European Public Administration Review, Vol. 21, No. 2/202384 Nejc Lamovšek connotation of the entire content of the provided scientific articles in PDF format, analysing the overall tone, sentiment, and thematic elements. En- sure that the interpretation is based strictly on the content provided, with- out any conjecture or fabricated results. 4 Results 4.1 Results of Literature Identification and Selection Only 22 scientific articles met the search criteria in WoS, and they are present- ed in Table 1. In terms of WoS categories, all 22 of the articles fell within the category of public administration. Six were also categorized under political science, two under management, and one under social sciences interdiscipli- nary. In relation to research areas, all 22 articles related to public administra- tion, six to government law, two to business economics, and one to social sciences other topics. From the subsequent textual analysis, we excluded three scientific articles due to lack of public accessibility, leaving a total of 19 articles for textual analysis. Central European Public Administration Review, Vol. 21, No. 2/2023 85 Analysis of Research on Artificial Intelligence in Public Administration: Literature Review and Textual Analysis Ta b le 1 . R es ul ts o f Li te ra tu re Id en ti fi ca ti o n an d S el ec ti o n A ut ho rs A rt ic le T it le So ur ce T it le P ub lic at io n Y ea r Ea rl y A cc es s D at e W ir tz , B . W ., & M ül le r, W . M . A n In te gr at ed A rt ifi ci al In te lli ge nc e Fr am ew o rk f o r P ub lic M an ag em en t Pu bl ic M an ag em en t R ev ie w 20 19 W ir tz , B . W ., W ey er er , J . C ., & St ur m , B . J . Th e D ar k Si d es o f A rt ifi ci al In te lli ge nc e: A n In te gr at ed A I G o ve rn an ce F ra m ew o rk f o r P ub lic A d m in is tr at io n In te rn at io na l J ou rn al o f Pu bl ic A dm in is tr at io n 20 20 St ef an is in o va , N ., M ut ho va , N . J ., St ra ng fe ld o va , J ., & Su la jo va , K . Im p le m en ta ti o n an d A p p lic at io n o f A rt ifi ci al In te lli ge nc e in S el ec te d P ub lic S er vi ce s C ro at ia n an d C om pa ra ti ve Pu bl ic A dm in is tr at io n 20 21 W ir tz , B . W ., La ng er , P . F ., & Fe nn er , C . A rt ifi ci al In te lli ge nc e in t he P ub lic S ec to r - A R es ea rc h A ge nd a In te rn at io na l J ou rn al o f Pu bl ic A dm in is tr at io n 20 21 8/ 20 21 W ir tz , B . W ., W ey er er , J . C ., & G ey er , C . A rt ifi ci al In te lli ge nc e an d t he P ub lic S ec to r- A p p lic at io ns a nd C ha lle ng es In te rn at io na l J ou rn al o f Pu bl ic A dm in is tr at io n 20 19 Si m o no fs ki , A ., To m b al , T ., D e Te rw an gn e, C ., W ill em , P ., Fr en ay , B ., & J an ss en , M . B al an ci ng F ra ud A na ly ti cs W it h Le ga l R eq ui re m en ts : G o ve rn an ce P ra ct ic es a nd T ra d e- O ff s in P ub lic A d m in is tr at io ns D at a & P ol ic y 20 22 B us ui o c, M . A cc o un ta b le A rt ifi ci al In te lli ge nc e: H o ld in g A lg o ri th m s to A cc o un t Pu bl ic A dm in is tr at io n R ev ie w 20 21 11 /2 02 0 M cD o na ld , B . D ., H al l, J. L ., O ’F ly nn , J ., & T hi el , S . Th e Fu tu re o f P ub lic A d m in is tr at io n R es ea rc h: A n Ed it o r’ s P er sp ec ti ve Pu bl ic A dm in is tr at io n 20 22 1/ 20 22 V o gl , T . M ., Se id el in , C ., G an es h, B ., & B ri gh t, J . Sm ar t Te ch no lo gy a nd t he E m er ge nc e o f A lg o ri th m ic B ur ea uc ra cy : A rt ifi ci al In te lli ge nc e in U K L o ca l A ut ho ri ti es Pu bl ic A dm in is tr at io n R ev ie w 20 20 Te rz id o u, K . Th e U se o f A rt ifi ci al In te lli ge nc e in t he J ud ic ia ry a nd It s C o m p lia nc e w it h th e R ig ht t o a F ai r Tr ia l Jo ur na l o f Ju di ci al A dm in is tr at io n 20 22 B o d o , B ., & J an ss en , H . M ai nt ai ni ng T ru st in a T ec hn o lo gi ze d P ub lic S ec to r Po lic y an d So ci et y 20 22 5/ 20 22 H o ff m an , I ., & K ar p iu k, M . E- A d m in is tr at io n in P o lis h an d H un ga ri an M un ic ip al it ie s - A C o m p ar at iv e A na ly si s o f th e R eg ul at o ry Is su es Le x Lo ca lis -J ou rn al o f Lo ca l Se lf -G ov er nm en t 20 22 Central European Public Administration Review, Vol. 21, No. 2/202386 Nejc Lamovšek A utho rs A rticle Title So urce Title P ublicatio n Y ear Early A ccess D ate H artm ann, K ., & W enzelb urger, G . U ncertainty, R isk and the U se o f A lgo rithm s in P o licy D ecisio ns: A C ase Stud y o n C rim inal Justice in the U SA Policy Sciences 2021 1/2021 G iest, S. N ., & K lievink, B . M o re Than a D igital System : H o w A I Is C hanging the R o le o f B ureaucrats in D iff erent O rganizatio nal C o ntexts Public M anagem ent R eview 2022 7/2022 A lo n-B arkat, S., & B usuio c, M . H um an-A I Interactio ns in P ub lic Secto r D ecisio n M aking: A uto m atio n B ias and Selective A d herence to A lgo rithm ic A d vice Journal of Public A dm inistration R esearch and Theory 2023 2/2022 W enzelb urger, G ., K o nig, P . D ., Felfeli, J., & A chtziger, A . A lgo rithm s in the P ub lic Secto r. W hy C o ntext M atters Public A dm inistration 2022 12/2022 P encheva, I., Esteve, M ., & M ikhaylo v, S. J. B ig D ata and A I - A Transfo rm atio nal Shift fo r G o vernm ent: So , W hat N ext fo r R esearch? Public Policy and A dm inistration 2020 K im , S., A nd ersen, K . N ., & Lee, J. W . P latfo rm G o vernm ent in the Era o f Sm art Techno lo gy Public A dm inistration R eview 2022 8/2021 C astelno vo , W ., & So rrentino , M . The N o d ality D isco nnect o f D ata-D riven G o vernm ent A dm inistration & Society 2021 3/2021 G rim m elikhuijsen, S. Exp laining W hy the C o m p uter Says N o : A lgo rithm ic Transp arency A ff ects the P erceived Trustw o rthiness o f A uto m ated D ecisio n-M aking Public A dm inistration R eview 2023 6/2022 W irtz, B . W ., & M üller, W . M . A n Integrative C o llab o rative Eco system fo r Sm art C ities - A Fram ew o rk fo r O rganizatio nal G o vernance International Journal of Public A dm inistration 2023 3/2022 C o m p to n, M . E., Yo ung, M . M ., B ullo ck, J. B ., & G reer, R . A d m inistrative Erro rs and R ace: C an Techno lo gy M itigate Ineq uitab le A d m inistrative O utco m es? Journal of Public A dm inistration R esearch and Theory 2023 9/2022 So urce: Tab le w as p ro d uced via W o S, 2023 Central European Public Administration Review, Vol. 21, No. 2/2023 87 Analysis of Research on Artificial Intelligence in Public Administration: Literature Review and Textual Analysis 4.2 Results of the Textual Data Analysis Using Voyant Tools In Table 2 below, we have identified the first 25 terms that most frequently appeared in the entire corpus of article texts. The terms “public” and “AI” were the most common, which is expected given the research theme. The word “AI” was the second most frequently used term in the analysed arti- cles, but the terms “artificial” and “intelligence” were also listed separately. Therefore, these terms can be combined differently, for example, as “artificial neural network,” “artificial discretion,” “intelligence technologies,” or “intel- ligence techniques.” The term “algorithms” also appeared as “algorithm” and was combined into a unified form “algorithms” for the purpose of analysis, ranking as the fourth most frequently used term in the analysed text. Table 2. First 25 Terms That Most Frequently Appeared in the Corpus of Article Texts Term Count public 2,099 ai 1,776 data 1,479 algorithms 805 administration 701 decision 686 research 640 intelligence 604 use 563 human 545 artificial 540 systems 533 sector 514 Term Count policy 509 risk 497 algorithmic 483 government 482 making 443 technology 431 information 419 social 417 review 413 big 398 governance 396 services 338 Source: Own 4.3 Frequency of Words with a Positive Connotation In the corpus of scientific articles, the terms “intelligence” (n = 604) and “trust” (n = 275) were most frequently used as terms with a positive connotation (Table 3). “Trust,” “ethical,” and “trustworthiness” also appeared on this list. Although these terms themselves carry positive connotations, the context in which they are used in texts is also crucial. For instance, when discussing a lack of trust or ethical issues in trustworthiness, words can convey a negative meaning in a text even though the Voyant Tools word database would classify these words as positively connotated. Central European Public Administration Review, Vol. 21, No. 2/202388 Nejc Lamovšek Table 3. First 25 Terms with a Positive Connotation Term Count intelligence 604 trust 275 smart 250 well 228 work 203 important 173 support 157 available 124 innovation 120 ethical 117 benefits 110 right 106 significant 77 Term Count advanced 75 integrated 73 better 71 creative 68 like 67 trustworthiness 62 regard 62 fairness 62 protection 58 improve 58 selective 55 fair 52 Source: Own 4.4 Frequency of Words with a Negative Connotation In the corpus of scientific articles, the terms “risk” (n = 497) and “bias” (n = 265) were most frequently used as terms with a negative connotation (Table 4). The word “issue” also appeared in the form “issues,” so we combined them into a unified form “issue.” Similarly, the term “bias” appeared in the forms “biases” and “biased,” so in the analysis, we combined them into a unified form “bias.” Unlike the list of terms with a positive connotation, in this list, it is harder to attribute an opposite positive meaning to the terms. Central European Public Administration Review, Vol. 21, No. 2/2023 89 Analysis of Research on Artificial Intelligence in Public Administration: Literature Review and Textual Analysis Table 4. First 25 Terms with a Negative Connotation Term Count risk 497 bias 265 fraud 234 problem 151 issues 147 criminal 90 complex 71 lack 71 limited 57 concerns 54 negative 50 critical 50 blame 47 Term Count discrimination 46 scandal 45 limitations 44 cancer 44 concern 36 regression 27 vice 25 manipulation 25 crime 24 discriminatory 22 error 21 limitation 20 Source: Own 4.5 Collocates Graph Figure 1 below shows the networking of word connections. The central part displays some of the keywords while the co-occurring terms are marked in orange and show the occurrences of terms in the context of the keywords. In our analysis, we noted that when using the term “use,” the terms “rules” and “conditions” often co-occurred, indicating that they were important when de- fining conditions and rules for the use of AI. Figure 1. Collocates Graph of Word Connections ai public sector systems challenges administration data use conditions rules decision review research policy big applications analytics fraud legal case processing information Source: Figure was produced via Voyant Tools. Central European Public Administration Review, Vol. 21, No. 2/202390 Nejc Lamovšek 4.6 Analysis of the Article Corpus with ChatGPT-4 Dated August 21, 2023 ChatGPT-4 identified the five most frequently used words in the corpus of articles as the following (response on key-word prompt): – Artificial – Intelligence – Technology – Public – Management ChatGPT-4 then responded to the given connotation prompt. Specifically, ChatGPT-4 identified that the texts in all 19 articles exhibited a diversity of topics and connotations. The scientific texts concentrated on various areas such as AI, technology, management, societal and ethical aspects, and chal- lenges associated with them. The connotations of the texts were a blend of positive and negative aspects, reflecting diverse perspectives addressed within the corpus. ChatGPT-4 also noted that positive connotations were evi- dent in descriptions of intelligent systems, technological advancements, trust in technology, and good work and approaches to management. Conversely, ChatGPT-4 established that negative connotations related to risks, bias, fraud, problems, and challenges encountered in the implementation of these solu- tions. The ChatGTP-4 analysis concluded that the overall picture of the ana- lysed documents was balanced because the corpus of articles engaged with both positive and potentially negative aspects of the use of AI and technology in management and society at large. 4.7 Analysis of the Content of Selected Articles Our brief review of the literature, as presented in the literature overview of selected articles, revealed a very broad scope of AI application in public ad- ministration. We found that researchers focused on both the advantages of using AI and the potential problems and threats associated with its use. Par- ticularly, researchers emphasized aspects of simplifying work processes and thereby optimizing operational workflows as well as ethical considerations and the protection of personal data. Overall, the researchers seemed to pre- sent a balanced view of all aspects of AI usage. 5 Discussion Both research tools we used employ algorithms in their analyses, but, as Alhu- dithi (2021) pointed out, Voyant Tools focuses primarily on the frequency of occurrence of certain terms and the further visualization of them. In contrast, ChatGPT-4 is an advanced language model intended for natural language processing, which also understands the meaning, content, and context of the analysed text (Moshirfar et al., 2023). Central European Public Administration Review, Vol. 21, No. 2/2023 91 Analysis of Research on Artificial Intelligence in Public Administration: Literature Review and Textual Analysis For our analysis of the entire content of selected scientific articles, we used both Voyant Tools and ChatGPT-4. We deepened the analysis by preparing short reviews of the articles’ content in collaboration with ChatGPT-4, reviews integrated an important human factor into the analysis. Thus, our analysis of the texts’ connotations via ChatGPT-4 plus our human analysis could offer the full meaning of the texts, which is important for a thorough analysis. On the contrary, Voyant Tools lacks this option. Voyant Tools searches for the occur- rence of individual terms in the texts that have a positive or negative connota- tion but does not understand the broader meaning of the entire text. Voyant Tools identified the terms “public,” “AI,” “data,” “algorithms,” and “administration” as the most frequently used in the corpus of articles. Con- sequently, it is evident that thorough understanding of these terms is vital when dealing with the use of AI in public administration. In our analysis of terms that often co-occurred with the keywords, we found the terms “use,” “rules,” and “conditions” often co-occurred. This result likely indicates that re- searchers often research and write about setting rules and conditions (i.e., regulations) when talking about the use of AI. Indeed, this is a very frequent topic in the scientific literature. For one, Wirtz et al. (2019) emphasized that regulation is one of the main challenges in the use of AI. Furthermore, in June 2023, the EU Parliament adopted the negotiating positions for regulating AI in the EU. The regulation would ensure transparent operation of AI and pro- vide privacy and security for users as well as human oversight of AI operations (European Parliament, 2023). Positive connotations generally occur when describing intelligent systems and technological advancements, and negative connotations when dealing with words related to risks and bias. In the literature review, we noted that the term “trust” was used mainly from the perspective of questioning the trustworthiness of AI. However, Voyant Tools marked it as a word that de- notes a positive connotation even though the actual meaning in the text was more negative. Also, among words with negative connotations, the terms “issue” and “bias” often appeared. This finding connects with research from Alon-Barkat and Busuioc (2023) who warned about the issues of automating bias and selective adherence to decisions and advice from AI or algorithms when they align with group stereotypes. Therefore, human control of deci- sions made with AI is important, and the type of texts on which AI learns is also crucial. We noted that the analysis of texts in terms of positive and negative con- notations reflected a very complex and diverse field of AI use in the public sector. Moreover, our literature review found similar observations to those in the connotation analysis with ChatGPT-4, specifically, that researchers deal with both positive and negative consequences of the use and implementation of AI in the public sector. As Giest and Klievink (2022) stated, the execution of certain tasks can be faster and more efficient with AI, but the use of AI is always linked to the use of technology (Bodó & Janssen, 2022), so it is impor- tant to also consider risks such as the protection of personal data (Simonofski et al., 2022) and ethics in using AI (Wirtz et al., 2019). Central European Public Administration Review, Vol. 21, No. 2/202392 Nejc Lamovšek Through a review of the literature, we also recognized the need for establish- ing rules and regulations in the field of AI, but we are aware that it is difficult to halt the progress of technology, and, in our opinion, halting the develop- ment of technology and AI would even be inappropriate. For example, the use of algorithms in connection with AI is significant in the field of medical science. Indeed, Wenzelburger et al. (2022) noted that algorithms are used in predicting skin cancer. Furthermore, AI technology allows detailed text analyses, as evident in our research. The results of using intelligent tools for analysis can provide different and broader insights into the subject matter, which can contribute to increased quality of life and development of science. In analysing the results, we also noticed that Voyant Tools and ChatGPT-4 classified keywords differently to a certain extent. Such differences may be the result of the different algorithms each analysis tool uses. For instance, ChatGPT-4 takes semantics into account, understands the context of terms, and considers various linguistic nuances of the text. The actual prompts can also influence outcomes with ChatGPT-4 depending on how the program understands an instruction. For example, instructions determine whether it searches for words in the text that are identical or for words that have a simi- lar meaning. Moreover, PDF plugins for use with ChatGPT-4 can segment and analyse the text in parts, or they can just summarize it to a certain extent (ChatGPT-4 has important limitations in the amount of data processed). One of the negative aspects of the literature review made in collaboration with ChatGPT-4 is the possibility of hallucination, i.e., the cosmetic reinterpretation of the actual state - the result of the analyzed scientific articles. This should be pointed out to the readers of our article, and readers should consider this in the final interpretation of the literature summary and highlight this issue. Similarly, technological advancements in the use of digital tools could influ- ence the accuracy of our results. For example, we noticed that during the ar- ticle review period, OpenAI (the provider of ChatGPT-4) offered updated and more advanced chat features, such as advanced data analysis, browse with Bing, and Dalle-E 3 (photo generation). Finally, the way each tool is used can influence the results. For instance, when we excluded meaningless words and combined words with the same root via Voyant Tools, we affected the results of the analysis. All the mentioned shortcomings of digital tools could have significantly im- pacted the results of our analysis. For a more detailed explanation, we would need to examine the background functioning of the algorithms of both tools. 6 Conclusions Analysis and research of texts with so-called digital smart tools can help us understand the deeper meaning of a text and can emphasize individual el- ements that are significant in addressing a particular subject matter. These analyses could influence the overall advancement of society and the develop- ment of science. Central European Public Administration Review, Vol. 21, No. 2/2023 93 Analysis of Research on Artificial Intelligence in Public Administration: Literature Review and Textual Analysis Textual analysis emphasizes individual keywords that are important in the subject being discussed and the general connotation of the text, which may indicate factors for or against the use of AI technology in the field of public administration. Terms that co-occur with keywords can express certain addi- tional views that need to be considered in the analysis. In our case, the terms “rules” and “conditions” were connected with the key word “use,” which may indicate a strong necessity for the regulation of the AI field and perhaps an indication and incentive for regulators in shaping policies for the use of AI in public administration. However, the weaknesses of this study were the limited number of scientific articles we analysed, possible hallucination of ChatGPT-4 and the overrepre- sentation of certain authors among those articles. Moreover, we included the entire text of each article in our analysis; for further research, we suggest re- moving individual parts of the article, such as references. This modification can change the determination of keywords and the general connotations of the text. Further research could also expand the selection of analysed re- search articles by using other databases of scientific literature and comparing research among individual countries. In sum, for the broader field of public administration, the simultaneous use of both text-analysis tools—ChatGPT-4 and Voyant Tools—is appropriate. Only in this way can we obtain results that are more broadly applicable and that can help us shape policy in the field of public administration. Central European Public Administration Review, Vol. 21, No. 2/202394 Nejc Lamovšek References Ahn, S. (2023). The impending impacts of large language models on medical education. Korean Journal of Medical Education, 35(1), pp. 103–107. https:// doi.org/10.3946/kjme.2023.253 Alhudithi, E. (2021). Review of Voyant Tools: See through your text. Language Learning & Technology, 25(3), pp. 43–50. http://hdl.handle.net/10125/73446 Alon-Barkat, S. and Busuioc, M. (2023). Human–AI interactions in public sector decision making: “Automation bias” and “selective adherence” to algorithmic advice. Journal of Public Administration Research and Theory, 33(1), pp. 153– 69. https://doi.org/10.1093/jopart/muac007 Androniceanu, A. (2023). The new trends of digital transformation and artificial intelligence in public administration. Administratie Si Management Public, 40, pp. 147–155. https://doi.org/10.24818/amp/2023.40-09 Bodó, B. and Janssen, H. (2022). Maintaining trust in a technologized public sector. Policy and Society, 41(3), pp. 414–429. https://doi.org/10.1093/ polsoc/puac019 Busuioc, M. (2021). Accountable artificial intelligence: Holding algorithms to account. Public Administration Review, 81(5), pp. 825–836. https://doi. org/10.1111/puar.13293 Castelnovo, W. and Sorrentino, M. (2021). The nodality disconnect of data-driven government. Administration & Society, 53(9), pp. 1418–1442. https://doi. org/10.1177/0095399721998689 European Parliament. (2023). MEPs ready to negotiate first-ever rules for safe and transparent AI. European Parliament Press Room. , accessed 20 August 2023. Gesk, T. S. and Leyer, M. (2022). Artificial intelligence in public services: When and why citizens accept its usage. Government Information Quarterly, 39(3), 101704. https://doi.org/10.1016/j.giq.2022.101704 Giest, S. N. and Klievink, B. (2022). More than a digital system: How AI is changing the role of bureaucrats in different organizational contexts. Public Management Review, pp. 1–20. https://doi.org/10.1080/14719037.2022.209 5001 Gregory, K., Geiger, L. and Salisbury, P. (2022). Voyant Tools and descriptive metadata: A case study in how automation can compliment expertise knowledge. Journal of Library Metadata, 22(1–2), pp. 1–16. https://doi.org/1 0.1080/19386389.2022.2030635 Grimmelikhuijsen, S. (2023). Explaining why the computer says no: Algorithmic transparency affects the perceived trustworthiness of automated decision- making. Public Administration Review, 83(2), pp. 241–262. https://doi. org/10.1111/puar.13483 Hartmann, K. and Wenzelburger, G. (2021). Uncertainty, risk and the use of algorithms in policy decisions: A case study on criminal justice in the USA. Policy Sciences, 54(2), pp. 269–287. https://doi.org/10.1007/s11077-020- 09414-y McDonald, B. D. et al. (2022). The future of public administration research: An editor’s perspective. Public Administration, 100(1), pp. 59–71. https://doi. org/10.1111/padm.12829 Central European Public Administration Review, Vol. 21, No. 2/2023 95 Analysis of Research on Artificial Intelligence in Public Administration: Literature Review and Textual Analysis Moshirfar, M. et al. (2023). Artificial intelligence in ophthalmology: A comparative analysis of GPT-3.5, GPT-4, and human expertise in answering StatPearls questions. Cureus, 15(6), e40822. https://doi.org/10.7759/ cureus.40822 OpenAI. (2023a). At , accessed 11 November 2023. OpenAI. (2023b). GPT-4 is OpenAI’s most advanced system, producing safer and more useful responses. OpenAI. At , accessed 10 October 2023. Pencheva, I., Esteve, M. and Mikhaylov, S. J. (2020). Big data and AI – A transformational shift for government: So, what next for research? Public Policy and Administration, 35(1), pp. 24–44. https://doi. org/10.1177/0952076718780537 Rao, V. C. S. (2017). A brief study of words used in denotation and connotation. Journal for Research Scholars and Professionals of English Language Teaching, 1(1). Reynolds, L. and McDonell, K. (2021). Prompt programming for large language models: Beyond the few-shot paradigm. arXiv. https://doi.org/10.48550/ arXiv.2102.07350 Simonofski, A. et al. (2022). Balancing fraud analytics with legal requirements: governance practices and trade-offs in public administrations. Data & Policy, 4, e14. https://doi.org/10.1017/dap.2022.6 Štefanišinová, N. et al. (2021). Implementation and application of artificial intelligence in selected public services. Hrvatska i Komparativna Javna Uprava, 21(4), pp. 601–622. https://doi.org/10.31297/hkju.21.4.2 Terzidou, K. (2022). The use of artificial intelligence in the judiciary and its compliance with the right to a fair trial. Journal of Judicial Administration, 31, 154. https://ssrn.com/abstract=4495715 Vijayakumar, S. and Sheshadri, K. N. (2019). Applications of artificial intelligence in academic libraries. International Journal of Computer Sciences and Engineering, 7(16), pp. 136–140. https://doi.org/10.26438/ijcse/ v7si16.136140 Vogl, T. M. et al. (2020). Smart technology and the emergence of algorithmic bureaucracy: Artificial intelligence in UK local authorities. Public Administration Review, 80(6), pp. 946–961. https://doi.org/10.1111/ puar.13286 VOSviewer. (2023). VOSviewer manual. Centre for Science and Technology Studies, Leiden University. At, accessed 10 October 2023. Voyant Tools. (2023). At , accessed 10 October 2023. Wenzelburger, G. et al. (2022). Algorithms in the public sector. Why context matters. Public Administration, pp. 1–21. https://doi.org/10.1111/ padm.12901 Wirtz, B. W., Langer, P. F. and Fenner, C. (2021). Artificial intelligence in the public sector – A research agenda. International Journal of Public Administration, 44(13), pp. 1103–1128. https://doi.org/10.1080/01900692.2 021.1947319 Central European Public Administration Review, Vol. 21, No. 2/202396 Nejc Lamovšek Wirtz, B. W. and Müller, W. M. (2019). An integrated artificial intelligence framework for public management. Public Management Review, 21(7), pp. 1076–1100. https://doi.org/10.1080/14719037.2018.1549268 Wirtz, B. W. and Müller, W. M. (2023). An integrative collaborative ecosystem for smart cities — A framework for organizational governance. International Journal of Public Administration, 46(7), pp. 499–518. https://doi.org/10.1080 /01900692.2021.2001014 Wirtz, B. W., Weyerer, J. C. and Geyer, C. (2019). Artificial intelligence and the public sector—Applications and challenges. International Journal of Public Administration, 42(7), pp. 596–615. https://doi.org/10.1080/01900692.2018. 1498103 Wirtz, B. W., Weyerer, J. C. and Sturm, B. J. (2020). The dark sides of artificial intelligence: An integrated AI governance framework for public administration. International Journal of Public Administration, 43(9), pp. 818–829. https://doi.org/10.1080/01900692.2020.1749851 Xu, Y. et al. (2021). Artificial intelligence: A powerful paradigm for scientific research. The Innovation, 2(4), 100179. https://doi.org/10.1016/j. xinn.2021.100179