DOI: 10.2478/v10051-009-0018-x
Enhancing the Effectiveness of Training through Information Feedback - The Case of Slovenia's Public Administration
Blaž Rodič1, Gozdana Miglič2, Goran Vukovič3, Bruno Završnik4, Robert Leskovar3
1Faculty of Ap plied Social Studies in Nova Go rica, Ki dričeva 9, 5000 Nova Go rica, Slove nia, blaz.ro dic @fuds.si 2Ministry of Public Administration of the Republic of Slovenia, Tržaška 21, 1000 Ljub Ijana, Slovenia, gmiglic@gmail.com 3Faculty of Organizational Sciences, University of Maribor, Kidričeva cesta 55a, 4000 Kranj, Slovenia, goran.vukovic@fov.uni-mb.si, robert.leskovar@fov.uni -mb.si 4Faculty of Economics and Business, University of Maribor, Razlagova 20, 2000 Maribor, Slovenia,
bruno.zavrsnik@uni-mb.si
After the fall of the Berlin wall, the public administrations in Central and Eastern European countries have had to adapt to an enormous number of changes in a relatively short time. Their civil servants are still facing a constantly changing regulatory framework which needs implementation and enforcement, as well as new demands from the growing private sector and the citizens. These challenges can only be met if the civil servants are highly qualified and constantly updating their qualifications. A sound system of in-service training is therefore crucial to keep the public servants on equal footing with the rapidly changing environment. The research presented in this paper analyzed the practices of training evaluation as an information feedback in a branch of Slovene state administration. We analyzed the attitudes of employees towards evaluation with the assumption that there probably are notab I e differences between the attitudes of public servants depending on their demographic qualities. The results of the research, that 414 public servants have participated in, show that the position in the hierarchy has the strongest influence on their attitude towards evaluation. Empirical data also show that most of the employees are willing to participate in a continuous and objective training evaluation, but the leading public servants are less involved in the evaluation than expected.
Keywords: civil servants, evaluation processes, information feedback, goal oriented systems, public administration, training, training effectiveness
1 Introduction
Since the early 1990's, the Slovenian government has made great efforts to transform the centrally planned economy into a market economy. This process was significantly influenced by a rapid and dynamic development of a modern social system which also required an appropriate adjustment and development of the administrative system. Initially, the public administration reform was focused mainly on the legislative and institutional structure, but it also laid the foundation of a modern and effective public administration. Important parts of the legislation, including the Civil Service Act (CSA), were adopted in 2002 and later amended (Official Gazette of the Republic of Slovenia, no. 56/02).
In fact, the construction a modern and advanced administrative system and the establishment of a professional civil service started with the implementation of the CSA. This Act describes the main elements of the human resources system: the selection of personnel, their employment, the rights and
duties of civil servants, career development, and the assessment of their professional activities. A key element in the application of an efficient human resource management (HRM) policy and development in the administration seems to be the establishment of a system for continuous development of civil servants' professional skills and expertise.
This paper and the research behind it are a response to the requirements of the CSA, connected to the civil servants' training effectiveness.
2 Understanding the Value of Training - Theoretical Framework
Training is a basic concept of HRM. Armstrong (1999: 507) describes training as the systematic modification of behaviour through learning which occurs as a result of education, instruction, development and planned experience. Training also attracts high-quality employees by offering them learning
and development opportunities, improving their competences and skills, resulting in higher job satisfaction, higher rewards and promotion within the organization, and increase the commitment of employees by encouraging them to identify with the mission and objectives of the organization. Training helps organizations manage change by increasing understanding of the reasons for change, providing people with necessary knowledge and skills, helps develop a positive culture in the organization, e.g. an orientation towards performance improvement, and provides higher levels of service to customers.
Planning of training is vital and requires attention to detail and careful organization. Go et al. (1996 in Cassidy, 2001) emphasize that some basic concepts should be involved in organizational training practice. The training system can be viewed as a continuously evolving system that has several phases: training needs assessment (TNA), identification of training objectives, planning, implementation, and evaluation of training. From the perspective of the system theory, training evaluation represents an information feedback loop, a key component of all goal-oriented systems (Škraba et al., 2003). Hamblin (1974, cited by Armstrong, 1999: 531) defines training evaluation as 'any attempt to obtain information (feedback) of the effects of a training programme and to assess the value of the training in the light of that information'. Thus training evaluation is seen as an analytical process that includes collecting and joining of subjective and objective data on all (or selected) phases of the training process coming from various sources and gained by different techniques. Evaluation results in the synthesis of data which includes the summary of results and recommendations referring to the evaluated programme (Stanley, 1987).
Bramley (1996) identified six general approaches to training evaluation, among which goal-based and systems-based approaches are predominantly used in the evaluation of training (Phillips, 1991). Goal-based models of evaluation are especially widely used because of their simplicity. Most of the practitioners (e. g Bramley, 1996, 2003; Kirkpatrick, 1998) have advocated a series of levels through which an assessment of the effects of individual learning and training activities is made. The activities include:
■	learners' reactions to the learning experience,
■	the learning achievements of participants,
■	changes in job behaviour,
■	the organisational effect of specific learning interventions.
Some theoreticians and practitioners (Kearns, 2005; Phillips, 1991) have also suggested a calculation process to determine the economic return on investment (ROI) for individual learning and training processes.
The reasons for incorporating evaluation into the overall programme serve a wide range of purposes. Since the basic purpose of evaluation is to influence the decisions on possible adjustments and improvement, the evaluation should fulfil at least six purposes. As Brinkerhoff (1988: 67) points out, evaluation should determine:
■	that an identified problem represents a training need and
to determine what the real goals are,
■	the most appropriate training strategy,
■	if the chosen strategy is successfully implemented,
■	if learning occurred and to what extent,
■	usage outcomes (at the individual level), and
■	impacts and value (at the organizational level).
Despite the importance of evaluation, there is evidence
that evaluations of training programmes are often inconsistent or missing (Alliger and Janek, 1989, cited in Goldstein, 1993: 167; Bramley, 1996: 90; Carnevale and Schulz, 1990; Mars-den, 1991). Evaluation of training is indeed identified as the most difficult part of the job (Foxon, 1989). This is not surprising since evaluation is poorly defined, having different meanings for different people in many different contexts. Part of the explanation may be that the task of evaluation is complex in itself. Evaluating training with regard to learning, transfer, and organizational impact involves a number of complexity factors. These complexity factors are associated with the dynamic and ongoing interactions of the various dimensions and attributes of organizational and training goals, trainees, training situations, and instructional technologies (Eseryel, 2002).
The issue of complexity is also connected with the limitation of upper level of evaluation (job behaviour changes, organisational results), which is due to several reasons. First, in the arena of human behaviour there are too many variables involved for us to establish absolute causal links between training, given behaviour, and results. Second, evaluation on the upper level demands close cooperation among different actors in the evaluation process, e. g. trainees, their direct superiors, personnel offices, trainers, trade unions, and public service users. Their interest for training evaluation results varies in nature and intensity, and they ask questions based on their own point of view and interest. Third, higher level evaluations take more time than relatively simple evaluation of trainees' reaction and learning achievements. Another problem with higher level evaluation is that it is less convincing when training involves "soft" behaviours such as communications skills or interpersonal sensitivity. Another point is that behavioural change and higher productivity, attributed to training could simply be a result of the attention given by the evaluator.
Possible reasons for inadequate evaluations also include: insufficient budget allocated; insufficient time allocated; lack of expertise; blind trust in training solutions; or lack of methods and tools (Eseryel, 2002).
3 Introducing Training Evaluation in the Slovene State Administration's Training System
Building a modern and advanced administrative system and establishing a professional civil service requires a continuous training of public servants to develop appropriate capacity, new administrative culture and professional ethics. Therefore, developing and implementing an effective training system have become one of the priorities for state administration. Slovenia's state administration under the government of 2004 in particular seemed very eager to climb aboard the new public management wagon, in contrast with bigger Continental-European countries, as described in Pollitt et al. (2007: 20).
One of the findings in a comprehensive study conducted by Saner, Strehl and Yiu (1997), which compared the in-service training within the public administration in several Euro-
pean countries, was that the in-service training of the civil servants in Slovenia was well organized and that need-oriented training concepts play a major role for the development of an efficient and effective administration.
Training as an instrument for change and improvement often does not provide expected results (Saner, Strehl, and Yiu, 1997). Many times, investments in training are not successful and intended objectives are not met leading to disappointments and unhelpful attribution of blame. Training is expensive; therefore it is essential for public managers to make sure that the effectiveness of training is carefully assessed. Present era of fiscal constraints demands that all organizational activities, including training, should be retained on the basis of their relative effectiveness vis-a-vis other programmes (Randma, 2002: 198).
In order to ensure the success of training and to achieve maximum efficiency of the training process, a demand for evaluation of training quality and efficiency was introduced in Slovene public administration training practice. The CSA specifies that the implementation of a training programme must be monitored, reports on the effectiveness of the programme produced and occasional testing of officials' theoretical and practical knowledge provided. As a consequence, the evaluation of training effectiveness is becoming a compulsory part of civil servants' training.
This paper is a complement to the report on research (Vukovič et al., 2008) we have conducted in order to investigate the civil servants' perception of training evaluation. The research was the first of its kind in the Slovene government administration. One of its goals was to identify and clarify the differences in understanding training evaluation between leading and non-leading civil servants. We believe a clearer understanding of the differences between the groups of civil servants will have an important implication for the practice of training evaluation within the Slovene state administration, especially for the design of a reliable and effective evaluation model, tailored to this state administration.
4 Methodology
4.1 Research Questions and Assumptions of the Research
This section presents the background to the four research questions listed in the previous section and the assumptions that the presented research is based upon.
The word evaluation can also trigger a series of defensive emotional reactions. Auluck (2006) noted that one of the obstacles in human resource development in public administration is the presentation and representation of the human resource development function. It often seems that many problems are associated with misunderstanding the role of evaluation, and a lack of awareness that training is a part of the dynamic organizational process and needs to be monitored, amended and improved until the expected results are achieved (Goldstein, 1993). That is why we believe that the influence of various factors on civil servants' attitude towards training
evaluation is worth examining. Therefore we formulated our first question as follows:
Question 1: What is the correlation between respondents' demographic characteristics and their attitude towards training evaluation?
The training system can be viewed as a continuously evolving system with several stages: TNA, identification of training objectives, planning, implementation, and evaluation of training (Stanley, 1987: 14-15). The next research question is concerned with the correlation between TNA and evaluation. In theory, evaluation is strongly connected with TNA. Bramley (1996) and Easterby-Smith (1995) state that the purpose of evaluation is to assess the constant flow of recurring information, which is a starting point for assessing and evaluating the needs for training once again. This means designing a new input for the next level of an individual's professional development. The manner in which training needs are identified is an important factor in the design of evaluation, i.e. the identification of training needs has to include the criteria that the changes in job performance and effectiveness are to be assessed with (Bramley, 1996). This lead us to formulate the second research question as follows.
Question 2: What is the correlation between training evaluation and TNA?
The third question addresses the correlations between evaluation and training effectiveness. For the purpose of this study, Bramley's definition of training effectiveness was used. Bramley (1996: XVI) suggests that training effectiveness should 'imply not only finding out whether the training was well done but also asking what it achieved and whether it was worthwhile for the organization to be sponsoring it'.
Torrington and Hall (1998) believe that the evaluation of the effectiveness of training is one of the most inadequately implemented aspects of training, with many organizations believing that training ends once the delivery of the training programme is complete. This belief has two consequences: it defeats the idea of training as a continuous process lasting throughout an employee's career and, more importantly, it will result in the failure to measure whether training has met the objectives set beforehand and whether it has developed/induced positive behavioural changes.
Torrington and Hall (1998) also believe that evaluation is vital in determining how successful the training programme has been and that it is vital for an organization to be able to demonstrate value for money. Armstrong (1999) states that by implementing an evaluation process the organization will have a degree of control and that it is therefore useful that the entire training programme is evaluated because it is important to assess whether the training programme has met the objectives set out at the planning stage, and to indicate where improvements or changes are required in order to ensure that training will be more effective. We tried to capture the essence of these issues in the third research question.
Question 3: What is the correlation between training evaluation and training effectiveness?
The final issue we were interested in was the correlation between the training evaluation and training results. Kirkpa-trick (1998: 23) gives a wide definition of this term and states that 'Results can be defined as the final results that occur-
red because the participants attended the programme'. For research purposes we defined the evaluation results as the new knowledge a participant has exhibited after training. These results form the basis for assessing the training effectiveness. They are obtained by measuring the knowledge, abilities and views that the employees acquire during training (Možina et al., 1998: 500). Easterby-Smith (1995: 18) states that evaluation contributes to the learning process and consequently to training results. Accordingly we have postulated the fourth question as follows.
Que stion 4: What is the correlation between trai ning progress and training results?
4.2 Research Population and Sample
The population of the study was identified prior to the beginning of the study, so that questions could be postulated according to the population. We focused a lot of attention on defining the subject of research because the topic we had chosen was rather extensive. We narrowed down the subject by only researching the current practice of training evaluation and the civil servants' attitudes toward evaluation, employed only in the Administrative Units (AUs) and not in any other state body, e.g. the ministries, the bodies within the ministries or governmental offices.
AUs are a form of territorial organization of state administration that was established in 1995 (Administration Act, Official Gazette of the Republic of Slovenia no. 67/1994). The main responsibility of AUs is to carry out all administrative tasks and support the competences of state administration that need to be organized and performed territorially. AUs make decisions at the first level of administrative matters and perform other tasks of state competence.
We have focused our research on the AUs because their network covers the entire country, and the organizational structure, scope of responsibilities, working procedures and the management of AUs varies very little from one AU to another, making the AUs comparable one to another.
At the time of research there were 58 AUs in Slovenia. Nine (15.5%) of them were identified and selected to participate in the study, and efforts were made to include representative AUs regarding the existing variations in size and organizational structure. The AUs selected were representative of the all 58 AUs as a whole in size, number of employees and geographic distribution. 502 employees (15.8% of employees in all AUs) were included in the research.
The research sample was divided in two groups. The first group consisted of the leading civil servants in the AUs (heads of internal organizational units) while the other group consisted of non-leading civil servants (clerks). The sample of the leading civil servants included 49 respondents (17.6% of all leading civil servants in all AUs) and the sample of non-leading civil servants included 453 respondents (15.6% of all non-leading employees in all the AUs); the samples were big enough to adequately represent both groups of employees in the AUs. Reliability and validity of the instrumentation was within acceptable limits.
4.3	Development of the Instrument and Data Sources
Initially, nine interviews with the heads of the AUs were performed to present the purpose of the research, acquire their consent for participation and verify the information about the current number of employees in the AUs.
Afterwards, questionnaire was selected as the instrument to gather the answers to the research questions. This instrument was based upon the main dimensions of the Instructional System Development (ISD) Model and it was developed following a review of related literature (Cascio, 1998; Goldstein, 1993) as well as the provisions of the Civil Servants Act. The questionnaire contained 29 items. Five to eight items were included in each of four categories with statements addressing the training practice of AUs. Responses to the instrument were anonymous. Respondents were asked to judge their current practice and the desired or ideal situation in their training system for 24 items. A five-point Likert scale ((1) Strongly Disagree, (2) Disagree, (3) No Opinion, (4) Agree, (5) Strongly Agree) was used. To complete the data gathering, one open-ended question was also used. Demographic data were also included (position, gender, age, level of education, and years of experience in the state administration) to aid in possible statistical comparisons and analyses of the groups. The instrument was validated by a panel of four heads of AUs and field tested with 15 civil servants employed in the AUs.
4.4	Data Collection Procedures
Study instrumentation was personally delivered to every head of an AU or their designee in each AU. Within three weeks, 414 of the 502 surveys were received for a return rate of 82.6% (the response rate of leading civil servants was 83.7% and the response rate of non-leading civil servants was 82.4%).
4.5	Statistical Analysis of the Data
Two surveys were removed from the study because they were not completed. Therefore 412 surveys representing nine AUs were used for data analysis.
The Statistical Package for Social Sciences (SPSS) software application and different statistical methods for analyzing the acquired data were used in the analysis. Descriptive statistics were computed on the demographic variables. Differences between the demographics groups were determined using ANOVA and the f-test. Spearman coefficients were computed to test for a relationship between responses on each item.
5 Results and Findings 5.1 Respondent Profile
The primary purpose of this section is to describe the participants in this study who completed the questionnaire with respect to the following demographic variables: position,
gender, age, level of education, and years experience in state administration.
Position. Of the 412 respondents who filled in this question, 49 (11.9%) were at a leading position (heads of department and other internal organizational units), and 363 of the civil servants were at non-leading positions (88.1%).
Gender. There were 93 male respondents (22.6%) and 319 female respondents (77.4%).
Age. The age of the respondents was divided into five categories: (1) under 30, (2) 30-39, (3) 40-49, (4) 50-60, and (5) over 60 years of age. The distribution of respondents into age groups is shown in Figure 1.
Level of Education. Educational level of the respondents was classified into six categories: (1) Doctorate degree, (2) Master's degree, (3) B.A. degree, (4) college degree, (5) secondary school degree, and (6) less than secondary school degree. The distribution of respondents into level of education groups is shown in Figure 2.
Years of experience within state administration. The years of experience of the respondents were divided into five categories: (1) under 5 years, (2) 5-9 years, (3) 10-14 years, (4) 15-20 years, and (5) more than 20 years. The distribution of respondents into years of experience groups is shown in Figure 3.
1-39 Vf i 4CM9vri
Figure 1: Distribution of respondents in age groups
Figure 2: Distribution of respondents into level of education groups
Figure 3: Distribution of respondents into years of experience groups
Taking into consideration the structure of the research participants and the data about the employees in all AUs (obtained from Directorate for Management and Personnel, Ministry of Public Administration of the Republic of Slovenia) we assessed that the representation of the realized research sample was statistically satisfactory and suitable for further analyses.
5.2 Analysis of Training Evaluation Perceptions
Eight items in the questionnaire involved participants' perceptions of training evaluation. These items were developed based
on the review of previous research published. One-way analysis of variance and f-test procedures were used to determine differences in the opinions regarding the training evaluation among different demographic groups (Table 1).
The leading and non-leading civil servants' opinions differ in 62.5% of the cases. The participants' position had a low, but statistically significant negative correlation to agreement with the following statements: 'Evaluation increases training effectiveness', 'Evaluation criteria exist', 'Feedback after training exists', and 'Consistent evaluation is performed'. This leads us to a conclusion that the leading civil servants are more likely to believe that existing evaluation procedures improve training effectiveness and that continuous and consistent
Table 1: Correlation between Demographics Variables and Opinions on Training Evaluation
Variables	Position	Gender	Age	Education	Experience
Evaluation increases training effectiveness	-.181*	-.039	.135*	-.096	.083
Evaluation should be based on training objectives	.132*	-.168*	.081	-.218*	.051
Evaluation detects new training needs	-.057	-.092	-.041	-.029	-.087
Evaluation encourages trainees to improve their training results	-.024	-.039	-.077	-.107	-.092
Evaluation criteria exist	-.136*	.007	-.136*	.000	-.045
Feedback after training exists	-.179*	-.042	-.019	-.091	-.020
Testing knowledge after training should be possible	.110*	.142*	-.159*	.133*	-.138*
Consistent evaluation is performed	-.163*	-.104	.001	-.093	-.095
Note. *p < .05
evaluation already exists and that its criteria are defined and known to all employees, than their clerk colleagues.
Participants' position shows a low, but statistically significant positive correlation to 'Evaluation should be based on training objectives', and 'Testing knowledge after training should be possible'. Non-leading civil servants are more likely to believe than the leading civil servants that training goals should serve as the basis for training progress evaluation. They are also more likely to favour knowledge evaluation after training is completed.
Participants' position is not correlated with the statements 'Evaluation detects new training needs', and 'Evaluation encourages trainees to improve their training results'.
The statement 'Testing knowledge after training should be possible' showed the strongest statistically significant correlation with the participants' demographic characteristics. In the theory (Kirkpatrick's Evaluation Model, 1998) testing the knowledge is one of the important levels of measuring training effectiveness. According to the data gathered, the groups most in favour of evaluation are the leading civil servants, women, younger and less experienced civil servants and the civil servants with a high level of education. That is not surprising, as leading civil servants take a successful test of a civil servant's
knowledge as an objective demonstration of knowledge and as a proven capability for work at a particular work post. The younger and less experienced civil servants probably see a professional examination as a chance to prove they meet the conditions for a promotion.
Testing the knowledge in Slovene state administration is in some cases mandatory and defined by legislation. They can be prescribed either by law, executive regulation or general act of the individual state body. On the basis of statutory provisions and the regulations in force, the professional examination for the execution of special duties and authorisations is prescribed for particular titles, official work posts or public tasks, e. g. "Professional examination in administrative procedure", "Professional examination for inspectors", "Professional examination in administrative operations", "Examination for Registrar", "Examination of professional competence for conducting and decision making in administrative violations procedure", "Certification examination for obtaining a licence to conduct the business of real estate brokerage", and "Certification examination in the field of safety and health at work". Officials cannot perform public tasks for which a professional examination is prescribed, unless they pass the prescribed examination.1
Tab^e 2: Leading and Non-Leading Civil Servants' Opinions on Evaluation - Results of t-Test of Group Means
Variables	Group	N	Mean"		t	Sig. (2-tailed)
Evaluation increases training effectiveness	L-^	48	3.95	.74	2.752	.008
	ML"	356	3.58	1.04		
Evaluation should be based on training objectives	L	48	4.00	.40	1.296	.199
	N	354	3.90	.65		
Evaluation detects new training needs	L	48	3.39	.63	-0.910	.128
	N	245	3.62	.74		
Evaluation encourages trainees to improve their training results	L	48	3.41	.63	4.072	.178
	N	345	3.51	.74		
Evaluation criteria exist	L	48	3.51	1.05	-1.541	.003
	N	353	2.80	1.29		
Feedback after training exists	L	45	2.55	.15	3.280	.002
	N	359	1.92	.98		
Testing knowledge after training should be possible	L	48	3.70	.82	7.111	.000
	N	362	3.41	.89		
Consistent evaluation is performed	L	48	2.32	.43	1.162	.006
	N	359	1.67	.40		
Note:
^Mean -1 - Strongly Disagree, 5 - Strongly Agree ''SD - Standard deviation "T^ - leaders. ''N - non-leaders, p <.05
1 Professional examination for appointment to title, which was required for entrants in the civil service, has been substituted with mandatory training programme (CAS_OCT4, Article 89). Training primarily covers the following fields of knowledge: constitutional order and the institutions of European Union and its legal system; the system of legislative, executive and judicial powers, and their mutual relationship; local self-government, the system of public finances, the operation of state bodies and the bodies of the local self-governing community administrations, the system of administrative law, administrative procedure and administrative dispute, legislative procedure, and the rules of office operation, including electronic operation, the rules of the protection of personal and other data, and the rules of the production of other materials.
Because significant differences (p < .05) were found between leading and non-leading respondents, we examined the opinions of leading and non-leading civil servants about evaluation more closely. To compare their opinions more effectively, the mean scores of both groups were subjected to analysis. Table 2 displays eight items from survey along with the item mean, standard deviation, and number of respondents by position.
The examination of the means obtained for the variables concerning evaluation issues indicate that Slovene civil servants have a positive attitude towards the value of training evaluation although the opinions of both groups differed moderately. Questionnaire item means (M) ranged from 1.67 (non-leaders - for item 'Consistent evaluation is performed') to 4.00 (leaders - 'Evaluation should be based on training objectives'). Standard deviations for items ranged from .15 (leaders - 'Feedback after training exists') to 1.29 (non-leaders - 'Evaluation criteria exist').
For the purposes of this study, the strength of agreement indicated by each level of response was categorized as low if less than 2.25, moderate if between 2.26 and 3.75, and high if more than 3.76. Low agreement was found with non-leading respondent on two items ('Feedback after training exists', and 'Consistent evaluation is performed'). Non-leaders decidedly agreed that evaluation should be based on training objectives (M = 3.90), and in five cases non-leaders showed moderate agreement with the stated items ('Evaluation increases training effectiveness', 'Evaluation detects new training needs', 'Evaluation encourages trainees to improve their training results', 'Evaluation criteria exists', and 'Testing knowledge after training should be possible'). Leaders decidedly agreed on two times ('Evaluation increases training effectiveness', and 'Evaluation should be based on training objectives'), while in six cases their agreement was moderate ('Evaluation detects new training needs', 'Evaluation encourages trainees to improve their training results', 'Evaluation criteria exists', 'Feedback after training exists', 'Testing knowledge after training should be possible', and 'Consistent evaluation is performed').
T-tests were used to determine if significant differences exist between the two groups. Significant differences (p < .05) between leaders and non-leaders were found on five occasions. Leaders agreed (M = 3.95) more strongly than non-leaders (M = 3.58) that 'Evaluation increases training effectiveness' (p = .008). Leaders rated the item 'Feedback after training exists' significantly higher (M = 2.55) than non-leading respondents (M = 1.92) who more often tended to disagree with the state-
ments (p = .002). For the item 'Evaluation criteria exist' the average rating by leaders (M = 3.51; p = .003) was significantly greater than those of non-leaders (M = 2.80). Similarly, for the item 'Consistent evaluation is performed' the average rating by leaders (M = 2.32; p = .006) was greater than that of non-leaders (M = 1.67). Leaders also rated the statement 'Testing knowledge after training should be possible' significantly higher (M = 3.70; p = 000) than non-leaders (M = 3.41).
Non-leading civil servant respondents gave a higher rating (M = 3.51) to the statement 'Evaluation encourages trainees to improve their training results' than leaders (M = 3.41). Similarly, for the item 'Evaluation detects new training needs' the average rating by non-leaders (M = 3.62) was greater than that of leaders (M = 3.39).
For the item 'Evaluation should be based on training objectives' the item-level group means were similar (M = 4.00 for leaders; M = 3.90 for non-leaders).
5.3 Correlations between Training Dimensions
A correlation matrix was constructed in order to show the interrelationships between the five dimensions of training: training effectiveness, TNA, planning, evaluation, and training results. We calculated the Pearson's coefficient correlation for the variables (Table 3).
The matrix of relationships shows that one of the correlations was in the category of little if any correlation (correlation coefficient between .00 and .30), five of the correlations were in the low positive category (correlation coefficients between .30 and .50), two of the correlations were in the moderate correlation (correlation coefficients between .50 and .70), and one correlation was in the high positive category (correlation coefficient higher than .70). The values of correlation factors were ranked according to Hinkle, Wiersma, and Jurs (1988, cited by Poston, 1997).
Several interrelationships among the dimensions of systematic training cycle were found (significant at the .05 level). The correlation between TNA and evaluation was confirmed (r = .47). A significant positive relationship was also found between evaluation and planning (r = .47). A high correlation was also calculated between planning and TNA (r = .71). In addition, a correlation of .43 was found between TNA and training effectiveness, and a correlation of .65 between planning and training effectiveness.
Tab^e 3: Correlation Matrix for the Five Dimensions of Training
Variables	1	2	3	4	5
1 TNA	1.00				
2 Planning	0.71**	1.00			
3 Evaluation	0.47**	0.47**	1.00		
4 Results	0.15*	0.09	0.47**	1.00	
5 Training e ffectiveness	0.43**	0.65**	0.53**	0.38**	1.00
Note:
*p < .05. **p< .01
These results are in accordance with theory found in literature. Namely, Scriven (1991) suggests that a comprehensive programme evaluation of training should be comprised of three essential components: preformative evaluation, formative (inner) evaluation, and summative (external) evaluation. Preformative evaluation comprehensively assesses individual and organizational needs during the planning process. During this phase, intended goals are clarified and strategies are set for the training pathway. Formative evaluation should be conducted during the training programme. Formative evaluation provides feedback and determines modifications or adjustments that can be made to improve the learning process. The final, closing evaluation phase is summative evaluation, which allows trainees and their superiors to judge the overall value of their experience. An example of summative, i. e. external evaluation are the user satisfaction surveys which have been executed regularly in AUs from 2002 onwards.
A correlation of .53 was found between evaluation and training effectiveness. A significant positive relationship was found between evaluation and training results (r = .47), and between training effectiveness and training results (r = .38). This fact may be regarded as an instance of the well-known Hawthorne effect: the observations increase the volume of work and learning. People may improve their performance or behavior simply because of all the attention they receive (Bee and Bee, 1995: 83). On the other hand, feedback is one of the chief purposes of evaluation.
On the grounds of the demonstrated correlation between evaluation and training effectiveness we have established that civil servants generally have a positive attitude towards training evaluation.
5.4 Recommendations
We recommend that more quantitative and qualitative research should be conducted to complement the presented survey. The research should include not only AUs but other state bodies, e. g. ministries, bodies within ministries, government offices, as well. More precise information can give better insight in the training practice of the whole state administration.
However, in practice there is a noticeable gap between actual and desired state that is not the result of failing to recognize the importance of training in achieving greater individual and organizational success, but the result of not being informed about the fundamental principles of systematic training. Therefore, training for managers, personnel managers, and trainers on main issues of TNA, planning, and evaluation (including mastering various evaluation models, methods and techniques, statistical methods, etc.) is needed.
On the basis of the research it would be sensible to consider the design and implementation of an evaluation model that would connect the training goals, plans, actions and outcomes, including the transparent and standardized measurements of performance and training effectiveness. Evaluation should have two dimensions. The first dimension refers to evaluation stakeholders. Apart from the civil servants, who are directly involved in the training process, certain other individuals and groups should also have an interest in the training achievements. The close cooperation with the most important actors in
the process evaluation process is recommended, especially the groups and individuals that will utilize the results of the evaluation. Having in mind that the evaluation is a learning and action oriented management tool, this category should include trainers, the management of training institutes, trainees, their supervisors, and top management.
An external evaluator, who can play the role of a "disinterested" professional and therefore execute an unbiased objective evaluation, is also recommended. The external evaluation process - a long term follow-up evaluation - should concentrate on information concerning the training person in the job. Evaluation should also include the users of civil services.
Another dimension refers to the measuring of training, and includes the perception of its quality, suitability and influence as the means for achieving positive changes in employees' work performance and, last but not least, for the return of the financial investment. Both dimensions together provide information necessary for constant progress of the individual as well as the organization. Therefore, the concept of evaluation would include three phases - analysing training needs and planning of training as a preprogramme evaluation, formative evaluation aimed at a direct monitoring of training process, and summative evaluation focused on consequences and results of training.
We further recommend a wide ranging approach to training, which involves: aligning learning processes and investment to organisational strategic priorities, using a range of methods to assess and evaluate the contribution of learning and finally establishing the most relevant approach(es) to assessing and reporting on the value of learning for the organisation. Knowing training purposes and objectives beforehand is necessary for successful training, and forms the basis for measuring progress. A system of feedback information about the training progress of the individual has a positive influence on the training effectiveness. Exams and knowledge tests are the most favourable method of determining training progress.
6 Conclusions
The assumption of our research was that there probably are notable differences between the attitudes of public servants depending on their demographic qualities. The results of the research, that 414 public servants have participated in, show that the position in the hierarchy has the strongest influence on their attitude towards evaluation. Empirical data also show that most of the employees are willing to participate in a continuous and objective training evaluation, but the leading public servants are less involved in the evaluation than expected. All research questions received positive answers, therefore we can confidently make the final assessment that evaluation is a factor that significantly influences the effectiveness of training of civil servants. Based on the results and findings from data analysis, we can also conclude that:
In general, both groups of respondents agreed that evaluation positively influences training effectiveness.
Civil servants do not oppose the evaluation of training effectiveness. After analyzing the influence of evaluation on the quality of training results especially in the views of non-
leading civil servants there was a moderate correlation, from which we conclude that the civil servants understand the evaluation of training results as stimulation for further training.
Leaders understand evaluation as a management tool to ensure that training is organized efficiently in regard to the use of resources (finances, time, and energy) and effectively in regard to closing the performance gap.
Furthermore, the surveyed were of opinion that TNA, planning of training and evaluation of its effectiveness exerted a substantial impact on training effectiveness. With regard to the level of correlation factors, a significant positive relationship between TNA, planning, and evaluating training effectiveness in AUs was found, even though the correlation between training effectiveness and evaluation was not as strong as we had anticipated. We can probably attribute this to the absence of systematic and consistent evaluation in current practice.
Accurately identifying training needs, planning of training, and evaluating training effectiveness encourages the achievement of better training results;
Regardless of the positive attitudes towards evaluation, the research also uncovered an almost complete absence of evaluation of training effectiveness in the current practice of the AUs. Training effectiveness is monitored only from time to time. Monitoring is not systematically organized but is left up to the self-initiative of the individual heads of the internal units. The weakest connection to evaluation was perceived to be with receiving and giving feedback after training and existence of training criteria. Training evaluation is marked by subjectivism. Objective measuring tools have not been yet implemented in evaluation of training effectiveness in the AUs, making a better use of technology, i.e. computer aided programmes for evaluation, impossible. However, we are aware of the fact that the quality and effectiveness of administrative work, and consequently, the effectiveness of training are very difficult to define. This might be the reason, why generally only the time spent in training is taken into consideration when measuring training effectiveness of the individual. We can confidently say that two biggest hindrances to the introduction of systematic training evaluation in state administration are inadequate qualifications and the lack of experience of those responsible for training and development.
Even though the research proved that there is no lack of positive attitude towards training and awareness of the necessity for training evaluation, and therefore the attitude of civil servants is not an obstacle to a successful realization of principles of systematic training in state administration, including systematic evaluation, many areas and aspects of training evaluation still need more work.
References
Armstrong, M. (1999). A Handbook of Human Resource Management Practice, 7th edn., London: Kogan Page.
Auluck R.K. (2006). The Human Resource Development function: the ambiguity of its status within the UK public service, International Review of Administrative Sciences 72(1): 27-41. DDI: 10.1177/0020852306061614
Bee, F., & Bee, R. (1995). Training Needs Analysis and Evaluation. Exeter: Institute of Personnel and Development.
Bramley, P. (1996). Evaluating Training Effectiveness, 2nd ed. London: McGraw-Hill.
Brinkerhoff, R. O. (1988). An integrated evaluation model for HRD, Training and Development Journal 42(2): 66-68.
Carnevale, A. P., & Schulz, E. R. (1990). Return on investment: Accounting for training, Training and Development Journal 44(7): 1-32.
Cascio, W. F. (1998). Managing Human Resources: Productivity, Quality of Work Life, Profits, 5th edn., McGraw-Hill, New York.
Cassidy, T. (2001). Education and Training in the Hospitality Industry. Outlining the Importance of Hospitality Management Training, Available at http://www.cookeryonline.com/Thesis/ Training/litreview.html (29th June 2009).
Easterby-Smith, M. (1995). Evaluating Management Development, Training and Education, 2nd edn., Gower, Aldershot.
Eseryel, D. (2002). Approaches to Evaluation of Training: Theory and Practice, Educational Technology and Society 5(2). Available at http://www.ifets.info/journals/5_2/eseryel.html (29th June 2009).
Foxon, M. (1989). Evaluation of training and development programs: A review of the literature, Australian Journal of Educational Technology 5(2): 89-104. Available at http://www.ascilite.org. au/ajet/ajet5/foxon.html (29th June 2009).
Goldstein, 1. L. (1993). Training in Organizations: Needs assessment, development and evaluation, 3rd edn., Brooks/Cole Publishing Co., Pacific Grove.
Kearns, P. (2005). Evaluating the ROI from learning: how to develop value-based training, Chartered Institute of Personnel and Development, London.
Kirkpatrick, D. L. (1998). Evaluating Training Programs, 2nd edn, Berrett-Koehler Publishers, San Francisco.
Marsden, J. (1991). Evaluation: Towards a definition and statement of purpose, Australian Journal of Educational Technology 7(1): 31-38. Available at http://www.ascilite.org.au/ajet/ajet7/mars-den.html (29th June 2009).
Možina, S., Florjančič, J., Svetlik, 1., Jamšek, F., Lipičnik, B., Vodov-nik, Z., Svetic, A., Stanojevič, M., & Merkač Skok, M. (1998). Management kadrovskih virov, Fakulteta za družbene vede, Ljubljana.
Mumford, A. (1995). Effective Learning. London: Institute of Personnel and Development.
Phillips, J. J. (1991). Handbook of training evaluation and measurement methods, 2nd edn., Gulf, Houston.
Pollit, C. (2007). Convergence or Divergence: What has been happening in Europe? In Pollitt, C., Van Thiel, S. and Homburg (Eds.), New Public Management In Europe: Adaptation And Alternatives, pp. 10-25, Palgrave Macmillan, Basingstoke. DO1: 10.1057/9780230625365.
Poston, W. K. Jr. (1997). Comprehensive Study of Factors Impacting Perceived Quality in School Organizations, Educational Policy Analysis Archives 5(19). Available at http://olam.de.asu.edu/ epaa/v5n19.html (29th June 2009).
Randma, T. (2002). Public Personnel Management, 1n: Wright, G., and Nemec, J. (Eds.) Public Management in the Central and Eastern European Transitions: Concept and Cases. pp. 198-199, NlSPAcee, Bratislava.
Saner, R., Strehl, F., & Yiu, L. (1997). In-Service Training as an Instrument for Organizational Change in Public Administration. Brussels: 11AS/1AS1A.
Scriven, M. (1991). Evaluation thesaurus, 4th edn., Sage, Newbury Park.
Škraba, A., Kljajic, M., & Leskovar, R. (2003). Group exploration of system dynamics models - is there a place for a feedback loop in the decision process?, System dynamics review 19(3): 243-263. DOl: 10.1002/sdr.274.
Stanley, L. A. (1987). Guide to Evaluation of Training, International Centre for Public Enterprises in Developing Countries, Ljubljana.
Torrington, D., & Hall, L. (1998). Human Resource Management, Prentice Hall, Harlow.
Vukovič, G., Završnik, B., Rodič, B. & Miglič, G. (2008) The training of civil servants in the Slovene state administration: issues introducing training evaluation. Int. rev. adm. sci. 74(4): 653-676. D0I:10.1177/0020852308098473
Blaž Rodič received his Ph D at the University of Ma ri bor, Faculty of Organizational Sciences, where he worked as a researcher in the Laboratory of Cybernetics and Decision Support Systems. He is an assistant professor and researc -her at the Faculty of Applied Social Studies, University of Nova Gorica. His research interests include information society, decision support systems, and business intelligence.
Gozdana Miglič received her PhD in sociological sciences at the Faculty of Social Sciences, University of Ljub Ijana. From years 2002 to 2007 she was the director of Admini -stration Academy, the central national institution for civil servants training at the Ministry of Public Administration of the Republic of Slovenia. Her research focus is on adult education with emphasis on training needs analysis and evaluation of training effects within the Slovene Public Admi nistration.
Goran. Vukovič received his PhD at the University of Maribor, Faculty of Organizational Sciences, where he works as an associate professor in the area of human resources and marketing. Dr. Vukovič is an experienced consultant in the areas of human resources and organisation in the profit and non-profit sectors of the economy. Furthermore, he is the author of many authentic scientific papers, co author of many books, editor, critic, member of the editorial board for the journal Organizacija and member of the editorial council for the journal »Iskanja«.
Bruno Završnik is a Professor of Marketing at the Uni -versity of Maribor, Faculty of Economics and Business. He received his PhD in Marketing from the same Faculty. He teaches a variety of courses at the undergraduate and graduate levels at Faculty of Economics and Business. His research interests include empirical and theoretical issues in B2B marketing communications, effectiveness of purchasing, retailing strategy and implementation.
Robert Leskovar obtained his PhD in Organizational Sciences from the University of Maribor. He is an associa te professor in the field of informatics at the University of Maribor, Faculty of Organizational Sciences. His research work covers the modeling and simulation of systems, model validation, multi criteria decision making, and IS develop ment methods.
Izboljšanje učinkovitosti urjenja s pomočjo povratnih informacij - primer slovenske javne uprave
Po padcu berlinskega zidu so se javne uprave v srednji in vzhodni Evropi morale v zelo kratkem času prilagoditi mnogim spre -membam. Njihovi javni uslužbenci so še vedno soočeni s stalno spreminjajočim se zakonodajnim okvirjem, ki ga je potrebno vpeljevati v prakso ter novimi zahtevami pravnih in fizičnih oseb. Javna uprava lahko odgovori na te izzive samo, če so njeni us I užbenci visoko kvalificirani in se njihova znanja stalno posodabljajo, za ustrezno prilagajanje kompetenc javnih uslužbencev hitro spreminjaj očemu se okolju pa je potreben kvaliteten sistem urj enja na delovnem mestu. V pričujoči raziskavi smo analizirali prakse evalvacije izobraževanja kot vira povratnih informacij v delu slovenske javne uprave. Analizirali smo odnos zaposlenih do evalvacije z domnevo, da obstajajo opazne razlike med skupinami javnih uslužbencev z različnimi demografskimi lastnostmi. Rezultati raziskave, v kateri je sodelovalo 414 javnih uslužbencev so pokazali, da ima položaj v hierarhiji najmočnejši vpliv na njihov odnos do evalvacije. Empirični podatki so pokazali tudi, da je večina zaposlenih priprav Ijena sode -lovati v kontinuirani in objektivni evalvaciji izobraževanja, vendar pa se vodstveni javni uslužbenci manj ukvarj ajo z evalvacijo, kot smo pričakovali.
Ključne besede: javni uslužbenci, evalvacijski procesi, povratne informacije, ciljno orientirani sistemi, javna uprava, urjenje, učinkovitost urjenja