Evaluating the knowledge of experts in the maritime regulatory field

“This is an Accepted Manuscript of an article published by Taylor & Francis in Maritime Policy & Management on 2017, available online at the Taylor & Francis Ltd web site: www.tandfonline.com https://doi.org/10.1080/03088839.2017.1298865 .

ABSTRACT
Regulations are introduced by the International Maritime Organization (IMO) into the maritime industry as the result of safety accidents and/or pollution incidents. When there is lack of historical data, then the IMO appoints experts in order to collect information regarding the costs and benefits generated to a stakeholder of the maritime industry once implementing a maritime regulation. Therefore, the role of experts in providing qualitative and quantitative information is crucial with respect to the quality of the maritime regulatory process within the IMO or other regulatory authorities. In this article, a methodology is proposed involving common criteria in determining the expertise of an individual in the maritime regulatory field. As essential part of the research methodology, analytic hierarchy process (AHP) is utilized to determine the expertise of an individual based on his/her own judgements. The regulatory authorities and other stakeholders of the maritime industry may use this method when selecting experts for decision-making. In this article, a simulation is carried showing the potentials of the AHP methodology in expertise evaluation followed by a case study.

KEYWORDS
Expert knowledge; analytic hierarchy process; maritime regulations

1. Introduction
In January 2013, the total volume of goods loaded worldwide was 9.6 billion tons by a world fleet reaching more than 1.6 billion deadweight tons (UNCTAD 2014). These figures show how important is the sea trade for the wealth of the nations. The carriage of goods by sea has lately been affected by the rising influence of developing countries, and the containerization (Zhao 2016). With respect to global trade, the development of an appropriate framework of regulating safety at sea is governed by a variety of international conventions introduced by the International Maritime Organization (IMO). The IMO succeeded in producing a high number of conventions, codes, and circulars, which are referred in this article as the ‘Maritime Regulations’. However, the implementation of the maritime regulations in many geographical regions is slow and inadequate (Lambrou et al. 2008; Sampson and Bloor 2007; McMahon 2007; Karahalios et al. 2011). A main result of this inadequate implementation is the existence of a multiform regulatory regime.

The IMO recognizing the need for uniform implementation of maritime regulations promoted the Formal Safety Assessment (FSA) as a part of the regulatory process (IMO 2000). The FSA method was proposed to the IMO by the UK’s Maritime and Coastguard Agency, which was accepted as an essential tool to evaluate maritime regulations (Ruuda and Mikkelsen 2008). However, an outcome of the FSA method is very likely to depend on the different data that will be selected and evaluated (Rosqvist and Tuominen 2004). Regulations are usually introduced into the maritime industry as the result of accidents and/or pollution incidents. The historical records of those incidents are often used to carry out risk analysis studies. However, the validity of recognized risk analysis methods depends on experienced analysts (Arslan and Er 2008). Furthermore, in global relevance, there is no certain consensus on the statistical distribution on the causes of maritime accidents due to the different viewpoints of accident analysis and investigation approaches (Celik, Lavasani, and Wang 2010). Consequently, the safety standards of a ship are usually determined by the agreement among experts (Lee 2006). An example to illustrate this issue is the differences between the FSA conclusions made by Greece and the UK in respect of double-skin bulk carriers’ efficiency (IMO 2004). Therefore, the role of experts in providing qualitative and quantitative information is crucial with respect to the quality of an FSA (Rosqvist and Tuominen 2004). Consequently, there is a need to identify individuals with expertise in maritime regulatory field. Those individuals would participate in FSA studies carried out by the IMO or other maritime stakeholders involved in maritime regulatory enforcement. The use of experts to collect information is broadly used in the private organizations of maritime industry. The introduction of a maritime regulation affects many areas in the maritime industry and consequently a group of experts may have different academic or industrial background (Karahalios et al. 2011). For instance, ship operators dedicate risk
managers as system experts to read accident reports or use algorithms to identify the causal networks of the accidents (Hyttinen, Mazaheri, and Kujala 2014). Furthermore, at a minor level, ship operators need to recruit such individuals that would assist in compliance with maritime regulations and train their co-workers. As in any other firm, the know-how is achieved by the use of in-house, or the combination of in-house and external expertise (Monteiro, Neto, and Noronha 2014). Alternative quality of services may be significantly improved by having an expert readily available to users when human experts are in short supply (Liu et al. 2014). A continuous advance of quality services is a dimension that affects the performance of a ship operator (Pantouvakis and Karakasnaki 2016). The constraints that affect the future size of large container ships have also been examined with the assistance of experts (Gomez et al. 2015). The aim of this article is to develop a methodology suitable of measuring the expertise of an individual with respect to his/her knowledge on maritime regulations. Section 1 presents a discussion on the significance of choosing experts in the maritime industry regarding maritime regulations. A critical review regarding the measurement of expertise is presented in Section 2, and Section 3 introduces a proposed methodology in measuring the expertise of individuals. The applicability of the proposed methodology is demonstrated as a simulation, which is shown in Section 4. Eventually in Section 5, a case study is carried out, and in Section 6, conclusions are presented.

2. Literature review on expertise

The choice of an expert in the maritime regulations field is a crucial issue for safety at sea. Fenstad, Dahl, and Kongsvik (2016) mentioned that there has been considerable discussion on how well ‘subjective’ safety evaluations made by lay persons correspond to ‘objective’ safety estimated by risk analyses experts. However, to date, there is not an acceptable standard to evaluate the knowledge of an expert for a specific topic. Relevant literature is presented in Section 2.1. Nevertheless, there is significant research with respect to measurement of the expertise of an individual, which is discussed in Section 2.2, that could be applicable in maritime industry as well.

2.1. Criteria for determining an expert
One of the most common criteria in determining the expertise of an individual is the years of his/her working experience in a field. Experts are usually determined based on the length of their experience and reputation (Rassafiani et al. 2008). Supporting evidence for working experience as a main criterion of an expert can be mainly found in a variety of industries. For instance, initial experiments of Malhotra, Lee, and Khurana (2007) in distinguishing experts have been encouraged, even though results are only indicative given a small sample of 20 participants. Malhotra, Lee, and Khurana (2007) in their research indicate that decision quality practicing managers in the oil and gas industry substantially depend upon the decision-makers’ level of expertise. Their analysis shows expertise to be highly correlated with the breadth of experience and poorly with experience in the same working environment. Another method for determining expertise is to look at how individuals make decisions in their area of expertise (Rassafiani et al. 2008). On the other hand, the experience of an individual may not be a sufficient factor to identify this person as an expert. A variety of studies have shown that increasing experience is not always associated with better judgements. For instance, Witteman, Weiss, and Metzmacher’s (2012) results showed that novice counsellors performed
almost at the same level as very experienced counsellors. Certification is another mean of measuring expertise of an individual. Proof of knowledge, education, qualifications, and training could be presented in the way of transcripts, diplomas, or certificates of achievement. Clearly, the objective expertise
items are evidence based (Germain and Tejeda 2012). However, Pauley, O’Hare, and Wiggins (2009) in their attempt to measure aeronautical experience found that the degree of expertise was not significantly correlated with age, level of certification, or holding an instrument or instructor rating. An individual in order to be qualified as an expert should be consensus with other experts. Consensus appears to be the most important criterion in deterring expert judgement although the differences in judgement should be retained (Martin et al. 2012). Holm et al. (2014) proposed that consensus is a necessary condition for expertise. The experts in a given field should agree with each other if they do not, then it suggests that at least some of the candidate experts are not really, what they claim to be (Weiss 2003; Holm et al. 2014). Consensus reliability (that was found between experts) can be used to identify or confirm experts; however, this can be confounded by artificial consensus such as group thinking in which a group of experts may agree on a course of action without this necessarily being the most appropriate (Rassafiani et al. 2008). Consensual answers have often been proposed as surrogates for correct answers, although the logic of doing so has been criticized (Weiss and Shanteau 2003). An approach that is useful for determining the degree of consensus within a scientific community, and for exploring collective views on ranges of uncertainties, is to conduct a structured expert elicitation with formalized pooling of opinions (Bamber and Aspinall 2013). The gist of the criticism is simply that people may agree on poor answers (Weiss et al. 2009). However, in a variety of studies, consensus is considered of paramount importance. Wang et al. (2009) used the fact that their team members reached a consensus as acceptable due to the difficulty in precisely assessing risk factors and their relative importance weights in their study. In a similar way, the judgements of the experts on a topic are determined after long-run discussions to ensure the different points of view as final group decisions in a consensus (Celik, Deha, and Ozok 2009; Yang, Bonsall, and Wang 2011). In brain-storming methods, such as the Delphi method, typically a forecast is generated for many events, without providing any direct information as to the respondents’ views as to whether questioned events are interrelated. Thus, it is possible that the outcomes of a Delphi implementation could produce a forecast of events mutually reinforcing or exclusive of each other, and that an artificial consensus may be reached (Scapolo and Miles 2006).

Discrimination refers to the person’s differential evaluation of similar stimuli, and consistency refers to the person’s assessment of identical stimuli over time (Rassafiani et al. 2008). Experts must be able to discriminate between different stimuli, and they must be able to make these discriminations reliably or consistently (Lee et al. 2012). Protocols for measuring expertise in terms of these two properties are well developed and have been applied in settings. However, because these protocols need to assess discriminability and consistency, they have two features that will not work in all applied settings. Some
researchers have reported substantial work in the area of expertise, arguing that only people who are able to (a) differentiate between similar, but not identical, cases and (b) repeat their judgement with consistency qualify as ‘experts’ (Malhotra, Lee, and Khurana 2007). However, they acknowledge in their methodology it is possible (even for non- experts) to obtain inappropriate high scores using a consistent, but incorrect rule. An expert must make consistent decisions when repeatedly faced with the same or similar cases (Witteman, Weiss, and Metzmacher 2012). Many researchers suggest that the

consistency of an expert’s answers is an indication of his expertise (Weiss 2003; Shanteau et al. 2002). Weiss (2003) suggested that a key element to determine the expertise of an individual in a certain area is his ability to be consistent at his judgements. Therefore, a further criterion that should be considered for an expert is the consistency of his judgements over the time.

2.2. Measurement of expertise

In applications that generate numerical data, discrimination and inconsistency have been operationalized using terms familiar from analysis of variance (Weiss et al. 2009). An experimental design suitable for analysis may be as simple as the presentation of each of several stimuli more than once (Weiss et al. 2009). For instance, Witteman, Weiss, and Metzmacher (2012) applied the Cochran–Weiss–Shanteau (CWS) index, which assesses the ability to consistently discriminate. The team of researchers named the assessment tool after their respective initials, CWS (Cochran, Weiss, and Shanteau). The CWS was Cochran, Weiss, and Shanteau’s approach to assessing expertise purely from data. The CWS index calculates expertise as the ratio between discrimination and inconsistence of an expert. However, the index can only be interpreted relatively, not absolutely. Consequently, the CWS can be used only to determine which of two candidate experts is performing better (Weiss and Shanteau 2001). Germain and Tejeda (2012) as an alternative method proposed a psychometric scale. This scale consists of 18 items that measure the expertise of an employee. However, as a limitation, the authors noted that
the samples were relatively small and instruments can be improved only through rigorous field testing. It was further suggested that future research should explore the proposed measures to alternative structures that may range from a single factor to more complex structures.

6. Conclusions
The AHP is a tool that could be used to identify the degree of an individual’s expertise in the maritime regulatory field. AHP could be used as an alternative to CWS index to measure and quantify the inconsistency of a candidate expert. The contribution of the CRind is significant in this research because they pointed out the confusion among industrial experts for regulatory issues. The inclusion of more individuals without prior evaluation of their expertise can provide a ranking of priorities that could not necessarily be real. The CRover value is used as a reference point to measure the knowledge distance that an individual may has from the remaining panel as measured by CRind. The several rounds of questionnaires give the opportunity to each candidate to comment in each survey question at his/her convenience without any pressure that may be generated if panel was physically formed. In this article, it was also found that the inconsistency of a candidate may be due to careless participation in a survey. By examining the survey material, it appears that a candidate could be inconsistent when he is using extreme ranking while doing pairwise comparisons. Another reason identified is that some pairwise comparisons may not be completed with care by a candidate.

Further research is required to use the earlier method in measuring the inconsistency of individuals in other fields of maritime sectors. The maritime industry suffers from overregulation, and the proposed methodology could be used to measure the expertise of regulators and industrials. The methodology could be useful for ship operators with more complicated structures. Furthermore, maritime administrations that need accurate results for multiple activities may find the suggested approach helpful. It may be valuable to test if the proposed methodology can be equally applicable on the practices of other organizations such as nuclear plants, aviation, petrochemical plants, and governments.