                                                                   Computer Standards & Interfaces 97 (2026) 104122


                                                                        Contents lists available at ScienceDirect


                                                            Computer Standards & Interfaces
                                                                journal homepage: www.elsevier.com/locate/csi




A multi-criteria process for IT project success evaluation–Addressing a
critical gap in standard practices
João Carlos Lourenço a , João Varajão b,*
a
    CEGIST, Instituto Superior Técnico, Universidade de Lisboa, Av. Rovisco Pais 1, 1049-001 Lisboa, Portugal
b
    Centro ALGORITMI, Universidade do Minho, Campus de Azurém, 4804-533 Guimarães, Portugal




A R T I C L E I N F O                                         A B S T R A C T

Keywords:                                                     The evaluation of project success is widely recognised as valuable for improving IT (Information Technology)
Project success                                               project performance and impact. However, many processes fail to adequately address the requirements for a
Project evaluation                                            sound evaluation due to their inherent complexity or by not complying with fundamental practical and theo­
Multi-criteria evaluation
                                                              retical concepts. This paper presents a process that combines a problem structuring method with a multi-criteria
MACBETH
Process
                                                              decision analysis approach to evaluate the success of IT projects. Put into practice in the context of a software
Methodology                                                   development project developed for a leading global supplier of technology and services, it offers a new way of
                                                              creating a model for evaluating project success and tackling uncertainty, bringing clarity and consistency to the
                                                              overall assessment process. A strong advantage of this process is that it is theoretically sound and can be easily
                                                              applied to other evaluation problems involving other criteria. It also serves as a call to action for the development
                                                              of formal standards in evaluation processes. Practical pathways to achieve such standardization include
                                                              collaboration through industry consortia, development and adoption of ISO frameworks, and embedding eval­
                                                              uation processes within established maturity models. These pathways can foster consistency, comparability, and
                                                              continuous improvement across organizations, paving the way for more robust and transparent evaluation
                                                              practices.




1. Introduction                                                                                         Additionally, several errors identified by decision analysis literature
                                                                                                    [12,13] are often made, generating meaningless project success evalu­
    The sustainable success of virtually any organisation is strongly                               ations [14]. Some common mistakes involve not including relevant
associated with the success of its projects [1]. A key factor for project                           criteria in the evaluation model, not distinguishing the performance of a
success is that project managers clearly understand what success means                              project from its value, assigning weights to evaluation criteria without
[2], which is usually not the case [3]. Despite different notions about                             considering the ranges of variation of their performance scales, and
what constitutes “project success” and the many criteria that can be used                           making calculations that violate measurement scales’ properties. In
for evaluation (e.g., cost, time, and performance, among others) [4], a                             other words, such evaluations are inconsistent with multi-attribute
project must satisfy its clients to be considered successful [5–8].                                 value theory (MAVT) and value measurement foundations.
    Given the importance and complexity of the evaluation of projects,                                  Considering these limitations, this research proposes a process that
companies should define and implement systematic processes for eval­                                combines a problem structuring method with a multi-criteria approach
uating success to improve project management performance and the                                    for evaluating the success of information technology (IT) projects sup­
impact of deliverables [9]. However, despite the models and techniques                              ported by a real-world case. This process was developed and applied in
that are currently available for assessing project success, they are typi­                          the context of a project of GlobalSysMakers (for confidentiality reasons,
cally challenging to implement for a variety of reasons, notably the                                the name of the company herein is anonymized), a leading global sup­
complexity caused by using multiple and often conflicting objectives (e.                            plier of technology and services.
g., minimise cost and maximise quality), the scarcity of empirical studies                              In the GlobalSysMakers project, the need for a new process arose
reporting their genuine use in projects [10], and the fact that practices                           because the project management team felt that the scoring model
employed in companies are generally informal and simplistic [11].                                   initially defined for success assessment, while helpful, lacked accuracy.


    * Corresponding author.
      E-mail address: varajao@dsi.uminho.pt (J. Varajão).

https://doi.org/10.1016/j.csi.2025.104122
Received 12 August 2025; Received in revised form 7 November 2025; Accepted 23 December 2025
Available online 24 December 2025
0920-5489/© 2025 The Author(s). Published by Elsevier B.V. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
J.C. Lourenço and J. Varajão                                                                                      Computer Standards & Interfaces 97 (2026) 104122


Following an appraisal of several methodological alternatives, a new              weights of several stakeholders without a discussion obliterates their
multi-criteria approach combined with a problem structuring method                individual differences [26]. Additionally, the “importance of the
was shown to be the best solution, providing the required precision and           criteria” should consider their respective performance ranges; other­
transparency to the process, along with a better understanding of the             wise, the resulting weights would be arbitrary [27].
real meaning of the relative importance of each evaluation criterion.                 Basar [28] proposes a methodology to evaluate the performance of IT
This paper describes the process developed in detail so that it can be            projects in a fuzzy environment. She first identifies the evaluation
replicated in other projects. Also, the results are presented and dis­            criteria using the balanced scorecard method. Second, she determines
cussed, including contributions to theory and practice.                           the criteria weights with expert judgments and hesitant fuzzy weights.
    The proposed process, which combines a problem structuring                    Then, the weights are used to evaluate the performance of IT projects in
method with a multi-criteria approach for evaluating IT project success,          a Turkish company. The weighting process described in this paper is
offers several theoretical implications. First, it advances the conceptu­         difficult for a non-expert evaluator to understand. Additionally, the
alization of project success by integrating both subjective stakeholder           quantitative performances of projects on the criteria are systematically
perspectives and objective performance criteria, addressing the multi­            normalised to scores between 0 and 1 with a linear transformation that
dimensional and context-dependent nature of success in IT projects.               may not correspond to the preferences of evaluators (which may be
Second, it contributes to decision theory and project management                  non-linear). The paper does not explain how to address the evaluation of
literature by demonstrating how problem structuring methods—typi­                 the qualitative criteria.
cally underutilized in IT evaluation—can enhance the clarity and rele­                Ismail [29] applies the Delphi method and conducts a seminar with
vance of criteria selection and prioritization. Third, the integration of         experts to identify a construction project’s potential evaluation criteria
these methodologies provides a foundation for developing more robust,             and group them into clusters. A relative importance index is calculated
transparent, and adaptable evaluation frameworks, which can inform                for each criterion with a weighted average of the responses to a survey
future theoretical models and empirical studies. Ultimately, this                 expressed on a Likert scale. In a subsequent step, the experts 1) reduced
research supports the movement toward standardization by offering a               the number of clusters and criteria and 2) assigned the same weight to
replicable and theoretically grounded process that can be refined and             the latter. Then, a priority index was calculated for each criterion with
generalized across different organizational and project contexts.                 the Priority Evaluation Model (PEM) [30], which combines the “satis­
    The remainder of this paper is organised as follows. Section 2 briefly        faction” rate (assigned by the experts) and the “importance” of the cri­
reviews previous related work on project evaluation methods, cases, and           terion. The overall project success is obtained with a weighted sum of
multi-criteria evaluation methods. Section 3 describes the case context           the averages of the priority indexes obtained on each cluster and the
and the development of the success evaluation model using a process               clusters’ weights. However, the paper does not explain how these
that combines a problem structuring model with a multi-criteria deci­             weights were assessed. Additionally, the Likert scale classifications
sion analysis approach. Section 4 discusses the results obtained. Finally,        cannot be used for calculating averages or other arithmetic calculations.
Section 5 presents the conclusions and avenues for further work.                      Nguvulu et al. [31] use a Deep Belief Network (DBN) to evaluate eight
                                                                                  IT projects’ performances after training the DBN with five projects of 12
2. Previous related work                                                          months duration. The DPN automatically assigned weights and scores to
                                                                                  the criteria, considering possible interactions between them. The au­
2.1. Success of projects                                                          thors stress the advantage of this approach by not considering human
                                                                                  subjectivity. However, from our point of view, this is a weakness
    Evaluation can be defined as the assessment and analysis of the ef­           because the subjective preferences of project managers, clients, and
ficiency and effectiveness of the project’s activities and results. The           other stakeholders should be considered in an evaluation process to
evaluation looks at what is planned to do, what has been achieved, and            avoid arbitrary results generated by inadequate analytical approaches.
how it has been achieved [15]. Kahan and Goodstadt [16] conceive                      Wohlin and Andrews [32] apply principal component analysis and
evaluation as a set of questions and methods properly articulated to              subjective evaluation factors to estimate which projects are successful or
review processes, activities, and strategies to achieve better results.           unsuccessful out of a set of projects. This statistical approach may be
Therefore, the purpose of an evaluation is not just to find out what              used to identify key project characteristics, but it does not allow for
happened but to use that information to make the project better [17,18].          evaluating the project’s success according to stakeholders’ preferences.
    There are several evaluation approaches in the literature, some                   Yan [33] suggests the combined use of the balanced scorecard (BSC)
considerably complex regarding their practical operationalisation and             [34], the Analytic Hierarchy Process (AHP), and the Fuzzy Comprehensive
use. Varajão et al. [10] present a comprehensive review of models and            Analysis method (FCA), respectively, to construct a performance criteria
methods for evaluating information systems project success. Some ex­              system, assess the criteria weights, and obtain an overall evaluation
amples are described and analysed next.                                           score. The author explains how to obtain the performance criteria sys­
    Bannerman and Thorogood [19] propose a framework for defining IT              tem, but does not explain the weighting and scoring components.
project success that provides a common language for communication                     Yang et al. [35] apply a multi-criteria model for evaluating a soft­
and compares what stakeholders perceive as important. The authors list            ware development project’s success using the Analytical Network Process
the criteria that should be used to assess the success of a project within        (ANP) [36] to assess the criteria weights at several hierarchical levels.
five domains (process, project management, product, business, and                 The scores of a project on a given criterion were obtained by calculating
strategy). However, they do not explain how to consider these domains             the average of the scores assigned by five experts using a 5-point Likert
and criteria together.                                                            scale. Note that, as mentioned above, averages should not be calculated
    Barclay and Osei-Bryson [20] describe a structured framework                  with ordinal scales. In addition, ANP is based on AHP, a method with
named Project Objectives Measurement Model (POMM) to identify the                 known issues that affect the validity of the criteria weights (see, e.g.,
criteria for evaluating an information system (IS) project and assigning a        [37–39]).
performance measure to each criterion. POMM applies value-focused                     Section 2.2 reviews important concepts and methods related to
thinking principles [21] and goal question metric methods [22]. An                multi-criteria evaluation that are needed to create a proper value mea­
illustrative case is presented in which the importance of each criterion is       surement model [40,41] to assess the success of a project.
directly assessed using an average of the stakeholders’ answers based on
a 5-point Likert scale. However, despite its virtues, this operation is           2.2. Multi-criteria evaluation
neither quantitatively nor substantively meaningful [23], respectively,
because a Likert scale is an ordinal scale [24,25] and averaging the                 In a multi-criteria value model, the measure of success of a project is

                                                                              2
J.C. Lourenço and J. Varajão                                                                                              Computer Standards & Interfaces 97 (2026) 104122


given by the additive value function model:                                               generates a proposal of weights compatible with the inputted qualitative
                         n                        n
                                                                                          judgments by solving the linear programming problem described in
                         ∑           ( )          ∑
V(x1 , x2 , …, xn ) =           wj vj xj , with         wj = 1 and wj > 0, ∀j   (1)       Bana e Costa et al. [52]. The evaluators should validate the proposed
                         j=1                      j=1                                     weighting scale and adjust it if needed.

Where V is the overall value score of the success of the project, wj is the               2.2.2. Methods to build value scales
weight of criterion j, vj(xj) is the value score on criterion j of the per­                   We must assign fixed scores to the previously defined references to
formance xj, and nrepresents the number of evaluation criteria.                           build a criterion value scale. For example, we may assign 100 and
    Despite being straightforward in form, this model is often poorly                     0 value units to the “best” and the “worst” performances in each crite­
applied. We highlight that the criteria weights wj are scaling constants                  rion, respectively, although two other scores could be used so that the
[42], which represent trade-offs between criteria and not the erroneous                   highest score is assigned to the most preferred reference. Though this
notion of criteria’s measures of importance [21]. In addition, vj is a                    arbitrary assignment of scores leads to obtaining interval value scales
measurable value function, which represents both a preference order                       [25]. Additionally, the score of a project on a given criterion should
between performances on criterion j and a strength-of-preference order                    consider the preferences expressed by the evaluators upon performance
on differences of performances [43]. Moreover, the model requires the                     ranges within the criterion [43] (e.g., the difference in value between
criteria to be mutually preferentially independent [44], which entails                    performances A and B is worth twice the difference between C and D).
special care during the model structuring phase.                                          Hereinafter, we present two numerical scoring methods and a qualita­
    There are some fundamental aspects to note regarding the desired                      tive one.
properties for each evaluation criterion and also for the whole set of                        Edwards [53] presents the direct rating method. This numerical
criteria [45]. Each criterion should be essential for the evaluation and                  procedure first requires evaluators to rank the project performances in
controllable in the sense that the performance of the project influences                  order of decreasing attractiveness. The highest score (100 units) is
the degree to which the criterion is satisfied, independently of other                    assigned to the “best” performance and the lowest score (0 units) to the
additional decisions. Also, a family of evaluation criteria should be:                    “worst”. Intermediate scores are assigned to other performance levels
complete (the set of criteria should represent all of the relevant conse­                 considering the intensities of preferences between each two of them,
quences of the project); nonredundant (the criteria should not repeat the                 knowing that the difference between the “best” and “worst” is worth 100
same concerns); concise (the number of criteria should be kept to the                     value units. This method allows scoring a project directly or indirectly
necessary minimum to evaluate the project); specific (each criterion                      using a performance measure (e.g., quantitative continuous, quantita­
should be able to assess the consequences of the project, instead of being                tive discrete, or qualitative). von Winterfeldt and Edwards [54] describe
so broad that it compromises this purpose); and understandable (the                       the bisection method, also known as the mid-value splitting technique [55],
evaluation criteria should be clear in the eyes of any interested                         to create a value scale for a criterion. This numerical method assigns the
individual).                                                                              highest score to the “best” performance (100) on the criterion and the
    Depending on the ability to use appropriate numerical principles and                  lowest score (zero) to the “worst”. Then, it is asked which performance p
fluency to express oneself in words, an evaluator may prefer to apply a                   has a value equally distant from the “best” and the “worst” perfor­
numerical method or a non-numerical one [46]. In light of this, the                       mances, which means that the ranges “p–to–best” and “p–to–worst” have
remainder of this section focuses on quantitative and qualitative tech­                   the same strength-of-preference. Therefore, the performance p would get
niques tailored for these two types of evaluators. Specifically, we delve                 a midpoint score of 50. Similar midpoint questions are asked to identify
into methods for criteria weighting and building a value scale for each                   other points that can be used to form a piecewise linear value function or
criterion.                                                                                a curve. This method allows the creation of value functions upon a
                                                                                          quantitative and continuous performance measure on the criterion.
2.2.1. Weighting methods                                                                      Bana e Costa and Vansnick [50] developed MACBETH [51] to create
    A theoretically sound weighting method must consider the perfor­                      a value scale for a criterion (and to weight criteria, as described in the
mance ranges defined by two fixed references on each criterion. Com­                      preceding section). Still, contrary to the above-mentioned methods, it
mon references are, for example, the “worst” and the “best”                               needs only to elicit qualitative judgments. An evaluator judges the dif­
performances [39] or “neutral” and “good” performances [47]. Below,                       ference in attractiveness between two performances at a time, using the
we briefly describe two quantitative weighting procedures and one                         qualitative scale presented in the previous section, and inputs them into
qualitative.                                                                              the software tool M-MACBETH. This tool verifies the consistency of the
    Keeney and Raiffa [48] developed the trade-off procedure, which is a                  inputted judgments and generates a proposal of a value scale compatible
numerical method that requires establishing indifferences between two                     with them and with the scores assigned to the reference performances
fictitious projects using two criteria at each time. After establishing n – 1             “best” and “worst” (or “good” and “neutral”) [52]. In the final step, the
indifference relationships for the n criteria, a system of equations is                   evaluator must validate and adjust the proposed value scale if needed.
solved, including one equation in which the sum of the weights equals 1,                  As in direct rating, this method allows scoring a project directly or
to obtain the criteria weights.                                                           indirectly using any performance measure.
    Edwards and Barron [49] created the swing weighting method, which
is a numerical method that involves measuring the relative importance                     2.3. Review summary
of the improvements (swings) that can be achieved on the criteria,
considering a change from the “worst” to the “best” performance on                            In the project success literature reviewed, most papers address the
each of them.                                                                             identification of IT criteria (e.g., Lobato et al. [4] and Assalaarachchi
    Bana e Costa and Vansnick [50] developed MACBETH [51] to weight                       et al. [56]) or success factors (e.g., Pinheiro et al. [57] and Jayakody and
the criteria. This procedure requires ranking the worst–best swings and                   Wijayanayake [58]), but only a few present an evaluation approach. In
judging them using the qualitative scale of difference in attractiveness:                 addition, the evaluation methods identified suffer from one or more
no (difference), very weak, weak, moderate, strong, very strong, or extreme.              theoretical errors (e.g., weights used as indicators of importance, aver­
This qualitative scale is also used to judge the difference in attractive­                ages calculated with ordinal scales, application of techniques with
ness between two swings at a time. The elicited judgments are used to fill                known flaws, and normalisation procedures that do not consider
in the upper triangular part of a matrix in the software tool                             non-linear preferences). Furthermore, as far as we know, there is no
M-MACBETH, which validates each judgment’s consistency with those                         description of a formal process that may guide the evaluators from
previously inputted (see [52], pp. 425–443). Then, the software tool                      beginning to end, i.e., from identifying the evaluation criteria until

                                                                                      3
J.C. Lourenço and J. Varajão                                                                                      Computer Standards & Interfaces 97 (2026) 104122


reaching an overall measure of project success. Therefore, a gap in the IT        different roles in the project; all of them were somehow interested in the
project literature needs to be addressed, which will be done by applying          project’s outcomes. The group had three members: two from TEAMGSM
multi-criteria evaluation principles.                                             and TEAMUNI, and one external consultant. The team members were
   Given the characteristics of the evaluators, the simplicity of use of          selected considering their managerial responsibilities and to ensure
the MACBETH method and its software tool M-MACBETH, including its                 representativeness of all the involved parties. All the members agreed to
ability to validate the consistency of the value judgments expressed by           be involved in the model development tasks. Note that larger groups
evaluators and to work with any performance measure (be it qualitative            require different group processes, typically having separate meetings
or quantitative, continuous or discrete), this was the approach selected          with stakeholders of different areas of interest to develop parts of the
to weight the criteria and build a value function for each criterion in the       model, and with merge meetings gathering higher-level representatives
real-world case described in this paper.                                          of the client to validate the work done by the stakeholders and to finish
                                                                                  the overall model [63].
3. Model development                                                                  Fig. 1 depicts the model development tasks. The first task involves
                                                                                  identifying the aspects of interest for evaluating the project’s success
3.1. Research setting                                                             (“problem structuring”, described in Section 3.3). This is a critical task
                                                                                  because it is not possible to develop a proper evaluation model without
    GlobalSysMakers develops solutions in four business areas: mobility           understanding the problem, which is the reason why several publica­
solutions, industrial technology, consumer goods, and energy and                  tions have been devoted to identifying the fundamental evaluation
building technology. It has several divisions, including automobile               concerns to be addressed (e.g., [28,64]). Second, all the relevant eval­
multimedia, automobile accessories, electric tools, heating and hot               uation criteria should be included in the model, and a descriptor of
water, and home appliances. It employs roughly 410,000 associates                 performance should be identified for each of them, enabling the
worldwide, has about 440 subsidiaries and regional companies in 60                assessment of the extent to which each criterion is met (“model struc­
countries, and employs nearly 70,000 associates in research and devel­            turing”, Section 3.4). Third, the evaluation component of the model must
opment at 125 locations.                                                          be built (“value model building”, Section 3.5), which includes the con­
    The target project, here identified as PROJRD, was part of an R&D             struction of a value function for each criterion to transform the perfor­
program that had the participation of GlobalSysMakers and a university.           mances of the project into value scores (Section 3.5.1), and weighting
The project had as its primary goal the development of a software tool to         the criteria to depict their trade-offs (Section 3.5.2). Last, the evaluation
automate the assessment of printed circuit boards (PCBs) design. PCBs             model should be tested for adequacy and consistency (Section 4.1).
are essentially boards that connect electronic components used in all
(but the simplest) electronic products, such as household appliances or
vehicles. In addition to the software tool, the project deliverables              3.3. Problem structuring
included technical specifications, prototypes, and presentations.
    The software development process adopted was based on a hybrid/                   The problem structuring task aims to identify the fundamental ob­
agile methodology supported by SCRUM [59]. Agile methods for soft­                jectives [45] that determine the project’s success from the client’s
ware development have been increasingly used in the IT sector [60] and            perspective. Such objectives are essential reasons for the project’s suc­
are now mainstream [61]. In this project, agility enabled greater                 cess. Therefore, they should be used as criteria in the evaluation model.
adaptability of the development phases according to the company’s                     However, the identification of these objectives in ill-structured
needs and requirements, which evolved along with the project lifecycle.           problems may not be easy, which is why we opted to apply a problem
Thus, it was possible to deal with changes in the requirements that were          structuring method (PSM) known as group map [65], which can be used in
reflected in the final deliverables during the project development. In a          combination with a multi-criteria decision analysis approach [66].
later phase of the project, the SCRUM was coupled with a waterfall                    To begin structuring the problem, the decision-making group was
process since the objectives stabilised without needing a periodic up­            asked to say which aspects or concerns were relevant to evaluate the
date. The project team was multidisciplinary, incorporating engineers             project’s success. Then, for each of the concerns expressed, it was asked,
from GlobalSysMakers (TEAMGSM) and researchers from the university                “Why is that important?” or “What would be the consequences of doing
(TEAMUNI). Together, the teams (TEAMGSM and TEAMUNI) had                          that?”, which allowed us to identify other aspects.
electronics, software engineering, and project management skills.                     Fig. 2 depicts the complete group causal map built with the answers
    On average, the team allocated 1040 h per month to the project
(approximately 6.5 Full-Time Equivalent), distributed by the different
tasks of the project and according to the functions performed by each
element (three of the team members were not full-time in the project).
The project had a duration of 36 months.
    The project’s overall success was first assessed using a simple grid
scoring model built by non-specialists in evaluation, which directly
scored the project on several criteria and assigned importance weights.
However, the project management team felt the need for a more
advanced model to improve confidence in the evaluation. More in-depth
research on multi-criteria evaluation revealed some misinterpretations
in that process, which ultimately led to the development of a new model
in line with decision analysis principles. This paper describes the new
evaluation model.

3.2. Development tasks

   The model development process started by asking the project man­
ager to identify the members who should form the decision-making
group [62], i.e., the group in charge of developing the model to eval­
uate the project’s success. It was recommended to select members with                                  Fig. 1. Model development tasks.

                                                                              4
J.C. Lourenço and J. Varajão                                                                                         Computer Standards & Interfaces 97 (2026) 104122




                                                                      Fig. 2. Group map.


of the elements of the group using the software tool “Decision Explorer”
(from Banxia Software Ltd., https://banxia.com/dexplore), which
automatically numbered the concerns for identification purposes. This
map results from several iterations, adding some aspects and removing
others. Note that a specific concern may be expressed by one statement
(e.g., “(33) good requirements definition”) or by two statements sepa­
rated by an ellipsis, which depicts a positive pole and a negative one to
clarify the meaning of the concern (e.g., “15 time fulfilment… time
exceeded”). An arrow between two concerns indicates the direction of
causality. When an arrow points to a concern with two poles, it means
that the concern affected is the one at the positive pole (e.g., a “(29) good                        Fig. 3. Project’s success evaluation criteria.
contract management” contributes to the positive pole of “(1) cost
fulfilment… cost exceeded”; in the reverse case, the arrow would have a              problem structuring task.
negative sign near its head).                                                           The concerns represented by these criteria are as follows:
    In Fig. 2, it is possible to identify chains of means-ends objectives. For
example, an “(31) effective change management” contributes to the                     • Scope/quality fulfilment (ScoQual)—the extent to which the planned
“(36) deliverables use”, which respectively allows to “(41) reduce users’               (functional and non-functional) requirements were fulfilled (this
repetitive work”, which contributes to “increase users’ satisfaction”.                  criterion resulted from concern 14 in Fig. 2).
Although the “(41) reduce users’ repetitive work” is a means-objective
to the end-objective “(39) increase users’ satisfaction”, the group                      The prime deliverable of the project is a software tool to support the
considered the former a fundamental objective because it is important in             PCB’s design assessment, the other deliverables being subsidiary to this
itself and not because of its contribution to the latter. Therefore, “(41)           tool. In the end, if the software tool does not comply with a minimum set
reduce users' repetitive work” will be used as an evaluation criterion.              of planned requirements, it will not be able to assess the PCB’s design
Objective “(39) increase users' satisfaction” was considered too broad to            and will compromise the investment objectives.
evaluate the project’s success and thus will not be used.
                                                                                      • Cost fulfilment (Cost)—the extent to which the planned cost was
                                                                                        fulfilled (this criterion resulted from concern 1 in Fig. 2).
3.4. Model structuring
                                                                                         The budget defined for the project needs to be carefully managed due
3.4.1. Evaluation criteria                                                           to being financed by an external R&D entity with a very narrow margin
   Fig. 3 depicts the seven evaluation criteria that emerged from the                of deviation.
concerns highlighted in bold in the group causal map developed in the

                                                                                 5
J.C. Lourenço and J. Varajão                                                                                              Computer Standards & Interfaces 97 (2026) 104122


 • Time fulfilment (Time)—the extent to which the planned time was                   direct (the descriptor levels should directly describe the performances on
   fulfilled (this criterion resulted from concern 15 in Fig. 2).                    the corresponding criterion), operational (the information concerning
                                                                                     the performances of the project can be obtained and value judgments
     Since this project is part of a large program, time fulfillment is a            can be made), understandable (performances and value judgments made
significant management aspect because all the program’s projects must                using the descriptor can be clearly understood and communicated).
be finished simultaneously due to the program’s constraints. In other                    Table 1 presents the list of all the descriptors created to measure the
words, not meeting the deadline in this project would mean completing                performance of the project, as well as two reference performance levels,
it in whatever form it is in when the program reaches its end, complying             “neutral” and “good”, for each of them. Note that the definition of two
or not with the scope, and delivering or not what was planned.                       reference performance levels is required to weigh the criteria, allowing
                                                                                     comparisons between criteria preference ranges and defining two fixed
 • Increase of the number and type of errors identified in each verification         anchors for the value scales (see Section 2.2). Furthermore, the use of a
   cycle (IncNoType)—the extent to which the number and type of errors               “neutral” performance level (which corresponds to a performance that is
   identified in each PCB’s verification cycle increase (this criterion              neither positive nor negative on the criterion) and of a “good” perfor­
   resulted from concern 43 in Fig. 2).                                              mance level (which corresponds to a very positive performance on the
                                                                                     criterion) allows to increase the understandability of the criterion, and
   Before the project was implemented in the company, the PCB designs                are thus preferable to the “worst” and the “best” references used as ex­
had been checked mainly in a semi-automatic way by specialised engi­                 amples in Section 2.2.
neers. Due to the many PCB components, details, and rules to review, it                  As shown in Table 1, the criteria scope/quality fulfilment and increase
was virtually impossible to check all of the required features. The                  in the number and type of errors identified in each verification cycle do not
consequence was the late detection of some errors in more advanced                   have direct descriptors of performance. For these criteria, constructed
stages of the projects, or, in other words, in later verification cycles. This       descriptors were developed combining the characteristics inherent to
accounts for the importance of the new software tool to increase the                 those criteria, as explained next (Bana e Costa et al. [67] describe a
number and type of errors identified early on in each verification cycle,            detailed procedure for creating constructed descriptors).
thereby reducing the design costs.                                                       To measure the performance of the project on the scope/quality
                                                                                     fulfilment criterion, several requirements that deliver different contri­
 • Reduction of the number of verification cycles (RNVC)—the extent to               butions to the project’s success were considered, following the MoSCoW
   which the number of verification cycles is reduced (this criterion                method principles [68]. These requirements were classified into three
   resulted from concern 37 in Fig. 2).                                              types (“must have”, “important to have”, and “nice to have”) and
                                                                                     combined to obtain the performance levels of the descriptor presented in
     A PCB typically needs to go through several verification cycles until           Table 2.
it is free from errors and ready for production. When errors are detected                To measure the performance of the project on the increase of the
in a verification cycle, the PCB design needs to be corrected and tested             number and type of errors identified in each verification cycle criterion,
again, possibly requiring a new verification cycle. Each verification                several combinations of the number and type of errors identified at each
cycle of a PCB design implies high costs. Furthermore, there is the risk of          verification cycle (based on a past project) need to be considered (see
detecting errors only at the production stage, with even more severe                 Table 3). For example, a “5 % increase in the number of identified er­
consequences. A primary expected result of the new software tool is to               rors” and a “10 % increase in the type of identified errors” is a perfor­
reduce the number of verification cycles by enabling the early detection             mance depicted as level “E5 T10”. A verification cycle includes a series
of errors.                                                                           of tests to check for errors in the PCB’s design or if it is ready for pro­
                                                                                     duction (free from errors).
 • Improve efficiency (ImpEff)—the extent to which the number of                         We note that the indicators used in the constructed scales presented
   verified rules increases in each verification cycle without increasing            in Tables 2 and 3 cannot be considered in isolation, as they are mutually
   the involved human resources (this criterion resulted from concern                preferentially dependent. For example, in Table 3, an increase of 10 % in
   42 in Fig. 2).

   Since the process for verifying the PCB’s design rules is semi-                   Table 1
automatic, with a substantial part of manual labour, the current num­                Descriptors of performance.
ber of specialised engineers can only check some of the relevant aspects.             Criterion                        Descriptor             Neutral     Good
With the new software tool, it is expected that the same number of en­
                                                                                      Scope/quality fulfilment         Constructed            L2          L3
gineers can check a greater number of design rules, not spending more                   (ScoQual)                      descriptor (see
time doing it.                                                                                                         Table 2)
                                                                                      Cost fulfilment (Cost)           Cost of the project    Planned     95 % of the
 • Reduction of the repetitive work of the users (RRWU)—the extent to                                                  (k€)                   cost        planned cost
                                                                                                                                              (k€ 500)    (k€ 450)
   which the number of rules manually verified is reduced in each                     Time fulfilment (Time)           Project duration       Planned     95 % of the
   verification cycle (this criterion resulted from concern 41 in Fig. 2).                                             (weeks)                time        planned time
                                                                                                                                              (96         (90 weeks)
   In the semi-automatic verification of PCB’s design rules, manual la­                                                                       weeks)
                                                                                      Increase in the number and       Constructed            E5 T0       E10 T5
bour is repetitive and prone to errors due to the fatigue of specialists.
                                                                                        type of errors identified in   descriptor (see
Automating most of the rules’ assessment is expected to reduce the re­                  each verification cycle        Table 3)
petitive work of these specialists and free them to perform other tasks.                (IncNoType)
                                                                                      Reduction of the number of       Number of              1 cycle     2 cycles
3.4.2. Descriptors of performance                                                       verification cycles            verification cycles
                                                                                        (RNVC)                         decreased
    In this task, we associate a descriptor of performance with each                  Improve efficiency (ImpEff)      Number of verified     0%          40 %
evaluation criterion to measure how much the project satisfies the cri­                                                rules increased ( %)
terion. According to Keeney [45], a descriptor should be unambiguous (to              Reduction of the repetitive      Number of rules        0%          10 %
describe the performances on the associated criterion clearly), compre­                 work of the users (RRWU)       manually verified
                                                                                                                       reduced ( %)
hensive (to cover the range of possible performances on the criterion),

                                                                                 6
J.C. Lourenço and J. Varajão                                                                                                 Computer Standards & Interfaces 97 (2026) 104122


Table 2                                                                                      scope/quality fulfilment criterion with a discrete descriptor, and time
Scale for “scope/quality fulfilment” criterion.                                              fulfilment criterion with a continuous descriptor.
  Performance levels                                                                             Fig. 4 presents the matrix of judgments for the scope/quality fulfilment
                                                                                             criterion. Table 2 shows the constructed descriptor for this criterion
  The project…                                                               ​
  …satisfied all the requirements “must have” and “important to have”        L1              where: L1 means “the project satisfied all the requirements ‘must have’
    and most of the “nice to have”                                                           and ‘important to have’ and the majority of the ‘nice to have’”, L2 means
  …satisfied all the requirements “must have” and at least 85 % of the       L2 = Good       “the project satisfied all the requirements ‘must have’ and at least 85 %
    “important to have” and at least 20 % of the “nice to have” (or an                       of the ‘important to have’ and at least 20 % of the ‘nice to have’ (or an
    equivalent performance on the requirements “important to have”
    and “nice to have”)
                                                                                             equivalent performance)”, and L3 means “the project satisfied all the
  …satisfied all the requirements “must have” and at least 60 % of the       L3 =            requirements ‘must have’ and at least 60 % of the ‘important to have’
    “important to have” and at least 20 % of the “nice to have” (or an       Neutral         and at least 20 % of the ‘nice to have’ (or an equivalent performance)”.
    equivalent performance on the requirements “important to have”                           We can see in Fig. 4 that the difference in attractiveness between “L1”
    and “nice to have”)
                                                                                             and “L2 = Good” was deemed weak by the evaluators, whereas the
  …did not satisfy one requirement “must have”, or satisfied less than 60    L4
    % of the requirements “important to have”                                                difference in attractiveness between “L2 = Good” and “L3 = Neutral”
  …did not satisfy more than one requirement “must have”                     L5              was considered moderate. Therefore, the difference in value between
                                                                                             “L1” and “L2 = Good” should be lower than the difference between “L2
                                                                                             = Good” and “L3 = Neutral”, which can be confirmed in the value scale
Table 3                                                                                      presented in Fig. 6a, where the former difference corresponds to 65
Constructed scale for “increase of the number and type of errors identified in               value units and the latter to 100.
each verification cycle” criterion.                                                              The time fulfilment criterion has the descriptor of performance
  Increase in the number of            Increase in the type of              Level
                                                                                             “project duration (in weeks)” with the references “96 weeks = Neutral”
  identified errors (E)                identified errors (T)                                 and “90 weeks = Good”. To build a value function for this criterion, first,
                                                                                             we created three more equally spaced performance levels: one worse
  10 %                                 10 %                                 E10 T10
  10 %                                 5%                                   E10 T5 =         than “neutral” (99 weeks), one between “neutral” and “good” (93
                                                                            Good             weeks), and one better than “good” (87 weeks). Then, the evaluators
  10 %                                 0%                                   E10 T0           judged the differences in attractiveness between each two of these
  5%                                   10 %                                 E5 T10           levels, together with the “neutral” and the “good” levels, resulting in the
  5%                                   5%                                   E5 T5
  5%                                   0%                                   E5 T0 =
                                                                                             matrix of judgments presented in Fig. 5.
                                                                            Neutral              Looking at the diagonal (above the grey shaded cells) of the matrix in
  0%                                   0%                                   E0 T0            Fig. 5 we see that the intensities of the differences in attractiveness
                                                                                             between each two consecutive levels increase more when the number of
                                                                                             weeks exceeds 93 weeks: the evaluators considered weak the differences
the number of identified errors (E) is valued more highly when the per­
                                                                                             in attractiveness between “87” and “90 = Good” (and also between “90
centage increase in the type of identified errors (T) is greater. Otherwise, the
                                                                                             = Good” and “93”), whereas they considered moderate the difference in
number and the type of identified errors could have been used as in­
                                                                                             attractiveness between “93” and “96 = Neutral”, and very strong the
dicators for two separate evaluation criteria.
                                                                                             difference between “96 = Neutral” and “99”. Therefore, the difference in
    After the seven criteria had been clearly identified and their de­
                                                                                             value between “87” and “90 = Good” (and also between “90 = Good”
scriptors of performance established, the decision-making group was
                                                                                             and “93”) should be lower than the difference in value between “93” and
asked whether there was any additional aspect that might be considered
                                                                                             “96 = Neutral”, and the latter should also be lower than the difference in
in assessing the project’s success. The negative response indicated that
                                                                                             value between “96 = Neutral” and “99”, which can be confirmed in the
this set of criteria was exhaustive and, consequently, that the value tree
                                                                                             value function presented in Fig. 6c (each of the first two intervals cor­
presented in Fig. 3 could be considered complete.
                                                                                             responds to 40 value units, whereas the third and fourth equal 60 value
                                                                                             units and 160, respectively). Therefore, this function shows that the
3.5. Value model building                                                                    evaluators considered that increments in time after 93 weeks are
                                                                                             increasingly penalizing for the project’s success.
3.5.1. Value functions                                                                           We emphasize that the decision group made these judgments for
     As previously described, a descriptor of performance provides a way                     each criterion independently of the performance levels or the differences
of measuring the project’s performance on its associated criterion.                          in attractiveness on the remaining criteria, thereby supporting the
However, to build a value model, we also need to obtain the value of                         assumption of mutual preferential independence between criteria.
each plausible performance of the project (in the form of a value scale or                       Fig. 6 (6a–6g) presents the value functions of all the evaluation
value function), which requires knowing the preferences of the evalua­                       criteria.
tors upon differences in performances on the corresponding criterion.
     For that purpose, we applied the MACBETH method [51]. As                                3.5.2. Criteria weighting
described in Section 2.2, the questioning procedure of MACBETH re­                               Weighting requires establishing trade-offs between criteria, which is
quires the evaluators to answer questions of difference in attractiveness                    typically demanding because it implies comparing performance im­
between two performance levels at each time, using the qualitative                           provements on different criteria. The improvements (swings) are defined
scale: no (difference in attractiveness), very weak, weak, moderate,                         between the two predefined performance references, “neutral” and
strong, very strong, and extreme. The answers provided are used for                          “good”, in each criterion.
filling in a matrix of judgments in the M-MACBETH software tool, which                           According to the MACBETH weighting procedure, the first step was
analyses the consistency of the answers as soon as they are inserted, and                    to rank the “neutral–good” swings in order of decreasing preference
then generates (by linear programming) a proposal of value scale which                       (Fig. 7). The evaluators considered the swing from “1 to 2 verification
is compatible with the answers provided, given the fixed value scores                        cycles decreased” as the most important one (1st in Fig. 7), which im­
assigned to the “neutral” and the “good” performances (0 and 100 value                       plies that the criterion “reduction of the number of verification cycles
units, respectively).                                                                        (RNVC)” will have the highest weight. In contrast, the criterion
     We present two examples of applying the MACBETH method to build                         “reduction of repetitive work of the users (RRWU)” will obtain the
value functions for criteria with different descriptors of performance:                      lowest weight because it has the least important “neutral–good” swing

                                                                                         7
J.C. Lourenço and J. Varajão                                                                                             Computer Standards & Interfaces 97 (2026) 104122




                                       Fig. 4. MACBETH judgment matrix for the “Scope/quality fulfilment” criterion.




                                           Fig. 5. MACBETH judgment matrix for the “time fulfilment” criterion.


(7th in Fig. 7).                                                                  criteria, because their performances are not worse than “neutral” in any
    In the second step, the improvements provided by the criteria swings          of the criteria and are better than it in several criteria. Therefore, both
were judged qualitatively using the MACBETH semantic scale (Fig. 8),              scenarios dominate [69] a “neutral project”. Additionally, we may see
which allowed filling in the rightmost column in Fig. 9. For example, the         that scenario “PCB red 2 cycles” has an overall score very close to that of
improvement provided by the most important swing [RNVC] was                       a “good project” (100 units), whereas the value of scenario “PCB red 1
considered extreme, whereas the least important “neutral–good” swing              cycle” is almost mid-distance from a “neutral project” and a “good
[RRWU] was judged weak.                                                           project”.
    Then, the differences in attractiveness between each two “neu­                    However, it is not robust to say that the scenario “PCB no red of
tral–good” swings were assessed to fill in the remaining cells of the first       cycles” corresponds to an unsuccessful project, looking only at its overall
row of the weighting matrix and fill in the diagonal above the shaded             value score. We must determine if its overall result will always be worse
cells in Fig. 9. For example, Fig. 10 depicts the comparison of the               than that of a “neutral project” when in the face of the uncertainty
“neutral–good” swings in the reduction of the number of verification cycles       defined for the model parameters (i.e., the value scores and criteria
(RNVC) criterion and in the increase in the number and type of errors             weights). In fact, the evaluators considered it plausible that: a) each
identified in each verification cycle (IncNoType) criterion, which was            criterion weight (wj,j = 1, …, 7) may vary within an interval defined by
                                                                                                              (                           )
deemed as very strong (v. strong in Fig. 9). The other cells with no              the lower and upper limits wj ≤ wj ≤ wj , j = 1, …, 7 shown in Table 6;
judgments were filled in automatically (by transitiveness) with “P”               and b) the value scores of the scenario “PCB no red of cycles” may have
(positive) judgments by M-MACBETH.                                                                                                            ( )        ( )
                                                                                  plus or minus 5 value units (respectively denoted by vj yj and vj yj ,
    Finally, the software tool applied the linear programming model
described in Bana e Costa et al. [51] to generate a proposal of a                  j = 1,…,7) in all the criteria for which this scenario has a performance
weighting scale consistent with the qualitative judgments expressed in            different from “neutral” and “good”, otherwise it will keep 0 and 100,
the weighting matrix, which were subsequently validated by the eval­              respectively.
uators (with some minor adjustments), resulting in the weights pre­                   The linear programming (LP) problem (2) was then used to test
sented in Fig. 11.                                                                whether a “neutral project” additively dominates [70] the scenario “PCB
                                                                                  no red of cycles”, which would require a negative maxD. The result
                                                                                  maxD = 9.575denotes that there is at least one combination of plausible
4. Results and discussion
                                                                                  scores and weights for which scenario “PCB no red of cycles” has a
                                                                                  higher overall value than that of a “neutral project”.
4.1. Model testing and results
                                                                                      The worst possible overall value for scenario “PCB no red of cycles”
                                                                                  was also calculated, with the LP problem (3), resulting in minD =
    At this point, the actual performances of the project are already
                                                                                  –14.10. Therefore, in the face of the uncertainty, the overall value score
known for most of the criteria, but not for the reduction of the number of
                                                                                  of scenario “PCB no red of cycles” may vary between –14.10 and 9.575.
verification cycles (RNVC) criterion, which will only be identified in the
long term. Therefore, three alternative scenarios were created with                           7
                                                                                              ∑          [ ( )      (         )]
hypothetical future performances on RNCV: no reduction at all (PCB no             maxD =               wj vj yj − vj neutralj                                         (2)
                                                                                                 j=1
red cycles), a decrease of one verification cycle (PCB red 1 cycle), and a
decrease of two verification cycles (PCB red 2 cycles). The performances                Subject to:
of these scenarios are shown in Table 4.
                                                                                  7
                                                                                  ∑
    Applying the value functions previously defined for each criterion to               wj = 1
the performances presented in Table 4, we obtain the partial and the              j=1
overall value scores of the three scenarios shown in Table 5 using the
previously assessed criteria weights.                                             wj ≤ wj ≤ wj , j = 1, …, 7
    As seen in Table 5, the most advantageous scenario corresponds to
                                                                                                         [ ( )                  ]
“PCB red 2 cycles” with 94.60 overall value units, followed by “PCB red                       7
                                                                                              ∑                     (         )
1 cycle” with 49.60, and “PCB no red of cycles” with –6.65.                       minD =               wj vj yj − vj neutralj                                         (3)
                                                                                              j=1
    Scenarios “PCB red 2 cycles” and “PCB red 1 cycle” undoubtedly
denote a successful project independently of the weights assigned to

                                                                              8
J.C. Lourenço and J. Varajão                                                                                                 Computer Standards & Interfaces 97 (2026) 104122




Fig. 6. Value functions of criteria: (a) scope/quality fulfilment, (b) cost fulfilment, (c) time fulfilment, (d) increase in the number and type of errors identified in each
verification cycle, (e) reduction of the number of verification cycles, (f) improve efficiency, (g) reduction of the repetitive work of the users.




                                                                                      9
J.C. Lourenço and J. Varajão                                                                                        Computer Standards & Interfaces 97 (2026) 104122




                                                              Fig. 7. Neutral–good swings ranking.




                                                       Fig. 8. Neutral–good swings’ weighting judgments.




            Fig. 9. MACBETH weighting matrix (the P and I within the matrix respectively mean positive difference in attractiveness and indifference).


subject to:



                                                                                10
J.C. Lourenço and J. Varajão                                                                                                     Computer Standards & Interfaces 97 (2026) 104122


                                                                                              members. Therefore, the model has a form and content sufficient to
                                                                                              evaluate the project’s success [71].

                                                                                              5. Discussion

                                                                                                  The absence of a formal evaluation of project success results in the
                                                                                              waste of relevant lessons that can be used to enhance project manage­
                                                                                              ment practices [9,72]. This is a strong reason for implementing
                                                                                              well-structured processes to evaluate project success.
                                                                                                  Any evaluation process should start by identifying the success
                                                                                              criteria according to the decision-makers’ preferences and systems of
                                                                                              values, which are inherently subjective. We underscore that an evalua­
                                                                                              tion model has an objective component (factual data) and a subjective
                                                                                              one (value judgments), which should be independently addressed.
                                                                                              Therefore, subjectivity is a key component in an evaluation process, but
                                                                                              it should not be confused with ambiguity, which should be avoided. That
                                                                                              is why the success evaluation criteria should be carefully identified, and
                                                                                              a measure of the performance of a project on each of those criteria must
                                                                                              be operationalised. The “neutral” and “good” references of intrinsic
                                                                                              value allow identifying the project’s success level.
Fig. 10. Assessment of the difference in attractiveness between the “neu­                         Throughout the development of the evaluation model, the members
tral–good” swings in RNVC and IncNoType.                                                      of the decision-making group were encouraged to engage in open dis­
                                                                                              cussion whenever differences of opinion arose. This approach enabled a
                                                                                              better understanding of their points of view and helped the group reach
                                                                                              an agreement on the way forward.
                                                                                                  In the case described herein, the success of the project may depend
                                                                                              on the future performance of the reduction of the number of verification
                                                                                              cycles (RNVC) criterion. With “no reduction of verification cycles”, the
                                                                                              project may be unsuccessful, with –6.65 overall value units, caused by
                                                                                              its low performance and corresponding negative score (–125 value
                                                                                              units) on this criterion. However, as we have seen, given the uncertainty
                                                                                              defined for the partial value scores and the criteria weights, this scenario
                                                                                              is not guaranteed to correspond to a negative evaluation. In fact, its
                                                                                              overall value may vary between –14.10 and 9.575 units.
                                                                                                  With a “reduction of 1 verification cycle”, the project would obtain
                                                                                              49.60 overall value units, which is nearly a mid-distance evaluation
                                                                                              between a “good project” and a “neutral project”. With a “reduction of 2
                                                                                              verification cycles”, the project would obtain 94.60 overall value units,
                                Fig. 11. Criteria weights.                                    which is very close to that of a “good project”.
                                                                                                  Developing a transparent evaluation process, such as the one
7
∑                                                                                             described here, will promote the decision-making group’s understand­
      wj = 1                                                                                  ing and acceptance of the results. The participation of the decision-
j=1                                                                                           makers in all of the process phases is a key element for this purpose,
                                                                                              which will allow them to develop a sense of ownership of the model
wj ≤ wj ≤ wj , j = 1, …, 7                                                                    [63]. However, this is not a practice found in the literature related to
    After concluding the robustness analysis, the evaluation group                            evaluating project success, which offers an opportunity for
revisited the model and considered that it could deal with all the plau­                      improvement.
sible performances and adequately considered the value judgments of its                           The proposed process, which integrates a problem structuring


Table 4
Performance profiles of the project’s success for the three scenarios.
  Scenario / Criterion               ScoQual             Cost (k€)           Time                 IncNoType            RNVC                           ImpEff               RRWU
                                                                             (weeks)                                                                  ( %)                 ( %)

  PCB no red of cycles               L2                  480                 96                   E10 T10              No decrease                    60                   15
  PCB red 1 cycle                    L2                  480                 96                   E10 T10              Decrease 1 cycle               60                   15
  PCB red 2 cycles                   L2                  480                 96                   E10 T10              Decrease 2 cycles              60                   15




Table 5
Value scores of the project success for the three scenarios.
  Scenario / Criterion             ScoQual          Cost             Time          IncNoType             RNVC            ImpEff             RRWU               Overall value score
                                   (15 %)           (5 %)            (8 %)         (22 %)                (45 %)          (3 %)              (2 %)

  PCB no red of cycles             100              40               0             115                   –125            150                140                –6.65
  PCB red 1 cycle                  100              40               0             115                   0               150                140                49.60
  PCB red 2 cycles                 100              40               0             115                   100             150                140                94.60


                                                                                         11
J.C. Lourenço and J. Varajão                                                                                     Computer Standards & Interfaces 97 (2026) 104122


Table 6
Plausible intervals for the criteria weights.
  Criterion                        ScoQual         Cost             Time                IncNoType              RNVC                ImpEff                RRWU

  Index (j)                        1               2                3                   4                      5                   6                     7
  Current weight (wj)              15 %            5%               8%                  22 %                   45 %                3%                    2%
              ( )
  Upper limit wj                   18 %            7%               10 %                25 %                   45 %                4%                    2.5 %
  Lower limit (wj )                12 %            5%               8%                  19 %                   40 %                3%                    2%



method with a multi-criteria decision analysis (MCDA) approach for                encouraging future research to refine, validate, and extend the proposed
evaluating the success of information technology (IT) projects, offers            framework. Ultimately, this work not only enriches theoretical under­
several significant theoretical contributions to the fields of project            standing but also provides a foundation for more consistent, transparent,
management, decision sciences, and IS. First, it advances the conceptual          and stakeholder-aligned evaluation practices in the IT project domain.
understanding of IT project success by addressing its inherently multi­
dimensional and context-dependent nature. Traditional models often                6. Conclusions
rely on narrow success criteria—such as time, cost, and scope—while
this research introduces a more holistic and stakeholder-sensitive                   Evaluating the success of IT projects should be a mandatory project
framework. By incorporating problem structuring methods, the pro­                 management activity. However, this is not observed in the practice [11,
cess facilitates the elicitation and organization of the stakeholder per­         72]. There are several contributions given by the process herein
spectives, which are often overlooked or underrepresented in                      described, which can be easily adapted to other evaluation problems:
conventional evaluation models. This contributes to theory by empha­
sizing the social and interpretive dimensions of project success, aligning         • It shows how a multi-criteria approach may be used to evaluate IT
with contemporary views that success is not an objective outcome but a               (software development) projects while avoiding committing critical
negotiated construct [73].                                                           mistakes.
    Second, the integration of MCDA techniques provides a rigorous and             • It offers a transparent process.
transparent mechanism for prioritizing and aggregating evaluation                  • It involves the decision-makers in all of the model development
criteria, thereby enhancing the methodological robustness of success                 tasks.
assessment. This methodological synthesis bridges a gap in the literature          • It identifies the fundamental objectives of decision-makers with the
by demonstrating how qualitative insights from problem structuring can               help of a problem structuring method, avoiding ending up solving
be systematically translated into quantitative decision models. Theo­                the wrong problem [76].
retically, this supports the development of hybrid evaluation frame­               • It allows establishing quantitative and substantive meaningful [23]
works that are both contextually grounded and analytically sound.                    trade-offs between criteria (i.e., mathematically valid and unam­
Third, the application of the proposed process in a real-world case adds             biguously understood).
empirical depth to the theoretical model, offering evidence of its prac­           • It allows the management of the project to focus on what matters for
tical relevance and adaptability. This empirical grounding strengthens               the project’s success.
the external validity of the framework and encourages further theoret­             • It can be implemented to evaluate the success of other projects, in
ical exploration across different organizational and project contexts.               similar or different contexts.
    The MACBETH approach has been successfully employed, with                      • The use of descriptors of performance clarifies what is intended to be
different nuances and across various processes, to evaluate projects or              achieved in each criterion.
decision alternatives in diverse problem settings and for a wide range of          • It distinguishes performance from value, instead of directly attrib­
organizations [74]. The process described in this paper, which combines              uting scores to the project, mixing these two components.
problem structuring with the MACBETH approach and robustness                       • And, it allows creating value scales adjusted to the preferences of
analysis, may also be applied in other contexts, subject to the necessary            evaluators, upon different types of performance (e.g., qualitative or
adjustments.                                                                         quantitative, continuous or discrete).
    Our proposed process can also be scaled to the program or portfolio
level, although this should be done with caution. In the case presented              Additionally, it enables the identification of alternative scenarios to
here, we applied an additive value function model, which is compen­               deal with unknown future performances and to test the robustness of the
satory—meaning that poor performance on one criterion can be offset               conclusions considering uncertainties on the model parameters.
by good performance on others. However, this assumption may not al­                  In the target organization, given the shortcomings recognised in a
ways hold. In a program or portfolio context, for instance, if a key              previous “grid scoring model”, the multi-criteria evaluation model of the
project performs poorly, that alone may render the entire program or              real-world case described in this paper was built during an advanced
portfolio unsuccessful, regardless of the performance of the remaining            stage of the project’s development. This late development can be
projects. In such cases, a mixed model should be adopted, combining               considered a threat to internal validity regarding consistency and a
classification rules to address the non-compensatory criteria with an             limitation since the evaluation model should be built during the plan­
additive component for the compensatory ones.                                     ning phase of a project and revisited during the project development to
    Moreover, the research highlights the absence of standardized ap­             be improved, if needed, or adjusted to possible changes to the project
proaches for evaluating IT project success, which has long been a limi­           aim. Another threat to external validity should also be disclosed.
tation in both academic and professional domains. Standardization                 Namely, concerning scalability, further research is needed to test if the
facilitates the dissemination of knowledge and enhances predictability,           proposed process can be scaled or adapted for different project sizes or
thereby minimizing uncertainty and reducing risk [75]. By proposing a             types.
replicable and adaptable process, the study lays the groundwork for the              In future work, it would be interesting to create a process capable of
development of formalized evaluation standards. This has implications             dealing with all project phases, allowing the evaluation of its develop­
for theory-building, as it suggests a pathway toward unifying frag­               ment and evolution at several milestones, from the project initiation
mented evaluation practices under a coherent, theoretically informed              until its termination. The process described in this paper may be
model. In doing so, it contributes to the ongoing discourse on stan­              extended to evaluate project success throughout the project lifecycle.
dardization in project management and information systems evaluation,             This requires developing a model that includes both final and


                                                                             12
J.C. Lourenço and J. Varajão                                                                                                             Computer Standards & Interfaces 97 (2026) 104122


intermediate objectives (criteria) for measuring project success. The                             [10] J. Varajão, J.C. Lourenço, J. Gomes, Models and methods for information systems
                                                                                                       project success evaluation–a review and directions for research, Heliyon 8 (12)
intermediate objectives should be used during project development and
                                                                                                       (2022), https://doi.org/10.1016/j.heliyon.2022.e11977.
later deactivated by setting their weights to zero and rescaling the                              [11] J. Varajão, J.Á. Carvalho, Evaluating the success of IS/IT projects: how are
remaining criteria weights so that they sum to one. Monitoring the                                     companies doing it?, in: Proceedings of the 13th Pre-ICIS International Research
evolution of a project’s success against a well-defined set of criteria will                           Workshop on IT Project Management (IRWITPM 2018), San Francisco, USA, 2018.
                                                                                                  [12] R.L. Keeney, Common mistakes in making value trade-offs, Oper. Res. 50 (6)
allow identifying problems sooner and taking proper measures in time.                                  (2002) 935–945, https://doi.org/10.1287/opre.50.6.935.357.
Furthermore, the integration of the proposed evaluation process in the                            [13] J.E. Russo, P.J.H. Schoemaker, Decision Traps: The Ten Barriers to Brilliant
success management process [77] will add value to the management                                       Decision-Making and How to Overcome Them, Doubleday, 1989.
                                                                                                  [14] S. Lipovetsky, A. Tishler, D. Dvir, A. Shenhar, The relative importance of project
efforts.                                                                                               success dimensions, R&D Manag. 27 (2) (1997) 97–106, https://doi.org/10.1111/
    Finally, since artificial intelligence technology, especially with the                             1467-9310.00047.
rise of Large Language Models (LLMs), has shown great potential in                                [15] Shapiro, J. (2005). Monitoring and evaluation. C.-W. A. f. C. Participation. htt
                                                                                                       ps://www.civicus.org/view/media/Monitoring%20and%20Evaluation.pdf.
revolutionizing the automation of various complex tasks [78], it is                               [16] Kahan, B., & Goodstadt, M. (2005). The IDM manual: basics. http://sites.utoronto.
imperative to explore it in the context of success evaluation.                                         ca/chp/download/IDMmanual/IDM_basics_dist05.pdf.
                                                                                                  [17] V. Arumugam, J. Antony, M. Kumar, Linking learning and knowledge creation to
                                                                                                       project success in Six Sigma projects: an empirical investigation, Int. J. Prod. Econ.
CRediT authorship contribution statement                                                               141 (1) (2013) 388–402, https://doi.org/10.1016/j.ijpe.2012.09.003.
                                                                                                  [18] R. Linzalone, G. Schiuma, A review of program and project evaluation models,
    João Carlos Lourenço: Writing – review & editing, Writing – orig­                                 Meas. Bus. Excell. 19 (3) (2015) 90–99, https://doi.org/10.1108/MBE-04-2015-
                                                                                                       0024.
inal draft, Visualization, Validation, Software, Methodology, Investiga­                          [19] P.L. Bannerman, A. Thorogood, Celebrating IT projects success: a multi-domain
tion, Formal analysis, Conceptualization. João Varajão: Writing –                                    analysis, in: Proceedings of the 45th Hawaii International Conference on System
review & editing, Writing – original draft, Validation, Methodology,                                   Sciences, Maui, HI, 2012.
                                                                                                  [20] C. Barclay, K. Osei-Bryson, Determining the contribution of IS projects: an
Investigation, Data curation, Conceptualization.
                                                                                                       approach to measure performance, in: Proceedings of the 42nd Hawaii
                                                                                                       International Conference on System Sciences, Waikoloa, HI, 2009.
                                                                                                  [21] R.L. Keeney, Value-Focused Thinking: A Path to Creative Decisionmaking, Harvard
Declaration of competing interest
                                                                                                       University Press, 1992.
                                                                                                  [22] R. Solingen, E. Berghout, The Goal/Question/Metric Method: A Practical Guide for
    The authors declare that they have no known competing financial                                    Quality Improvement of Software Development, McGraw-Hill, 1999.
                                                                                                  [23] S. French, Decision Theory: An Introduction to the Mathematics of Rationality,
interests or personal relationships that could have appeared to influence
                                                                                                       Ellis Horwood, 1986.
the work reported in this paper.                                                                  [24] R. Göb, C. McCollin, M. Ramalhoto, Ordinal methodology in the analysis of Likert
                                                                                                       scales, Qual. Quant. 41 (5) (2007) 601–626, https://doi.org/10.1007/s11135-007-
                                                                                                       9089-z.
Acknowledgement
                                                                                                  [25] S.S. Stevens, On the theory of scales of measurement, Science 103 (2684) (1946)
                                                                                                       677–680, https://doi.org/10.1126/science.103.2684.677.
    This work has been supported by FCT – Fundação para a Ciência e                             [26] W. Edwards, J.R. Newman, Multiattribute evaluation, in: T. Connolly, H.R. Arkes,
Tecnologia within the R&D Unit Project Scope UID/00319/2025 -                                          K.R. Hammond (Eds.), Judgment and Decision Making: An Interdisciplinary
                                                                                                       Reader, 2nd ed, Cambridge University Press, 2000, pp. 17–34.
Centro ALGORITMI (ALGORITMI/UM). João C. Lourenço acknowledges                                   [27] R. von Nitzsch, M. Weber, The effect of attribute ranges on weights in
the financial support of Portuguese funds through FCT – Fundação para                                 multiattribute utility measurements, Manag. Sci. 39 (8) (1993) 937–943, https://
a Ciência e a Tecnologia, I.P., under the project UID/97/2025 (CEGIST).                               doi.org/10.1287/mnsc.39.8.937.
                                                                                                  [28] A. Basar, A novel methodology for performance evaluation of IT projects in a fuzzy
João C. Lourenço acknowledges the financial support of Portuguese                                     environment: a case study, Soft Comput. 24 (14) (2020) 10755–10770, https://doi.
funds through FCT – Fundação para a Ciência e a Tecnologia, I.P., under                              org/10.1007/s00500-019-04579-y.
the project UID/97/2025 (CEGIST).                                                                 [29] H.N. Ismail, Measuring success of water reservoir project by using delphi and
                                                                                                       priority evaluation method, in: Proceedings of the IOP Conference Series: Earth
                                                                                                       and Environmental Science 588, 2020 042021, https://doi.org/10.1088/1755-
Data availability                                                                                      1315/588/4/042021.
                                                                                                  [30] J.H. Yu, H.R. Kwon, Critical success factors for urban regeneration projects in
                                                                                                       Korea, Int. J. Proj. Manag. 29 (7) (2011) 889–899, https://doi.org/10.1016/j.
    The data is presented in the article.                                                              ijproman.2010.09.001.
                                                                                                  [31] A. Nguvulu, S. Yamato, T. Honma, Project performance evaluation using deep
References                                                                                             belief networks, IEEJ Trans. Electron. Inf. Syst. 132 (2) (2012) 306–312, https://
                                                                                                       doi.org/10.1541/ieejeiss.132.306.
                                                                                                  [32] C. Wohlin, A.A. Andrews, Assessing project success using subjective evaluation
 [1] R. Colomo-Palacios, I. González-Carrasco, J.L. López-Cuadrado, A. Trigo, J.
                                                                                                       factors, Softw. Qual. J. 9 (1) (2001) 43–70, https://doi.org/10.1023/a:
     E. Varajao, I-Competere: using applied intelligence in search of competency gaps in
                                                                                                       1016673203332.
     software project managers, Inf. Syst. Front. 16 (4) (2014) 607–625, https://doi.
                                                                                                  [33] X. Yan, Utilizing the BSC method for IT performance evaluation of construction
     org/10.1007/s10796-012-9369-6.
                                                                                                       companies, in: Proceedings of the First International Conference on Information
 [2] M.A. Kafaji, Interchange roles of formal and informal project management on
                                                                                                       Science and Engineering, Nanjing, China, 2009.
     business operational success, Prod. Plan. Control (2022) 1–21, https://doi.org/
                                                                                                  [34] R.S. Kaplan, D.P. Norton, The balanced scorecard–measures that drive
     10.1080/09537287.2022.2089265.
                                                                                                       performance, Harv. Bus. Rev. 70 (1) (1992) 71–79.
 [3] L.A. Ika, J.K. Pinto, The “re-meaning” of project success: updating and recalibrating
                                                                                                  [35] C.L. Yang, R.H. Huang, M.T. Ho, Multi-criteria evaluation model for a software
     for a modern project management, Int. J. Proj. Manag. 40 (7) (2022) 835–848,
                                                                                                       development project, in: Proceedings of the IEEE International Conference on
     https://doi.org/10.1016/j.ijproman.2022.08.001.
                                                                                                       Industrial Engineering and Engineering Management, Hong Kong, China, 2009.
 [4] B. Lobato, J. Varajão, C. Tam, A.A. Baptista, CrEISPS–a framework of criteria for
                                                                                                  [36] T.L. Saaty, The Analytic Hierarchy Process: Planning, Priority Setting, Resource
     evaluating success in information systems projects, Procedia Comput. Sci. 256
                                                                                                       Allocation, McGraw-Hill, 1980.
     (2025) (2025) 1821–1835, https://doi.org/10.1016/j.procs.2025.02.323.
                                                                                                  [37] C.A. Bana e Costa, J.C. Vansnick, A critical analysis of the eigenvalue method used
 [5] N. Agarwal, U. Rathod, Defining ‘success’ for software projects: an exploratory
                                                                                                       to derive priorities in AHP, Eur. J. Oper. Res. 187 (3) (2008) 1422–1428, https://
     revelation, Int. J. Proj. Manag. 24 (4) (2006) 358–370, https://doi.org/10.1016/j.
                                                                                                       doi.org/10.1016/j.ejor.2006.09.022.
     ijproman.2005.11.009.
                                                                                                  [38] J.S. Dyer, Remarks on the analytic hierarchy process, Manag. Sci. 36 (3) (1990)
 [6] R. Atkinson, Project management: cost, time and quality, two best guesses and a
                                                                                                       249–258, https://doi.org/10.1287/mnsc.36.3.249.
     phenomenon, its time to accept other success criteria, Int. J. Proj. Manag. 17 (6)
                                                                                                  [39] P. Goodwin, G. Wright, Decision Analysis for Management Judgment, 5th ed., John
     (1999) 337–342, https://doi.org/10.1016/S0263-7863(98)00069-6.
                                                                                                       Wiley & Sons, 2014.
 [7] H. Landrum, V.R. Prybutok, X. Zhang, The moderating effect of occupation on the
                                                                                                  [40] V. Belton, T.J. Stewart, Multiple Criteria Decision Analysis: An Integrated
     perception of information services quality and success, Comput. Ind. Eng. 58 (1)
                                                                                                       Approach, Kluwer Academic Publishers, 2002.
     (2010) 133–142, https://doi.org/10.1016/j.cie.2009.09.006.
                                                                                                  [41] R.L. Keeney, D. von Winterfeldt, Practical value models, in: W. Edwards, R.
 [8] J.K. Pinto, D.P. Slevin, Project success: definitions and measurement techniques,
                                                                                                       F. Miles Jr., D. von Winterfeldt (Eds.), Advances in Decision Analysis: From
     Proj. Manag. J. 19 (1) (1988) 67–72.
                                                                                                       Foundations to Applications, Cambridge University Press, 2007, pp. 232–252.
 [9] J. Varajão, L. Magalhães, L. Freitas, P. Rocha, Success management–from theory to
     practice, Int. J. Proj. Manag. 40 (5) (2022) 481–498, https://doi.org/10.1016/j.
     ijproman.2022.04.002.


                                                                                             13
J.C. Lourenço and J. Varajão                                                                                                           Computer Standards & Interfaces 97 (2026) 104122

[42] J.S. Dyer, J.E. Smith, Innovations in the science and practice of decision analysis:        [61] V. Henriquez, J.A. Calvo-Manzano, A.M. Moreno, T. San Feliu, Agile governance
     the role of management science, Manag. Sci. 67 (9) (2020) 5364–5378, https://doi.                practices by aligning CMMI V2.0 with portfolio SAFe 5.0, Comput. Stand.
     org/10.1287/mnsc.2020.3652.                                                                      Interfaces 91 (2025) (2025) 103881, https://doi.org/10.1016/j.csi.2024.103881.
[43] J.E. Smith, J.S. Dyer, On (measurable) multiattribute value functions: an                   [62] V. Ferretti, G. Montibeller, Key challenges and meta-choices in designing and
     expository argument, Decis. Anal. 18 (4) (2021) 247–256, https://doi.org/                        applying multi-criteria spatial decision support systems, Decis. Support Syst. 84
     10.1287/deca.2021.0435.                                                                          (2016) 41–52, https://doi.org/10.1016/j.dss.2016.01.005.
[44] J.S. Dyer, R.K. Sarin, Measurable multiattribute value functions, Oper. Res. 27 (4)         [63] L.D Phillips, Decision conferencing, in: W. Edwards, R.F. Miles Jr., D. von
     (1979) 810–822, https://doi.org/10.1287/opre.27.4.810.                                           Winterfeldt (Eds.), Advances in Decision Analysis: From Foundations to
[45] R.L Keeney, Developing objectives and attributes, in: W. Edwards, R.F. Miles Jr.,                Applications, Cambridge University Press, 2007, pp. 375–399.
     D. von Winterfeldt (Eds.), Advances in Decision Analysis: From Foundations to               [64] T.Y. Chen, H.F. Chang, Critical success factors and architecture of innovation
     Applications, Cambridge University Press, 2007, pp. 104–128.                                     services models in data industry, Expert Syst. Appl. 213 (2023) 119014, https://
[46] B. Fasolo, C.A. Bana e Costa, Tailoring value elicitation to decision makers'                    doi.org/10.1016/j.eswa.2022.119014.
     numeracy and fluency: expressing value judgments in numbers or words, Omega                 [65] C.M. Smith, D. Shaw, The characteristics of problem structuring methods: a
     44 (0) (2014) 83–90, https://doi.org/10.1016/j.omega.2013.09.006.                                literature review, Eur. J. Oper. Res. 274 (2) (2019) 403–416, https://doi.org/
[47] C.A. Bana e Costa, E.C. Corrêa, J.M. De Corte, J.C. Vansnick, Facilitating bid                  10.1016/j.ejor.2018.05.003.
     evaluation in public call for tenders: a socio-technical approach, Omega 30 (3)             [66] M. Marttunen, J. Lienert, V. Belton, Structuring problems for multi-criteria
     (2002) 227–242, https://doi.org/10.1016/S0305-0483(02)00029-4.                                   decision analysis in practice: a literature review of method combinations, Eur. J.
[48] R.L. Keeney, H. Raiffa, Decisions With Multiple Objectives: Preferences and Value                Oper. Res. 263 (1) (2017) 1–17, https://doi.org/10.1016/j.ejor.2017.04.041.
     Tradeoffs, John Wiley & Sons, 1976.                                                         [67] C.A. Bana e Costa, J.C. Lourenço, M.P. Chagas, J.C. Bana e Costa, Development of
[49] W. Edwards, F.H. Barron, SMARTS and SMARTER: improved simple methods for                         reusable bid evaluation models for the Portuguese Electric Transmission Company,
     multiattribute utility measurement, Organ. Behav. Hum. Decis. Process. 60 (3)                    Decis. Anal. 5 (1) (2008) 22–42, https://doi.org/10.1287/deca.1080.0104.
     (1994) 306–325, https://doi.org/10.1006/obhd.1994.1087.                                     [68] D. Clegg, R. Barker, Case Method Fast-Track: A RAD Approach, Addison-Wesley
[50] C.A. Bana e Costa, J.C. Vansnick, MACBETH – An interactive path towards the                      Longman Publishing, 1994.
     construction of cardinal value functions, Int. Trans. Oper. Res. 1 (4) (1994)               [69] M. Weber, Decision making with incomplete information, Eur. J. Oper. Res. 28 (1)
     489–500, https://doi.org/10.1016/0969-6016(94)90010-8.                                           (1987) 44–57, https://doi.org/10.1016/0377-2217(87)90168-8.
[51] C.A. Bana e Costa, J.M. De Corte, J.C. Vansnick, MACBETH, Int. J. Inf. Technol.             [70] C.A. Bana e Costa, P. Vincke, Measuring credibility of compensatory preference
     Decis. Mak. 11 (2) (2012) 359–387, https://doi.org/10.1142/                                      statements when trade-offs are interval determined, Theory Decis. 39 (2) (1995)
     S0219622012400068.                                                                               127–155, https://doi.org/10.1007/BF01078981.
[52] C.A. Bana e Costa, J.M. De Corte, J.C. Vansnick, On the mathematical foundations            [71] L.D. Phillips, A theory of requisite decision models, Acta Psychol. 56 (1–3) (1984)
     of MACBETH, in: S. Greco, M. Ehrgott, J.R. Figueira (Eds.), Multiple Criteria                    29–48, https://doi.org/10.1016/0001-6918(84)90005-2.
     Decision Analysis: State of the Art Surveys, Springer, 2016, pp. 421–463, https://          [72] J. Pereira, J. Varajão, N. Takagi, Evaluation of information systems project
     doi.org/10.1007/978-1-4939-3094-4_11.                                                            success–insights from practitioners, Inf. Syst. Manag. (2021) 1–18, https://doi.org/
[53] W. Edwards, How to use multiattribute utility measurement for social                             10.1080/10580530.2021.1887982.
     decisionmaking, IEEE Trans. Syst. Man Cybern. 7 (5) (1977) 326–340, https://doi.            [73] N. Takagi, J. Varajão, ISO 21502 and Success Management: A Required Marriage in
     org/10.1109/TSMC.1977.4309720.                                                                   Project Management, SAGE Open, 2025, pp. 1–11, https://doi.org/10.1177/
[54] D. von Winterfeldt, W. Edwards, Decision Analysis and Behavioral Research,                       21582440251355046. July-September.
     Cambridge University Press, 1986.                                                           [74] F.A.F. Ferreira, S.P. Santos, Two decades on the MACBETH approach: a
[55] C.W. Kirkwood, Strategic Decision Making: Multiobjective Decision Analysis with                  bibliometric analysis, Ann. Oper. Res. 296 (1) (2021) 901–925, https://doi.org/
     Spreadsheets, Duxbury Press, 1997.                                                               10.1007/s10479-018-3083-9v.
[56] L.I. Assalaarachchi, M.P.P. Liyanage, C. Hewagamage, A framework of critical                [75] J. Varajão, L. Lopes, A. Tenera, Framework of standards, guides and methodologies
     success factors of cloud-based project management software adoption, Int. J. Inf.                for project, program, portfolio, and PMO management, Comput. Stand. Interfaces
     Syst. Proj. Manag. 13 (2) (2025) e4, https://doi.org/10.12821/ijispm130204.                      92 (2025) (2025) 103888, https://doi.org/10.1016/j.csi.2024.103888.
[57] N. Pinheiro, J. Vrajão, I. Moura, Success factors of public sector information             [76] I.I. Mitroff, T.R. Featheringham, On systemic problem solving and the error of the
     systems projects in developing countries, Sustain. Futures 10 (2025) (2025)                      third kind, Behav. Sci. 19 (6) (1974) 383–393, https://doi.org/10.1002/
     101095, https://doi.org/10.1016/j.sftr.2025.101095.                                              bs.3830190605.
[58] J. Jayakody, W. Wijayanayake, Critical success factors for DevOps adoption in               [77] J. Varajão, Success Management as a PM knowledge area – work-in-progress,
     information systems development, Int. J. Inf. Syst. Proj. Manag. 11 (3) (2023)                   Procedia Comput. Sci. 100 (2016) (2016) 1095–1102, https://doi.org/10.1016/j.
     60–82, https://doi.org/10.12821/ijispm110304.                                                    procs.2016.09.256.
[59] K. Schwaber, J. Sutherland, The Scrum Guide - The Definitive Guide to Scrum: The            [78] Y. Kong, N. Zhang, Z. Duan, B. Yu, Collaboration with generative AI to improve
     Rules of the Game, scrumguides.org, 2020. https://scrumguides.org/docs/sc                        requirements change, Comput. Stand. Interfaces 94 (2025) (2025) 104013, https://
     rumguide/v2020/2020-Scrum-Guide-US.pdf.                                                          doi.org/10.1016/j.csi.2025.104013.
[60] M. Jovanovic, A.L. Mesquida, A. Mas, R. Colomo-Palacios, Agile transition and
     adoption frameworks, issues and factors: a systematic mapping, IEEE Access 8
     (2020) (2020) 15711–15735, https://doi.org/10.1109/ACCESS.2020.2967839.




                                                                                            14
