This paper is based on the authors' experience in creating the LiquidFeedback application development and decision-making process and in applying it in practice in a variety of use cases. The authors always pursued the goal of making all process design issues scientifically sound. However, it became apparent time and again that the existing mathematical models and the findings of social choice and game theory alone are neither sufficient to solve all process design issues in the conception of an application development and decision-making procedure nor provide enough guidance to plan practical application scenarios. The complexity of such procedures and their application in practice is principally high and the different aspects are interwoven, partly mutually dependent, and have an impact on each other while the existing findings are always based on massively simplified models that can only consider a small section of the overall problem. As an example, even more advanced considerations of complex relationships such as the mathematical analysis of delegation graphs in transitive delegation systems make massive simplifications of the model compared to reality, for example, the temporal dimension and the resulting dynamics are not considered. [Temporal] While it is understandable – and presumably also necessary – to severely limit the models to avoid unmangeable complexity, the consequence is necessarily that – while these models are of very high academic interest – only very limited conclusions can be drawn about reality from these models.
Nevertheless, this paper does not attempt to find more complex models or even to advocate the use of more comprehensive models in the future. Instead, the authors report on the process design issues they have faced in order to highlight possible questions whose consideration in science could be of benefit to practice. At the same time, the authors want to motivate to think outside the box when modelling application development and decision-making systems in the future, to keep the overall complexity in mind, and to consider these limiting factors when formulating their own findings. Furthermore, the authors place great hope in scientific approaches that go beyond small-scale modelling and attempt to empirically capture larger contexts by means of computer simulation (using so-called digital twins), and would like to give these research projects the wealth of process design issues that the authors were confronted with.
But even this paper has to simplify. Mainly the aspect of decision-making is considered, while the aspect of proposal development is left out. This does not at all mean that these two aspects can be considered separately without further ado. On the contrary, both the theoretical considerations of the authors and practical experience have shown that a mutual dependence of application development and decision-making cannot be ruled out. For the sake of manageability of the overall complexity of the topic, mainly the aspect of decision-making will be considered in this article.
Before addressing the specific design issues, the use of the terms “decision” and “decision-making system” should first be clarified. Decisions can cover a wide range of topics, but they have one thing in common: they always lead to some kind of action. This can be, for example, the enactment of a law, the execution of a financial transaction, or the installation of MPs in a parliament. However, even non-binding decisions usually result in an action, such as the preparation of a minute. This allows us to define it as follows: given a set of possible, mutually exclusive actions (abbreviated as MEA in the following sections), a decision is the act of selecting exactly one of these actions to be performed. A decision-making system, in turn, consists of certain procedural rules that enable such decisions to be made for certain sets of MEAs.
In practice, however, this model is not very helpful, as it quickly becomes confusing in multidimensional decision spaces. Let us consider, for example, the election of a parliament in which 600 seats are to be filled and there are a total of 2000 candidates. As can easily be shown, in this case there is the sheer quantity of more than 10^528 (a number with 529 digits) different constellations of how the parliament could be filled, i.e. more than 10^528 possible mutually exclusive actions. A meaningful analysis of the characteristics of the procedure is not possible on this basis.
In order to carry out analyses of the properties of decision-making procedures, another concept is used, especially in social choice theory: the concept of candidates. The concept of candidates must be clearly distinguished from the concept of MEAs. While the concept of MEAs looks at the overall outcome of a decision, the concept of candidates may deal with a smaller, divisible aspect of the decision. In the example above, the candidate can stand for one of the 2,000 people who run for election and can either get a seat in parliament – or not. In simple cases, these two concepts can coincide. For example, if the post of mayor is to be filled, i.e. exactly one person is to be selected from a set of persons, then each candidate corresponds to exactly one MEA.
When considering candidates, it is essential to bear in mind that in many cases there are different ways of defining candidates. Let us assume an election to a parliament in which party lists are up for election. In this case, both the party lists could be understood as candidates (since they can be selected on the ballot paper) and the individuals on the lists (since they will actually enter parliament later). Both variants result in exactly the same MEAs, but depending on how the candidate is defined, a different picture emerges when the scenario is considered theoretically.
One of the first questions that arises when planning a new participation process is the nature of the desired outcome. Should a single office, e.g. that of mayor, be filled or should a parliament be elected? Should the best proposal be selected from among several competing proposals, or should a set of proposals be selected for discussion at a convention? Or should athletes be evaluated by a jury according to a scoring system? These are just a few examples of the many different types of outcomes that are conceivable.
The question of the type of results can be clarified in a structured way with the help of the “candidates” discussed previously. A candidate can be a person, e.g. a mayoral candidate, but also a proposal. The concept of candidates allows us to distinguish decision systems according to how many candidates can become winners. Being among the winners means, for example, that candidates are elected to office, that a law or policy is made, etc.
So-called “single-winner” systems select at most one candidate as the winner, although some systems may select more than one candidate in the event of a tie. Some single-winner systems always select one of the candidates as the winner (e.g., the candidate with the most votes, regardless of whether a candidate can get a majority of the votes), while other systems may result in no winner (e.g., because the majority requirements are not met by any candidate).
So-called “multi-winner” systems result in multiple (or possibly all) candidates being selected as winners. The number of winners may be fixed or variable, and the winners may or may not be further ranked by the system.
Such ranking can be pre-orders (in the case of ties) or a total order (if there are no ties). The output can even include a score or other metric for each candidate, e.g., if the candidates are projects to be funded where each candidate may receive a different amount of funding.
Binary (“yes/no”) decisions can be viewed as a system that always selects a winner from two candidates “yes” and “no”, or as a system that selects one or zero winners where there is only one candidate, e.g., a single proposal. Accordingly, decisions between several alternative proposals can also be viewed in different ways, depending on whether the status quo is considered a candidate or not. As noted in the preliminary remarks, the use of different definitions for candidates leads to a formally different description of the system. If the status quo is included as a candidate, the decision system can be described as always producing a winner. If the status quo is omitted from the list of candidates, then the selection of the status quo is described by the fact that all candidates lose.
While it is easy to determine whether a multi- or single-winner system should be used, some accuracy is needed in the details, such as how to handle ties or whether it should be permissible to have no winner.
In addition to systems that select a number of winners, create a ranking, or assign a score to candidates, other forms of output are conceivable. For example, a decision-making system might produce a partial order as an output, where, for example, A is preferred over B, and C over D, but no statement is made on how A and C, A and D, B and C, or B and D compare. Such exotic forms of output could be particularly interesting when decision systems are chained, i.e., when the output of one decision is used as an input to another democratic process.
By feeding the outcome of one decision into another decision process, it is possible to create a decision system that consists of multiple rounds. Common examples are systems in which a second round of voting is conducted if more than one or less than one candidate has won the first round, but exactly one office, such as mayor, is ultimately to be filled.
As will be shown later, evaluating the mathematical properties of a system may be inadequate in practice because a system is embedded in a larger context in which subsequent decisions are made based on earlier developments.
Another important aspect in the planning of decision-making procedures is the question of how the different opinions of the participants are to be represented. This is particularly relevant for procedures in which more than one winner is to be determined.
Multi-winner systems usually result in some sort of representation of the electorate in the selection of winners. However, when multiple winners are selected, quite different goals may be pursued. In [LacknerSkowron], three principles are mentioned: “individual excellence”, “proportionality”, and “diversity”. See also [ElkindEtAll]. In the following, this terminology (“excellence”, “proportionality”, and “diversity”) will also be used.
In some cases, we do not primarily seek to represent the electorate, but simply to take into account the opinion of the electorate in order to select the (perceived) best candidates. We call this goal excellence. We can also look for a set of candidates (the winners of an election) that proportionally represent the electorate. Proportional systems are often used with the goal of diversity. However, diversity and proportionality are different goals, as the following example shows. Even though proportional systems always create a certain amount of diversity, we distinguish between diversity and proportionality.
To support the previous statements, let's look at the following example. There is a decision to be made to elect 9 people to a board. We assume three non-overlapping subgroups in the electorate:
We assume that A1 is preferred over A2, and so on, and that B1 is preferred over B2, and so on.
As we can see, there is a majority of 5/9 in favor of A1 to A9. An aggregation scheme that follows the principle of excellence would take into account the opinion of the majority and select A1 to A9 as the winners. However, in a proportional representation scheme, A1 to A5, B1 to B3, and C1 would have to win. A system that focuses on diversity (ignoring proportionality altogether), however, could select A1 to A3, B1 to B3, and C1 to C3 as winners, maximizing the representation of each group regardless of its size.
The main semantic difference between proportionality and diversity is that diversity aims to represent different groups in the electorate regardless of their size, whereas proportionality takes into account the size of these groups. With excellence, on the other hand, a majority gets all the power and minority opinions may be completely ignored as a result.
Note that our definition of diversity differs from the definition in [LacknerSkowron], which states that in systems that aim for diversity, “there is no or little weight put on giving voters a second representative in the committee”. Lackner and Skowron thus distinguish whether voters get one or more than one preferred representative elected. In contrast, we believe a better definition is that a system that maximizes diversity (and disregards proportionality and excellence altogether) aims to maximize representation for each group, regardless of size, and to go beyond a single representative for each group.
In [ElkindEtAll] there is an example of selecting movies for passengers on a airplane given as an illustration where diversity is desired:
If the flight is short enough, then it would be sufficient if each member gets at least one suitable movie of their choice provided. However, the example given in [ElkindEtAll] would be semantically different if it were a several day long cruise where passengers could easily see more than one of their favorite movies. We therefore conclude that limiting representation to one candidate per group is a restriction on the principle of diversity.
In many real-world cases, the situation is more complex than having non-overlapping groups with non-overlapping favorite candidates. In these cases, it is more difficult to find appropriate definitions and criteria for excellence, proportionality, and diversity. However, the above example is sufficient to show that these three goals justify different outcomes in at least some cases.
When selecting a multi-winner system for a particular use case, it is therefore important to have clarity about the three objectives excellence, proportionality, and diversity. In particular, a system that is suitable for one case may be completely unsuitable for another scenario.
It should be noted that in many situations a combination of two or three of the goals excellence, proportionality, and diversity may be desired. LiquidFeedback, for example, uses two different algorithms for sorting competing initiatives and possibly complementing suggestions. [PLF, subsection 4.10.2] While both fulfill certain aspects of proportionality, it could be argued that the algorithm for sorting competing initiatives (Harmonic Weighting) creates more diverse results because “participants who gain a good display position with one of their supported initiatives get less voting weight for their other supported initiatives to increase other people's ability to be represented as well.” [Evolution]
Another important feature of decision-making systems is whether they disclose information that is beyond the aggregated results. In particular, some decision-making systems disclose information about voters' ballots, while other systems guarantee secrecy for each individual. [Note: Care should be taken when using the term “anonymity” in this context, as social choice theory commonly uses the term anonymity to describe voting systems in which every voter is treated equally.]
Sometimes the disclosed information is published in a structured form and is thus equally accessible to every participant or the general public (roll call votes), sometimes this information is only partially accessible (e.g., votes by show of hands).
Conducting secret votes, i.e., votes in which each participant's voting behavior remains secret, is an important way to ensure that voters can cast their ballots freely without fear of personal consequences based on their own votes. This could lead to the erroneous conclusion that secret ballots are generally beneficial. However, conducting secret ballots also has some disadvantages.
First of all, ballot boxes must be used to ensure secrecy. If verifiability is a requirement, there must be some kind of public process that can be monitored by participants (so that no person can place more than one ballot in the ballot box). The preparation of the ballot box and the counting must also be public. This effectively eliminates the use of electronic systems. [PLF, Chapter 3]
Second, making members of the electorate accountable for their votes can sometimes have a desirable effect. A common example of this are decisions in parliaments, where each member of parliament is supposed to be held accountable for his or her decisions.
Another potential problem is that, in a secret ballot, it is possible for voters to vote tactically without incurring personal consequences if their actions are successful. This may increase the incentive to vote tactically and lead to unequal treatment of voters depending on which aggregation rule is used.
Moreover, the disclosure of information may not be obvious, as the following example shows. Consider a preferential voting system in which 20 candidates must be ranked by each voter. Voting is done on paper with a ballot box so that the information about which voter cast which ballot remains completely secret. However, the contents of the ballots are made public for verifiability. There are 2432902008176640000 ways to fill out a ballot (factorial of 20). If we consider only the possibilities that put a particular candidate first, there are still 121645100408832000 possibilities (factorial of 19).
A voter could then be blackmailed into filling out the ballot in exactly one of these ways. If all ballots are published (without link to the respective voters), a blackmailer can verify that the voter did not vote as instructed, since it is very unlikely that any of the 121645100408832000 possible choices would randomly appear in the set of ballots cast. Thus, secret voting in the strict sense (i.e., no information leakage) makes it virtually impossible to use voting systems that allow voters to express their preferences.
There may also be disclosure of information that does not relate to specific voters. If the voting system publishes intermediate results (e.g., a provisional winner based on votes cast before the election is closed), this may also lead to tactical considerations by voters or even unstable swing behavior. [GoD]
Disclosure of intermediate results is sometimes not obvious. Think of a vote by a show of hands. Not all hands are raised at the same time. A voter may look around the room to see how many other voters are raising their hands, and then decide whether to raise his or her own hand. It is also possible to put the hand down quickly. These effects must also be taken into account when assessing the likelihood of a tactical vote.
Some aggregation functions may be more appropriate than others for cases in which intermediate results are published. However, assessing the vulnerability of voting systems to tactical voting is not an easy task and may require game theoretic and psychological models.
Decision-making systems can follow deterministic rules or contain non-deterministic elements. Non-deterministic systems can lead to different results even if the same input is given. At first glance, this seems to be an unfair element. But apart from the use of randomness in case of ties (which also makes a decision-making system non-deterministic), even extreme cases such as “random dictatorship” models can have legitimate use cases and may even lead to more fairness in some circumstances.
Consider an organization that needs to decide where to hold its annual meeting. Suppose there are two locations to choose from. 60% of the voters prefer one location, while 40% prefer the other. If it is a normal majority vote, the location that 60% of the voters prefer will be chosen. This is true even if the procedure is repeated every year.
A non-deterministic system, in contrast, might choose the location based on “random dictatorship”, a concept from social choice theory in which a person is randomly selected to decide the outcome of a decision. Returning to the example, this would mean that the probability of one site being chosen is 60% and the probability of the other site being chosen is 40%. If this procedure is repeated every year, both groups will be represented in the overall result, even though only one group may be represented each year. In the long run, one location will be chosen for 60% of the years and the other location for 40% of the years.
But even in cases where not a single decision is repeated, but many different decisions are made on different issues, nondeterministic systems can achieve a kind of fairness by favoring certain voters based on probabilities, whereas deterministic systems can cause minorities to lose over and over again, especially in the case of single-winner systems where there can be only one winner.
At least in the case of ties, non-deterministic elements are very common in decision-making systems. However, some decisions are made just on the basis of chance, e.g., when it comes to deciding which teams will compete against each other in a sports league. Here, there is not even an electorate that decides, but the outcome depends solely on the list of candidates and chance.
However, non-deterministic systems can still be problematic for a single decision, as it may not reflect the opinion of the electorate. Moreover, when there are interrelated decisions to be made, nondeterministic rules could lead to an overall inconsistent process.
There are also many details to consider when it comes to what input voters can make. The choice of how voters can express themselves may have a significant impact on the process. Hereafter, the term “ballot” is used to refer to the input of a member of the electorate in a decision-making process in a single pass, regardless of whether physical “ballots” are actually used or whether other forms of input are used (e.g., electronic systems).
In most democratic cases, each person's input is treated independently of who cast the vote, i.e., it is possible to swap two people's input without affecting the outcome. [Note: This is called anonymity in social choice theory.] In many decision-making systems, especially those that can be formalized, each person's input is restricted to a particular scheme. For example, each voter might be allowed to select only one candidate from a list of candidates.
Another possibility would be that each voter can select as many candidates as he or she wishes, and can specify a ranking, and so on.
There are different types of rankings, depending on whether all candidates must be ranked in a distinct order by a voter or whether candidates can be ranked equally. Mathematically, a ranked list is a total (also: linear) order when no equal ranks are allowed, and a total pre-order when two or more candidates can be ranked equally. When the “status quo” is one candidate, this effectively results in up to three sections of the ballot: the candidates considered better, equal, or worse than the status quo.
Certain restrictions may apply to such rankings. For example, each voter could specify only up to 5 ranks to which each candidate must be assigned. Or, to give another example, only three candidates may be included in the ranked list, and all other candidates are considered to be of equal rank and less preferred than the three mentioned. These restrictions exist either for practical reasons, such as to facilitate counting, or as a measure to prevent certain forms of tactical voting.
If there are no restrictions on the order of candidates on a ballot, the “unrestricted domain” criterion of social choice theory is satisfied. However, a voting system that satisfies this criterion may still be more restricted than certain other systems because it would be possible to include even more information on a ballot. While ranked ballots contain information about which candidates are preferred by a particular voter over other candidates, they do not contain information about how much they are preferred. One way to express such information is for the voter to assign a score (e.g., from 0 to 10) to each candidate.
Another possibility is to allow voters to omit certain information from their ballot, but still allow them to indicate a total order or a total pre-order if they so choose. This expands the voter's options. One example is the ability to indicate a partial order on the ballot. [BrillTalmon] For example, they can rank candidate A ahead of B and candidate C ahead of D, but make no statement about how A or B compare to C or D. Such ballots can be of use when voters want to explicitly state that they cannot make judgments about how some candidates compare, which can be used in systems of liquid democracy.
Depending on the degree of freedom each voter has in expressing his or her opinion on a ballot, different forms of tactical voting are possible. For example, the Gibbard-Satterthwaite theorem [Gibbard] shows that tactical voting is possible in voting systems that satisfy an “unrestricted domain”, i.e., systems that allow each voter to indicate an overall order of all candidates on his or her ballot. But the problem of tactical voting exists in other voting systems as well. If a voter cannot express his or her true preferences, he or she must make a (possibly tactical) decision about what information to omit. For example, if only one candidate can be on the ballot, it may make sense not to put the preferred candidate on the ballot, but one who is thought to have a better chance of winning. If all voters make such considerations, incorrect assessments about the other voters could cause a candidate to lose who is actually preferred by a majority of voters over all other candidates.
But even when it is possible for a voter to express more information than mere preferences on a ballot, this can open up forms of tactical voting. For example, in score voting, where each voter assigns a number of points (e.g., from 0 to 10) to each candidate and the candidate with the best average score wins, voters may try to tweak the scores to increase their influence (i.e., only assign a score of 0 or 10).
In general, assigning semantics to ratings (e.g., 0 is “very bad”, 5 is “I can live with it”, 10 is “very good”) can help prevent voters from voting tactically, but ultimately these labels are irrelevant to the counting process. Voters who disregard these semantics and rate each candidate as either “very bad” or “very good” will have a greater impact. Honest voters, on the other hand, who would not rate any candidate as “very good,” for example, will lose influence because they are honest.
There are different motivations for voting tactically. In general, voters' motivations for voting in a particular way-whether honestly or tactically-can be classified according to whether they are conditioned by the voter's ballot or by the overall outcome of a decision. Psychological effects aside, voters are generally not influenced by their own ballots when all ballots are secret (i.e., when a ballot box is used).
But the outcome of the vote can still affect voters. Depending on the scenario, all voters may be affected in the same way (e.g., a decision about how to prevent a planet-threatening asteroid impact), or some voters may have an advantage and other voters may have a disadvantage (e.g., a decision about a property tax). In the latter scenario, motivation is driven by individual advantage; in the former scenario, motivation arises from personal opinion about what is the best way to avoid the catastrophe for humanity“
As mentioned above, the quantitative assessment of tactical voting can go beyond classical voting theory and also requires game-theoretic and psychological models. In general, the longer a voting system is used, the more familiar voters become with the effects of a particular voting behavior. While voters may initially follow the nominal labeling of voting options (e.g., “in favor”, “neutral”, “against”), over time the labeling becomes irrelevant, and voters consider only the actual impact of the mark on a ballot. This is very important to keep in mind when drawing conclusions from theorems proven for a particular system, because if voters are assumed to use ballots in a particular way (i.e., the intended way of filling in the ballot), one cannot assume that the conclusions correspond to the real properties of the system.
But even when voter input is reduced to the minimum, namely the opportunity to vote “yes” or “no”, there may still be reasons for voters not to vote honestly. First, in some cases, there may be a reason to abstain from voting if a quorum of participants must be reached in order to make decisions. For example, if there is a rule that at least 100 voters must participate in a vote, it may make sense to abstain rather than vote “no”. Such a situation could be given in an example where 90 people vote for a proposal and 9 people vote against it. With an abstention, the quorum would not be reached, but with a “no” vote, there would be a total of 100 votes, and the proposal would win. Of course, this is an ill-conceived voting system, but such rules are not uncommon. [Jung]
But even excluding such considerations, there is another possibility for tactical behavior. This is because in the real world, no decision is made independently of others. Consider the following example:
There are three independent proposals A, B and C. Any combination of these three proposals could be adopted (i.e. none, one of the three, two of the three, or all three). The electorate is divided into three groups (Groups I, II, and III) of 40%, 35%, and 25%, respectively. Group I is in favor of Proposal A, but against Proposals B and C. Group II is for Proposal B, but against Proposals A and C. And Group III is in favor of Proposal C, but against Proposals A and B. If we further assume that Group II and Group III are more interested in their favored proposal winning than in the proposals they oppose losing. Then groups II and III could get together and promise to vote for both B and C, which would lead to the following voting results:
Thus, the proposals that are preferred by the smallest groups win. In this case, voters also deviate from honestly answering a ballot, although according to social choice theory there would be no reason to do if a single yes/no-voting per proposal is considered isolatedly from the context. As no decision-making procedure will be free of context in the real-world, findings of social choice theory must always be interpreted with the necessary caution in regard to the assumed models.
Another challenge that may be encountered is the practical implementation of the complex computational steps required for the tallying of some aggregation procedures. A good example are single transferable vote (STV) systems, in which ballots are sorted into piles based on the voter's first preference and then transferred to other piles if a candidate receives more than the quota required for the election. Depending on the number of candidates and voters, as well as the exact counting rules, the implementation of such a procedure may take more than a few hours, but days or even be completely impracticable. There is an additional challenge here if the procedure is to be verifiable by the public, i.e. if it must be ensured that the ballots cannot be tampered with during the entire period between the close of voting and the completion of the count.
But even in electronic counting systems where all ballots are published, the complexity of the calculations can be a challenge. Some algorithms may have asymptotic computational complexity beyond practical applicability, e.g., polynomial or even higher order exponential or factorial complexity.
For example, the Schulze method used in LiquidFeedback first performs a pairwise comparison of all candidates. Doubling the number of candidates quadruples the number of pairwise comparisons, and a hundred times the number of candidates requires 10,000 times as many comparisons (with each comparison also taking more time because the ballots contain more information). Computer programs can easily handle the computational complexity of the Schulze method in practice, but this is not necessarily true for other algorithms.
In some cases, manual verifiability might be desirable. Fortunately, the complexity of verifying a result can be less than calculating the result itself. In practice, we have found that many results of decisions in LiquidFeedback lead to a small Schwartz set (the smallest set where every candidate within the set is not beaten by any candidate outside the set in pairwise comparison). In these cases, the calculated result can be checked manually relatively easily.
In addition to the computational effort, the complexity of the software code can also be a problem. The more complicated an algorithm is, the more difficult it would be to create an independent computer program to verify the results.
Ultimately, there may be a tradeoff between meeting certain characteristics and implementation complexity. It may therefore be appropriate, for reasons of simplicity and/or practicality, to use voting aggregation procedures that do not meet certain desired criteria.
Sometimes legal requirements or contractual agreements may limit the choice of an appropriate decision-making system. For example, systems that have a negative voting weight could be deemed inadmissible for certain scenarios. Unfortunately, social choice theory shows that sometimes a violation of some desired properties is inevitable. [Moulin] [PrefDeleg] When weighing the pros and cons of alternative decision-making systems, legal considerations may lead to different decisions than if there were no legal constraints.
In the previous sections, we have dealt with some fundamental questions and problems that the authors have encountered in practice.
The very first step is always to ask what kind of outcome is expected in the beginning, e.g., whether a single winner, a group of representatives, or a binary decision on a single proposal is expected. Directly related to this is the embedding in multi-stage decision-making procedures with multiple rounds. Moreover, when it comes to determining multiple winners, there is always the question of whether representation should aim for excellence, proportionality, diversity, or a mix of these. The question of whether a decision should be made in an open or a secret procedure also often takes on special significance, whereby regularly the intuitive feeling that secrecy only offers advantages can be deceptive. The use of non-deterministic elements in decision-making procedures may offer interesting potential in certain scenarios. Of particular importance is the question of what input participants can provide (e.g., whether only a single candidate can be selected, or multiple candidates can be ranked or scored), which is very difficult to evaluate in terms of impact on the overall decision process and the outcome. Practical issues of implementation and legal aspects should also not be underestimated.
It has been shown that there are very different types of decision processes and that a variety of aspects must be considered in their planning and implementation as well as in the development, design and selection of algorithms and their coding in software in order to adequately address the problem.