Formalising Ethical Constraints in Automated Decision Making


External Member

Dr Ignacio Ojea (ANU Philosophy)


Supervisors: The student will also have access to the expertise of other members of the Humanising Machine Intelligence Project (HMI), including Dr Pamela Robinson, Dr Claire Benn (philosophy), Dr Alban Grastien and Dr Hanna Kurniawati (computer science).

As autonomous agents become ubiquitous, it is increasingly important for these decision-making systems to align with our ethical values (Russell, Stuart, 2019.). One way to incorporate ethical values is to impose constraints on the decision problem, but there is so far no principled way of doing this. Furthermore, the incorporation of constraints can increase the complexity of the decision problem.

In this project, we will consider Markov Decision Processes (MDPs) and/or related stochastic shortest paths problems as the decision-making paradigm. Constraints on such decision problems can be placed on actions, states, state/action trajectories, or on the reward function. But the complexity of solving for the optimal policy depends on the form of the constraints, the formalism used to encode them, the type of policy sought (deterministic vs. stochastic), and on the solution method used for solving the MDP. Successfully implementing ethical constraints will require a careful analysis of the different options.

The literature on ethical constraints for autonomous decision making is still nascent (Bringsjord, Arkoudas, and Bello 2006, Lindner et al, 2019, Svegliato et al. 2020, Nashed et al. 2021). There are opportunities to make significant contributions to it, both to the project of rigorously formalising important philosophical concepts, and to the project of exploring the computational limitations, complexity and implementation of these formalisations.

In this thesis, we will first identify and taxonomise the class of ethical constraints that are both philosophically significant and formalisable as constraints over MDPs. Second, we will compare the levels of added complexity that those formalised constraints add to decision problems. Third, we will implement the most promising formalisations within an MDP planner.


  • Taxonomise and formalise a relevant class of ethical constraints.
  • Determine the added complexity of finding the optimal policy under different constraints and different formalisations of those constraints.
  • Implement formalisations of ethical constraints with an MDP planner.


  • An understanding of the basics of computational complexity and complexity classes. A love of formalising!
  • An interest in understanding moral philosophy.
  • An understanding of AI planning or Markov decision processes at the level provided by the AI or Advanced Topics in AI class COMP3620/COMP4620.

Background Literature

  • Felix Lindner, Robert Mattmüller, Bernhard Nebel: Evaluation of the moral permissibility of action plans. Artif. Intell. 287: 103350 (2020). Conference version in AAAI-19, pp 7635-7642.
  • Justin Svegliato, Samer Nashed, Shlomo Zilberstein: An Integrated Approach to Moral Autonomous Systems. ECAI 2020: 2941-2942
  • Samer Nashed, Justin Svegliato, Shlomo Zilberstein: Ethically Compliant Planning within Moral Communities. AIES 2021
  • Selmer Bringsjord, Konstantine Arkoudas, Paul Bello: Toward a general logicist methodology for engineering ethically correct robots (2006), Intelligent Systems 22.
  • Felipe W. Trevizan, Sylvie Thiébaux, Pedro Henrique Santana, Brian Charles Williams: Heuristic Search in Dual Space for Constrained Stochastic Shortest Path Problems. ICAPS 2016: 326-334
  • Peter Baumgartner, Sylvie Thiébaux, Felipe W. Trevizan: Heuristic Search Planning With Multi-Objective Probabilistic LTL Constraints. KR 2018: 415-424
  • Russ Shafer-Landau (2012) The Fundamentals of Ethics 2nd Ed; Introduction and Chapters 9-14, 17.
  • Russell, Stuart, 2019. Human Compatible: Artificial Intelligence and the Problem of Control.
  • John Stewart Mill, (1863) Utilitarianism, Ch 1,2,4.


  • You will pursue state-of-the-art research in Artificial Intelligence and Ethics.
  • The work, if successful, is (hopefully) going to be published in a top venue.
  • If successful, we can easily extend this to a PhD project.


  • Autonomous Decision Making
  • Ethics of Artificial Intelligence
  • Complexity Theory
  • Markov Decision Processes
  • AI Planning

Updated:  10 August 2021/Responsible Officer:  Dean, CECS/Page Contact:  CECS Marketing