An intelligent goblin

I've been exploring an idea for a decision-making system grounded less in contemporary AI techniques and more in traditional logic, philosophical psychology and metaphysics.

Ultimately, I'd like to explore questions such as how an individual thing possesses capabilities, how those capabilities change over its lifetime, how it perceives the world around it, and how perception might eventually lead to deliberation and action.

But before getting anywhere near that, I ran into a rather more concrete software question:

Can we use C# to give individual objects new capabilities at runtime, with natural semantics?

The original experiment

C# is fundamentally a statically typed language.

We normally define the structure and behaviour of an object through its CLR type. Once we've compiled a Person, the methods available to a Person are essentially determined.

But that isn't how every programming language approaches objects.

Ruby and Smalltalk were particular inspirations. They make it natural to think about objects and classes as things that can participate in a much more dynamic runtime environment.

So I started experimenting.

What would happen if I pushed C# in that direction?

Could I evaluate new C# code at runtime?

Could that code execute in the context of a particular object?

Could I define new methods and properties dynamically?

Could existing objects acquire new behaviour?

Could I send messages to them by name?

Could an object decide what to do when it received a message for which no method existed?

It turns out that C# and .NET will let you go surprisingly far.

The project is named Dynamic, and the source code is available on GitHub here.

Eval

One of the first pieces was runtime evaluation.

Conceptually:

Eval(...);

Instead of requiring all C# to have been compiled as part of the original application, the runtime can accept C# code, compile it using Roslyn, and execute it.

For example:

var result = Eval.Run("21 * 2");
        
Console.WriteLine(result); // 42

The expression is compiled using Roslyn, evaluated, and its result returned to the application.

Eval now supports running entire script files. Given this text file:

r.Name = "David Harper";
r.Title = "Software Engineer | C# • Delphi";
r.Availability = Availability.Immediately;

r.Eligible.UnitedKingdom.Options = Option.Remote;
r.Eligible.NorthEastUnitedKingdom.Options = Option.Remote | Option.Hybrid;

r.Achievements.Mvp.Organisation = "Embarcadero";       
r.Achievements.ArcticCodeVault("GitHub").Title = "Arctic Code Vault Contributor";

We can execute it like this:

var r = new Resume();

var text = File.ReadAllText("resume.txt");

Eval.Run(text, () => r);

ResumeWriter.Write(r, ResumeSection.All);

The example demonstrates how to capture objects for usage within the script (Resume).

Interestingly, Eligible does not have a property UnitedKingdom, nor does Achievements have a member Mvp.

Similary, we can create a skill on demand by declaring it:

Skill.ObjectPascal
    .ListedIn(Language)
    .Aka("OP")
    .Title("Object Pascal")
    .Versions(new[] { "3", "4", "5", "6", "7", "XE2", "10", "11", "12", "13" });

And dynamic properties, like Versions are created by declaration.

ListedIn begins the exploration of associations.

Instance Eval

Runtime evaluation becomes more interesting when the code is evaluated in the context of a particular object.

Suppose we have two people: 

dynamic bob = new Person { FirstName = "Bob" };
dynamic alice = new Person { FirstName = "Alice" };

We can evaluate new behaviour specifically against bob

bob.Eval("""
    public string DisplayName()
    {
        return FirstName;
    }

    public string Greeting()
    {
        return "Hello " + DisplayName();
    }
""");

Bob can now use that behaviour:

Console.WriteLine(bob.Greeting()); // Hello Bob
Console.WriteLine(alice.Greeting()); // Error - method doesn't exist

The important distinction is that the evaluated methods belong to this particular instance. Bob gains DisplayName and Greeting; Alice does not.

And the runtime-defined methods can refer directly to existing members such as FirstName, and one runtime-defined method can call another:

return "Hello " + DisplayName();

So this is no longer just arbitrary C# being compiled and executed at runtime. An individual object can acquire new behaviour while the application is running.

Class Eval

The experiment then went further.

Suppose we have two ordinary objects:

dynamic bob = new Person { FirstName = "Bob" };
dynamic alice = new Person { FirstName = "Alice" };

At runtime, we can evaluate code against their runtime class Person:

bob.ClassEval("""
        public string DisplayName()
        {
            return FirstName;
        }
    
        public string Greeting()
        {
            return "Hello " + DisplayName();
        }
        """);

And then:

Console.WriteLine(bob.Greeting());    // Hello Bob
Console.WriteLine(alice.Greeting());  // Hello Alice

The interesting part isn't merely that we've compiled some C# dynamically.

We've introduced behaviour at runtime that wasn't part of the original compiled Person type. Unlike instance evaluation, however, the behaviour is available to other instances of the same runtime class.

That behaviour can itself refer to other dynamically available behaviour.

At this point, C# is beginning to look rather less static than it normally does.

Generics

Generics are supported:

[Fact]
public void Eval_can_infer_generic_arguments_from_generic_interfaces()
{
    dynamic bob = NewPerson("Bob");

    bob.Eval(
"""
public IEnumerable<(T, V)> GetPairs<T, V>(IEnumerable<T> keys, IEnumerable<V> values)
{
    return keys.Zip(values, (k, v) => (k, v));
}
""");

    var keys = new[] { "A", "B", "C" };
    var values = new[] { 1, 2, 3 };

    IEnumerable<(string, int)> result = bob.GetPairs(keys, values);

    Assert.Equal(new[] { ("A", 1), ("B", 2), ("C", 3) }, result);
}

Expressions

Expressions can be used:

[Fact]
public void Eval_can_return_expression_tree()
{
    ActiveRuntime.Register(typeof(Person));

    dynamic bob = new Person { FirstName = "Bob" };

    bob.Eval(
"""
public Expression<Func<Person, bool>> FirstNameIs(string name)
{
    return person => person.FirstName == name;
}
""");

    Expression<Func<Person, bool>> specification = bob.FirstNameIs("Bob");

    Assert.Equal(ExpressionType.Equal, specification.Body.NodeType);

    var predicate = specification.Compile();

    Assert.True(predicate(bob));
}

Send

Once methods can exist dynamically, the next question is how to invoke them dynamically. We can use normal DLR dispatch, as in the previous examples. But the runtime also provides Send for message-based dispatch: 

var name = alice.Send("DisplayName");

Arguments can be supplied directly with the message: 

var result = alice.Send("Join", "A", "B");

There are also more command-like forms: 

var result1 = alice.Send("Add(10, 20)");
var result2 = alice.Send("Add 30 12");

So rather than writing: 

alice.SomeMethod(...);

The message itself can become runtime data.

Send resolves that message against the behaviour currently available to the object. That includes behaviour defined for the individual object, behaviour defined for its runtime class, and ordinary CLR methods.

For example, an individual object can override behaviour supplied by its runtime class:

alice.ClassEval("""
    public string Greeting()
    {
        return "Class";
    }
    """);

alice.Eval("""
    public string Greeting()
    {
        return "Object";
    }
    """);

Console.WriteLine(alice.Send("Greeting")); // Object

dynamic bob = new Person { FirstName = "Bob" };

Console.WriteLine(bob.Send("Greeting"));   // Class

Alice's object-level Greeting takes precedence, while Bob continues to use the behaviour defined for the runtime class.

Underneath this simple API, the runtime also handles CLR overload resolution, argument conversion, optional parameters, params arrays and fallback behaviour.

This moves the experiment closer to the message-oriented style that interests me in Ruby and Smalltalk.

MethodMissing

What happens when an object receives a message it cannot resolve?

The project follows Ruby's answer to this problem with method_missing.

A message is first resolved against the behaviour available to the object: its object-level methods, runtime class methods and ordinary CLR methods.

For example, an ordinary CLR method remains available through Send:

var result = alice.Send("ToString");

But if no matching method can be found, the message is routed to MethodMissing.

By default, an unresolved message behaves much as we would expect:

protected virtual object? MethodMissing(RuntimeMessage message)
{
    throw new MissingMethodException(
        $"Method '{message.Name}' was not found on '{typeof(TSelf).Name}'.");
}

However, MethodMissing is virtual, so a particular type can decide how it wants to respond to an otherwise unresolved message. For example, Person can override the default behaviour:

protected override object? MethodMissing(RuntimeMessage message)
{
    return $"Missing: {message.Name}";
}

Then an unknown message can be handled by the object:

var result = alice.Send("DoesNotExist");

Console.WriteLine(result); // Missing: DoesNotExist

MethodMissing therefore acts as the final interception point for messages that the normal dispatch system cannot resolve. An object can inspect the missing message and decide how to respond.

The runtime can also ask the same resolution system whether a method is currently available:

alice.HasMethod("SecretName");    // object-level method
alice.HasMethod("DisplayName");   // runtime class method
alice.HasMethod("ToString");      // CLR method
alice.HasMethod("DoesNotExist");  // false

Together, Send, HasMethod and MethodMissing give an object a runtime message interface spanning dynamically introduced behaviour and its underlying CLR type.

Underneath the surface

Underneath sits quite a collection of .NET machinery:

  • DynamicObject
  • the Dynamic Language Runtime (DLR)
  • Roslyn runtime compilation
  • reflection
  • runtime binders
  • argument conversion
  • syntax rewriting
  • dynamically defined methods and properties

There are some fascinating capabilities buried in .NET once you start exploring beneath ordinary C# syntax.

And the experiment works.

There is more that could be done, but the project now answers the question I originally asked:

Can C# do this?

Yes.

Quite a lot of it, in fact.

Here's a simple example from Program.cs:

using Dynamic;
using Dynamic.Runtime;

// our two dynamic objects

dynamic bob = new Person { FirstName = "Bob" };
dynamic alice = new Person { FirstName = "Alice" };

// validating object method precedence over class method

bob.ClassEval("""
              public string DisplayName()
              {
                  return FirstName;
              }

              public string Greeting()
              {
                  return "Hello " + DisplayName();
              }
              """);

Console.WriteLine(bob.Greeting());              // Hello Bob
Console.WriteLine(alice.Greeting());            // Hello Alice

alice.Eval("""
           public string Greeting()
           {
               return "Bonjour " + DisplayName();
           }
           """);

Console.WriteLine(bob.Greeting());               // Hello Bob
Console.WriteLine(alice.Greeting());             // Bonjour Alice

// overload and params resolution

Console.WriteLine(bob.Send("Double", 42L));      // calls the long overload
Console.WriteLine(bob.Send("Double", 42));       // calls the int overload
Console.WriteLine(bob.Send("Sum", 1, 2, 3, 4));  // handles params

// simple eval

Console.WriteLine(Eval.Run("18 + 24"));          // 42

// lexical scope

bob.ClassEval("""
              public string Nickname { get; set; }

              public string ScopeExample()
              {
                  var query =
                      from Nickname in new[] { "ONE", "TWO" }
                      select Nickname;

                  return string.Join(", ", query);
              }
              """);

bob.Nickname = "Robert";

Console.WriteLine(bob.ScopeExample());           // ONE, TWO
Console.WriteLine(bob.Nickname);                 // Robert

That simple example hides a surprising amount of plumbing, from argument and overload resolution and handling params, through runtime compilation, binding and lexical scope analysis, to rewriting this so dynamically installed code executes against the correct receiver.

So why am I pivoting? Two reasons.

Firstly, this framework can certainly be taken further, but it is structurally limited. I'd need to go much deeper into Ruby's internals—Ruby Under a Microscope is sitting on my reading list—and do a significant amount of work to reproduce further features such as include, extend, and so on. Then there's the natural progression towards composing types directly rather than evaluating raw text, followed by object tracking and a runtime manager allowing further introspection and dynamism.

Originally, I set out to layer some relatively simple dynamic behaviour over the DLR. But for my purposes, extending the runtime is gradually becoming the main work rather than supporting the work I actually want to do. Every additional piece pulls me further into building a runtime, while building a runtime was never the actual destination.

Secondly, I've realised something more fundamental:

I don't actually need to do this.

After getting the machinery working, I've started to think that I put the dynamism in the wrong place.

Perhaps changing what an individual can do doesn't require generating new executable behaviour at runtime at all.

The goblin remains a goblin.

What changes is what this particular goblin currently possesses and can exercise.

That suggests moving the dynamism away from code modification and into runtime composition.

The implementation of a capability can remain ordinary, statically compiled code. What changes dynamically is which capabilities belong to this particular individual, together with its current state and circumstances.

In concept, there's some similarity here to an Entity Component System (ECS): rather than continually changing an object's type, an individual can be composed from capabilities and state that can change at runtime.

But what I'm exploring is intended to be ontologically richer than a conventional ECS.

That sounds like a relatively small architectural change.

I increasingly suspect it isn't.

From mechanism to meaning

These distinctions have pushed me back toward the questions that motivated the experiment in the first place.

What does it mean to say that something can do something?

Is possessing a capability the same thing as being able to exercise it right now?

Presumably not.

An archer may possess the skill of archery while having no bow.

A person may know how to swim while currently being asleep.

An injured swordsman doesn't necessarily cease to possess the skill of sword fighting merely because an injury prevents its present exercise.

Those distinctions start sounding remarkably familiar if you've spent any time with Aristotle and Aquinas:

potency and act, substance and accident, powers and their exercise.

I'm becoming interested in whether taking the distinctions seriously might prevent us from reaching for the wrong computational abstraction too early.

The question is changing from:

How can I dynamically give this object another method?

to something closer to:

What is this thing, what powers does it possess, and under these circumstances, which of those powers can actually be exercised?

Dynamic has done its job

Dynamic answered the question it was built to investigate.

Building it exposed a more important question:

What should the model actually mean?

That's the question I'm taking into the next experiment.

An intelligent goblin

I want my goblins to perceive things.

Hear things.

Remember things.

Draw conclusions from incomplete information.

Possess different capabilities.

Be affected by their current condition.

Choose among actions actually available to them.

And occasionally make mistakes.

So that's where this journey is heading.

I want my goblin to think.

And apparently I've decided that the route to achieving that might pass through Aristotle, Aquinas, C#, the DLR, Roslyn, Ruby and a dungeon.

We'll see where the goblin takes me. 😄