AgentOS concept map and taxonomy

[nickjalbert]

Starting a discussion on the concepts proposed in the main slide deck. Let me know if it's better to comment directly in the slides or make a google doc!

[nickjalbert]

Thanks for thinking about this and writing it up @andyk! Jotting down some notes and thoughts here.

Project - an AgentOS project is an MLflow...

This seems like it might be easy to confuse with an "Agent Project" defined further down (would the full name for the second entry be an "AgentOS Agent Project"?).

a component must exist within a project

I know we've been discussed this in a few places. I gather one goal of this requirement is to allow multiple components to live in one repo. Is requiring this to be an {AgentOS, MLflow} project useful because that homogenizes the interface and allows for recursive composition (e.g. my Agent Project is made of a bunch of Component [Sub]Projects)? If so, is it worth noting that a Project can contain other Projects (sort of like Module in some of the NN libs)? Does MLflow Projects give us any nice features for composing projects?

[...more to come!]

[nickjalbert]

A few more comments (based on a combination of the slide and our meeting):

• I think the idea of a more dynamic dependency structure is cool and I'm interested in trying it out (maybe port another Acme agent or an RLlib agent?). Two main concerns (echoing from my meeting comments below):

  • Arbitrary dependency structures seem like they could complicate writing good support tools for agent development because those tools will have to support agents that are more general (or might not even look like agents in our current conception).
  • Will we be able to convince real agent developers that this tooling is worth the cost/benefit of including in their projects?

• I'm not quite convinced about the value-add of MLflow Projects yet (vs MLflow Tracking, which I think will be great for us), but it also seems lightweight enough that we can implement it and try it out.

• We might want to revisit your original idea that an agent is self-contained and self-described in a directory. My pitch (given my work and my understanding of your work) is that an agent is:

  • A set of components and dependencies captured in a manifest file (agentos.ini currently)
    • Can we add additional entries to an MLproject file and have that perform double duty as an agent manifest?
  • The implementation of those components if they are not available on github or similar (i.e. the local components)
  • The run history of that agent (including metrics which may be aggregated over the whole run history)
  • The backing data of that agent (e.g. serialized neural nets, etc)
  • The hyperparameters used to train and benchmark the agent
    • Could these be added to an MLproject file?

• In the above world, maybe we always require that one component be an environment and one component be an agent (that supports learn and run) but then everything else is up to the agent builder.

[nickjalbert]

Some notes from the zoom meeting between me and @andyk (9/22/2021):

• A desired outcome of this work is a more flexible model for porting existing agents from other frameworks. For example, Acme R2D2 had a number of structures and communication patterns that were difficult to accommodate in our initial architectural (e.g. passing the recurrent state to the reverb dataset was algorithmically important and not something we had facilities for). SB3 has a different set of structures and communication patterns which are difficult to accommodate in the architecture that fell out of the Acme port.

• A big barrier to development that we both noticed: having components separated into their own git repos tended to be unwieldy because edits would often have to be done across several components and this caused coordination headaches.

• Currently, we support a static graph of dependencies:

  • An Agent project has five Python classes we call components that are labeled Agent, Trainer, Policy, Dataset, Environment
  • Those classes get initialized by the runtime, and then the Agent class gets a pointer to the other four classes
  • We allow runtime dynamism by giving every class a pointer to the shared dictionary shared_data that, e.g., the Policy can use to communicate with the Dataset even though there is no static communication pathway between these components provided by AgentOS runtime (i.e. the Policy and Dataset do not have pointers to each other).

• In Andy's prototype, we allow each component to declare its dependencies in the agent manifest (not sure that's what we're currently calling this file, but the agentos.ini that specifies the components that make up the agent), so we might have something like this:

  [agent]
  file_path = /foo/bar/agent.py
  class_name = BasicAgent
  dependencies = ['dataset', 'policy', 'acme_r2d2==0.15.0#R2D2Trainer~trainer', 'cartpole#CartPole']
  
  [policy]
  file_path = /foo/bar/trainer.py
  class_name = R2D2Policy
  dependencies = ['dataset']
  
  [dataset]
  file_path = /foo/bar/dataset.py
  class_name = ReverbDataset
  dependencies = ['policy']
  

  Where the dependencies entry determines determines how the AgentOS runtime structures the agent; a component will get a pointer to all its dependencies from the AgentOS runtime. We distinguish between local components (i.e. agent, dataset, and policy that all refer to local labels in the above example) from published components that exist in AgentOS web registry (i.e. 'acme_r2d2==0.15.0#R2D2Trainer~trainer' and 'cartpole#CartPole' above).

• This allows us to more easily accommodate communication patterns unique to a particular RL framework without resorting to a kludge like shared_data.

• The above scheme allows for less headache in managing of components as it's easier to mix local and published components and they don't all have to be in separate repos (assuming that the web service does something smart when publishing a project with local components).

Some open questions:

• Does allowing a more dynamic definition of an agent allow for easier agent development?
• Alternatively, does having an "opinionated" view on the structure of an agent (e.g. it has five components etc) allow us to build tools that are more useful to agent developers because we can take advantage of that structure?

We hope to answer these questions by porting more agents!