v1.0.0rc11

* Improve text
* Move to docs
* Autogenerate with sphinx
* Fix naming issue `environment.run` (double name)
* Add to tests

.dockerignore

@@ -1,5 +1,7 @@
 **/soil_output
 .*
+**/.*
 **/__pycache__
 __pycache__
 *.pyc
+**/backup

.gitignore

@@ -9,3 +9,4 @@ docs/_build*
 build/*
 dist/*
 prof
+backup

.gitlab-ci.yml

@@ -20,7 +20,7 @@ docker:
 test:
   tags:
     - docker
-  image: python:3.7
+  image: python:3.8
   stage: test
   script:
     - pip install -r requirements.txt -r test-requirements.txt
@@ -31,7 +31,7 @@ push_pypi:
     - tags
   tags:
     - docker
-  image: python:3.7
+  image: python:3.8
   stage: publish
   script:
     - echo $CI_COMMIT_TAG > soil/VERSION
@@ -44,7 +44,7 @@ check_pypi:
     - tags
   tags:
     - docker
-  image: python:3.7
+  image: python:3.8
   stage: check_published
   script:
     - pip install soil==$CI_COMMIT_TAG

CHANGELOG.md

@@ -3,21 +3,39 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [0.30 UNRELEASED]
+## [1.0.0 UNRELEASED]
+
+Version 1.0 will introduce multiple changes, especially on the `Simulation` class and anything related to how configuration is handled.
+For an explanation of the general changes in version 1.0, please refer to the file `docs/notes_v1.0.rst`.
+
 ### Added
-* Simple debugging capabilities in `soil.debugging`, with a custom `pdb.Debugger` subclass that exposes commands to list agents and their status and set breakpoints on states (for FSM agents). Try it with `soil --debug <simulation file>`
-* Ability to run 
-* Ability to 
-* The `soil.exporters` module to export the results of datacollectors (model.datacollector) into files at the end of trials/simulations
 * A modular set of classes for environments/models. Now the ability to configure the agents through an agent definition and a topology through a network configuration is split into two classes (`soil.agents.BaseEnvironment` for agents, `soil.agents.NetworkEnvironment` to add topology).
-* FSM agents can now have generators as states. They work similar to normal states, with one caveat. Only `time` values can be yielded, not a state. This is because the state will not change, it will be resumed after the yield, at the appropriate time. The return value *can* be a state, or a `(state, time)` tuple, just like in normal states.
+* Environments now have a class method to make them easier to use without a simulation`.run`. Notice that this is different from `run_model`, which is an instance method.
+* Ability to run simulations using mesa models
+* The `soil.exporters` module to export the results of datacollectors (`model.datacollector`) into files at the end of trials/simulations
+* Agents can now have generators or async functions as their step or as states. They work similar to normal functions, with one caveat in the case of `FSM`: only time values (a float, int or None) can be awaited or yielded, not a state. This is because the state will not change, it will be resumed after the yield, at the appropriate time. To return to a different state, use the `delay` and `at` functions of the state.
+* Simulations can now specify a `matrix` with possible values for every simulation parameter. The final parameters will be calculated based on the `parameters` used and a cartesian product (i.e., all possible combinations) of each parameter.
+* Simple debugging capabilities in `soil.debugging`, with a custom `pdb.Debugger` subclass that exposes commands to list agents and their status and set breakpoints on states (for FSM agents). Try it with `soil --debug <simulation file>`
+* The `agent.after` and `agent.at` methods, to avoid having to return a time manually.
 ### Changed
-* Configuration schema is very different now. Check `soil.config` for more information. We are also using Pydantic for (de)serialization.
-* There may be more than one topology/network in the simulation
-* Ability
+* Configuration schema (`Simulation`) is very simplified. All simulations should be checked
+* Model / environment variables are expected (but not enforced) to be a single value. This is done to more closely align with mesa
+* `Exporter.iteration_end` now takes two parameters: `env` (same as before) and `params` (specific parameters for this environment). We considered including a `parameters` attribute in the environment, but this would not be compatible with mesa.
+* `num_trials` renamed to `iterations`
+* General renaming of `trial` to `iteration`, to work better with `mesa`
+* `model_parameters` renamed to `parameters` in simulation
+* Simulation results for every iteration of a simulation with the same name are stored in a single `sqlite` database
+
 ### Removed
+* The `time.When` and `time.Cond` classes are removed
 * Any `tsih` and `History` integration in the main classes. To record the state of environments/agents, just use a datacollector. In some cases this may be slower or consume more memory than the previous system. However, few cases actually used the full potential of the history, and it came at the cost of unnecessary complexity and worse performance for the majority of cases.
 
+## [0.20.8]
+### Changed
+* Tsih bumped to version 0.1.8
+### Fixed
+* Mentions to `id` in docs. It should be `state_id` now.
+* Fixed bug: environment agents were not being added to the simulation
 
 ## [0.20.7]
 ### Changed

README.md

@@ -1,20 +1,20 @@
 # [SOIL](https://github.com/gsi-upm/soil)
 
+
 Soil is an extensible and user-friendly Agent-based Social Simulator for Social Networks.
 Learn how to run your own simulations with our [documentation](http://soilsim.readthedocs.io).
 
-Follow our [tutorial](examples/tutorial/soil_tutorial.ipynb) to develop your own agent models.
+Follow our [tutorial](docs/tutorial/soil_tutorial.ipynb) to develop your own agent models.
 
-**Note**: Mesa 0.30 introduced many fundamental changes. Check the [documention on how to update your simulations to work with newer versions](docs/migration_0.30.rst)
+> **Warning**
+> Soil 1.0 introduced many fundamental changes. Check the [documention on how to update your simulations to work with newer versions](docs/notes_v1.0.rst)
 
-## SOIL vs MESA
+## Features
 
-SOIL is a batteries-included platform that builds on top of MESA and provides the following out of the box:
-
-* Integration with (social) networks
-* The ability to more easily assign agents to your model (and optionally to its network):
-  * Assigning agents to nodes, and vice versa
-  * Using a description (e.g., 2 agents of type `Foo`, 10% of the network should be agents of type `Bar`)
+* Integration with (social) networks (through `networkx`)
+* Convenience functions and methods to easily assign agents to your model (and optionally to its network):
+  * Following a given distribution (e.g., 2 agents of type `Foo`, 10% of the network should be agents of type `Bar`)
+  * Based on the topology of the network
 * **Several types of abstractions for agents**:
   * Finite state machine, where methods can be turned into a state
   * Network agents, which have convenience methods to access the model's topology
@@ -33,15 +33,17 @@ SOIL is a batteries-included platform that builds on top of MESA and provides th
   * Run models in parallel
   * Save results to different formats
 * Simulation configuration files 
-* A command line interface (`soil`), to run multiple
+* A command line interface (`soil`), to quickly run simulations with different parameters
 * An integrated debugger (`soil --debug`) with custom functions to print agent states and break at specific states
 
-Nevertheless, most features in SOIL have been designed to integrate with plain Mesa.
+## Mesa compatibility
+
+SOIL has been redesigned to integrate well with [Mesa](https://github.com/projectmesa/mesa).
 For instance, it should be possible to run a `mesa.Model` models using a `soil.Simulation` and the `soil` CLI, or to integrate the `soil.TimedActivation` scheduler on a `mesa.Model`.
 
-Note that some combinations of `mesa` and `soil` components, while technically possible, are much less useful or even wrong.
-For instance, you may add any `soil.agent` agent (except for the `soil.NetworkAgent`, as it needs a topology) on a regular `mesa.Model` with a vanilla scheduler from `mesa.time`.
-But in that case the agents will not get any of the advanced event-based scheduling, and most agent behaviors that depend on that will greatly vary. 
+Note that some combinations of `mesa` and `soil` components, while technically possible, are much less useful or might yield surprising results.
+For instance, you may add any `soil.agent` agent on a regular `mesa.Model` with a vanilla scheduler from `mesa.time`.
+But in that case the agents will not get any of the advanced event-based scheduling, and most agent behaviors that depend on that may not work. 
 
 
 ## Changes in version 0.3

benchmarks/noop-bench-async.csv

@@ -0,0 +1,12 @@
+command,mean,stddev,median,user,system,min,max,parameter_sim
+python noop/mesa_batchrunner.py,1.3258325165599998,0.05822826666377271,1.31279976286,1.2978164199999997,0.25767558,1.2780627573599999,1.46763559736,mesa_batchrunner
+python noop/mesa_simulation.py,1.3915081544599999,0.07311646048704976,1.37166811936,1.35267662,0.29222067999999995,1.32746067836,1.58495303336,mesa_simulation
+python noop/soil_step.py,1.9859962588599998,0.12143759641749913,1.93586195486,2.0000750199999997,0.54126188,1.9061700903599998,2.2532835533599997,soil_step
+python noop/soil_step_pqueue.py,2.1347049971600005,0.01336179424666973,2.13492341986,2.1368160200000004,0.56862948,2.11810132936,2.16042739636,soil_step_pqueue
+python noop/soil_gens.py,2.1284937893599998,0.03030587681163665,2.13585231586,2.14158812,0.54900038,2.0768625143599997,2.19043625236,soil_gens
+python noop/soil_gens_pqueue.py,2.3469003942599995,0.019461346004472344,2.3486906343599996,2.36505852,0.54629858,2.31766326036,2.37998102136,soil_gens_pqueue
+python noop/soil_async.py,2.85755484126,0.0314955571121844,2.84774029536,2.86388112,0.55261338,2.81428668936,2.90567961636,soil_async
+python noop/soil_async_pqueue.py,3.1999731134600005,0.04432336803797717,3.20255954186,3.2162337199999995,0.5501872800000001,3.1406816913599997,3.26137401936,soil_async_pqueue
+python noop/soilent_step.py,1.30038977816,0.017973958957989845,1.30187804986,1.3231730199999998,0.5452653799999999,1.27058263436,1.31902240836,soilent_step
+python noop/soilent_step_pqueue.py,1.4708435788599998,0.027193290392962755,1.4707784423599999,1.4900387199999998,0.54749428,1.43498127536,1.53065598436,soilent_step_pqueue
+python noop/soilent_gens.py,1.6338810973599998,0.05752539125688073,1.63513330036,1.65216122,0.51846678,1.54135944036,1.7038832853599999,soilent_gens

benchmarks/noop-bench.csv

@@ -0,0 +1,11 @@
+command,mean,stddev,median,user,system,min,max,parameter_sim
+python noop/mesa1_batchrunner.py,1.2559917394000002,0.012031173494887278,1.2572688413000002,1.2168630799999998,0.31825289999999995,1.2346063853,1.2735512493,mesa1_batchrunner
+python noop/mesa1_simulation.py,1.3024417227,0.022498874113931668,1.2994157323,1.2595484799999999,0.3087897,1.2697029703,1.3350640403,mesa1_simulation
+python noop/soil1.py,1.8789492443,0.18023367899835044,1.8186795393000001,1.86076288,0.5309521,1.7326687413000001,2.2928370642999996,soil1
+python noop/soil1_pqueue.py,1.9841675890000001,0.01735524088843906,1.9884363323,2.01830338,0.5787977999999999,1.9592171483,2.0076169282999996,soil1_pqueue
+python noop/soil2.py,2.0135188921999996,0.02869307129649681,2.0184709453,2.03951308,0.5885591,1.9680417823,2.0567112592999997,soil2
+python noop/soil2_pqueue.py,2.2367320454999997,0.024339667344486046,2.2357249777999995,2.2515216799999997,0.5978869,2.1957917303,2.2688685033,soil2_pqueue
+python noop/soilent1.py,1.1309301329,0.015133005948737871,1.1276461497999999,1.14056688,0.6027519,1.1135821423,1.1625753893,soilent1
+python noop/soilent1_pqueue.py,1.3097537665000003,0.018821977712258842,1.3073709358,1.3270259799999997,0.6000067999999998,1.2874580013,1.3381646823,soilent1_pqueue
+python noop/soilent2.py,1.5055360476,0.05166674417574119,1.4883118568,1.5121205799999997,0.5817363999999999,1.4490918363,1.6005909333000001,soilent2
+python noop/soilent2_pqueue.py,1.6622598218,0.031130739036296016,1.6588702603,1.6862567799999997,0.5854159,1.6289724583,1.7330545383,soilent2_pqueue

benchmarks/noop/_config.py

@@ -0,0 +1,25 @@
+import os
+
+NUM_AGENTS = int(os.environ.get('NUM_AGENTS', 100))
+NUM_ITERS = int(os.environ.get('NUM_ITERS', 10))
+MAX_STEPS = int(os.environ.get('MAX_STEPS', 1000))
+
+
+def run_sim(model, **kwargs):
+    from soil import Simulation
+    opts = dict(model=model,
+                dump=False,
+                num_processes=1,
+                parameters={'num_agents': NUM_AGENTS},
+                max_steps=MAX_STEPS,
+                iterations=NUM_ITERS)
+    opts.update(kwargs)
+    res = Simulation(**opts).run()
+
+    total = sum(a.num_calls for e in res for a in e.schedule.agents)
+    expected = NUM_AGENTS * NUM_ITERS * MAX_STEPS
+    print(total)
+    print(expected)
+
+    assert total == expected
+    return res

benchmarks/noop/mesa_batchrunner.py

@@ -0,0 +1,44 @@
+from mesa import batch_run, DataCollector, Agent, Model
+from mesa.time import RandomActivation
+
+
+class NoopAgent(Agent):
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.num_calls = 0
+
+    def step(self):
+        # import pdb;pdb.set_trace()
+        self.num_calls += 1 
+
+
+class NoopModel(Model):
+    def __init__(self, N):
+        super().__init__()
+        self.schedule = RandomActivation(self)
+        for i in range(N):
+            self.schedule.add(NoopAgent(self.next_id(), self))
+        self.datacollector = DataCollector(model_reporters={"num_agents": lambda m: m.schedule.get_agent_count(),
+                                                            "time": lambda m: m.schedule.time},
+                                           agent_reporters={"num_calls": "num_calls"})
+        self.datacollector.collect(self)
+
+    def step(self):
+        self.schedule.step()
+        self.datacollector.collect(self)
+
+
+if __name__ == "__main__":
+    from _config import *
+
+    res = batch_run(model_cls=NoopModel,
+                    max_steps=MAX_STEPS,
+                    iterations=NUM_ITERS,
+                    number_processes=1,
+                    parameters={'N': NUM_AGENTS})
+    total = sum(s["num_calls"] for s in res)
+    total_agents = sum(s["num_agents"] for s in res)
+    assert len(res) == NUM_AGENTS * NUM_ITERS
+    assert total == NUM_AGENTS * NUM_ITERS * MAX_STEPS
+    assert total_agents == NUM_AGENTS * NUM_AGENTS * NUM_ITERS
+

benchmarks/noop/mesa_simulation.py

@@ -0,0 +1,38 @@
+from mesa import batch_run, DataCollector, Agent, Model
+from mesa.time import RandomActivation
+from soil import Simulation
+from _config import *
+
+
+class NoopAgent(Agent):
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.num_calls = 0
+
+    def step(self):
+        # import pdb;pdb.set_trace()
+        self.num_calls += 1 
+
+
+class NoopModel(Model):
+    def __init__(self, num_agents, *args, **kwargs):
+        super().__init__()
+        self.schedule = RandomActivation(self)
+        for i in range(num_agents):
+            self.schedule.add(NoopAgent(self.next_id(), self))
+        self.datacollector = DataCollector(model_reporters={"num_agents": lambda m: m.schedule.get_agent_count(),
+                                                            "time": lambda m: m.schedule.time},
+                                           agent_reporters={"num_calls": "num_calls"})
+        self.datacollector.collect(self)
+
+    def step(self):
+        self.schedule.step()
+        self.datacollector.collect(self)
+
+
+def run():
+    run_sim(model=NoopModel)
+
+
+if __name__ == "__main__":
+    run()

benchmarks/noop/noop-bench.csv

@@ -0,0 +1,3 @@
+command,mean,stddev,median,user,system,min,max,parameter_sim
+python mesa1_batchrunner.py,1.2932078178200002,0.05649377020829272,1.2705532802200001,1.25902256,0.27242284,1.22210926572,1.40867459172,mesa1_batchrunner
+python mesa1_simulation.py,1.81112963812,0.015491072368938567,1.81342524572,1.8594407599999996,0.8005329399999999,1.78538603972,1.84176361172,mesa1_simulation

benchmarks/noop/soil_async.py

@@ -0,0 +1,24 @@
+from soil import BaseAgent, Environment, Simulation
+
+
+class NoopAgent(BaseAgent):
+    num_calls = 0
+
+    async def step(self):
+        while True:
+            self.num_calls += 1
+            await self.delay()
+
+
+class NoopEnvironment(Environment):
+    num_agents = 100
+
+    def init(self):
+        self.add_agents(NoopAgent, k=self.num_agents)
+        self.add_agent_reporter("num_calls")
+
+
+if __name__ == "__main__":
+    from _config import *
+
+    run_sim(model=NoopEnvironment)

benchmarks/noop/soil_async_pqueue.py

@@ -0,0 +1,25 @@
+from soil import BaseAgent, Environment, Simulation, PQueueActivation
+
+
+class NoopAgent(BaseAgent):
+    num_calls = 0
+
+    async def step(self):
+        while True:
+            self.num_calls += 1
+            await self.delay()
+
+
+class NoopEnvironment(Environment):
+    num_agents = 100
+    schedule_class = PQueueActivation
+
+    def init(self):
+        self.add_agents(NoopAgent, k=self.num_agents)
+        self.add_agent_reporter("num_calls")
+
+
+if __name__ == "__main__":
+    from _config import *
+
+    run_sim(model=NoopEnvironment)

benchmarks/noop/soil_gens.py

@@ -0,0 +1,24 @@
+from soil import BaseAgent, Environment, Simulation
+
+
+class NoopAgent(BaseAgent):
+    num_calls = 0
+
+    def step(self):
+        while True:
+            self.num_calls += 1
+            yield self.delay()
+
+
+class NoopEnvironment(Environment):
+    num_agents = 100
+
+    def init(self):
+        self.add_agents(NoopAgent, k=self.num_agents)
+        self.add_agent_reporter("num_calls")
+
+
+if __name__ == "__main__":
+    from _config import *
+
+    run_sim(model=NoopEnvironment)

benchmarks/noop/soil_gens_pqueue.py

@@ -0,0 +1,25 @@
+from soil import BaseAgent, Environment, Simulation, PQueueActivation
+
+
+class NoopAgent(BaseAgent):
+    num_calls = 0
+
+    def step(self):
+        while True:
+            self.num_calls += 1
+            yield self.delay()
+
+
+class NoopEnvironment(Environment):
+    num_agents = 100
+    schedule_class = PQueueActivation
+
+    def init(self):
+        self.add_agents(NoopAgent, k=self.num_agents)
+        self.add_agent_reporter("num_calls")
+
+
+if __name__ == "__main__":
+    from _config import *
+
+    run_sim(model=NoopEnvironment)

benchmarks/noop/soil_step.py

@@ -0,0 +1,21 @@
+from soil import BaseAgent, Environment, Simulation
+
+
+class NoopAgent(BaseAgent):
+    num_calls = 0
+
+    def step(self):
+        self.num_calls += 1
+
+class NoopEnvironment(Environment):
+    num_agents = 100
+
+    def init(self):
+        self.add_agents(NoopAgent, k=self.num_agents)
+        self.add_agent_reporter("num_calls")
+
+
+if __name__ == "__main__":
+    from _config import *
+
+    run_sim(model=NoopEnvironment)

benchmarks/noop/soil_step_pqueue.py

@@ -0,0 +1,22 @@
+from soil import BaseAgent, Environment, Simulation, PQueueActivation
+
+
+class NoopAgent(BaseAgent):
+    num_calls = 0
+
+    def step(self):
+        self.num_calls += 1
+
+class NoopEnvironment(Environment):
+    num_agents = 100
+    schedule_class = PQueueActivation
+
+    def init(self):
+        self.add_agents(NoopAgent, k=self.num_agents)
+        self.add_agent_reporter("num_calls")
+
+
+if __name__ == "__main__":
+    from _config import *
+
+    run_sim(model=NoopEnvironment)

benchmarks/noop/soilent_async.py

@@ -0,0 +1,29 @@
+from soil import Agent, Environment, Simulation
+from soilent import Scheduler
+
+
+class NoopAgent(Agent):
+    num_calls = 0
+
+    async def step(self):
+        while True:
+            self.num_calls += 1
+            # yield self.delay(1)
+            await self.delay()
+
+
+class NoopEnvironment(Environment):
+    num_agents = 100
+    schedule_class = Scheduler
+
+    def init(self):
+        self.add_agents(NoopAgent, k=self.num_agents)
+        self.add_agent_reporter("num_calls")
+
+
+if __name__ == "__main__":
+    from _config import *
+
+    res = run_sim(model=NoopEnvironment)
+    for r in res:
+        assert isinstance(r.schedule, Scheduler)

benchmarks/noop/soilent_async_pqueue.py

@@ -0,0 +1,27 @@
+from soil import Agent, Environment
+from soilent import PQueueScheduler
+
+
+class NoopAgent(Agent):
+    num_calls = 0
+
+    async def step(self):
+        while True:
+            self.num_calls += 1
+            await self.delay()
+
+class NoopEnvironment(Environment):
+    num_agents = 100
+    schedule_class = PQueueScheduler
+
+    def init(self):
+        self.add_agents(NoopAgent, k=self.num_agents)
+        self.add_agent_reporter("num_calls")
+
+
+if __name__ == "__main__":
+    from _config import *
+
+    res = run_sim(model=NoopEnvironment)
+    for r in res:
+        assert isinstance(r.schedule, PQueueScheduler)

benchmarks/noop/soilent_gens.py

@@ -0,0 +1,28 @@
+from soil import Agent, Environment, Simulation
+from soilent import Scheduler
+
+
+class NoopAgent(Agent):
+    num_calls = 0
+
+    def step(self):
+        while True:
+            self.num_calls += 1
+            # yield self.delay(1)
+            yield self.delay()
+
+class NoopEnvironment(Environment):
+    num_agents = 100
+    schedule_class = Scheduler
+
+    def init(self):
+        self.add_agents(NoopAgent, k=self.num_agents)
+        self.add_agent_reporter("num_calls")
+
+
+if __name__ == "__main__":
+    from _config import *
+
+    res = run_sim(model=NoopEnvironment)
+    for r in res:
+        assert isinstance(r.schedule, Scheduler)

benchmarks/noop/soilent_gens_pqueue.py

@@ -0,0 +1,28 @@
+from soil import Agent, Environment
+from soilent import PQueueScheduler
+
+
+class NoopAgent(Agent):
+    num_calls = 0
+
+    def step(self):
+        while True:
+            self.num_calls += 1
+            # yield self.delay(1)
+            yield
+
+class NoopEnvironment(Environment):
+    num_agents = 100
+    schedule_class = PQueueScheduler
+
+    def init(self):
+        self.add_agents(NoopAgent, k=self.num_agents)
+        self.add_agent_reporter("num_calls")
+
+
+if __name__ == "__main__":
+    from _config import *
+
+    res = run_sim(model=NoopEnvironment)
+    for r in res:
+        assert isinstance(r.schedule, PQueueScheduler)

benchmarks/noop/soilent_step.py

@@ -0,0 +1,24 @@
+from soil import BaseAgent, Environment, Simulation
+from soilent import Scheduler
+
+
+class NoopAgent(BaseAgent):
+    num_calls = 0
+
+    def step(self):
+        self.num_calls += 1
+
+class NoopEnvironment(Environment):
+    num_agents = 100
+    schedule_class = Scheduler
+
+    def init(self):
+        self.add_agents(NoopAgent, k=self.num_agents)
+        self.add_agent_reporter("num_calls")
+
+
+if __name__ == "__main__":
+    from _config import *
+    res = run_sim(model=NoopEnvironment)
+    for r in res:
+        assert isinstance(r.schedule, Scheduler)

benchmarks/noop/soilent_step_pqueue.py

@@ -0,0 +1,24 @@
+from soil import BaseAgent, Environment, Simulation
+from soilent import PQueueScheduler
+
+
+class NoopAgent(BaseAgent):
+    num_calls = 0
+
+    def step(self):
+        self.num_calls += 1
+
+class NoopEnvironment(Environment):
+    num_agents = 100
+    schedule_class = PQueueScheduler
+
+    def init(self):
+        self.add_agents(NoopAgent, k=self.num_agents)
+        self.add_agent_reporter("num_calls")
+
+
+if __name__ == "__main__":
+    from _config import *
+    res = run_sim(model=NoopEnvironment)
+    for r in res:
+        assert isinstance(r.schedule, PQueueScheduler)

benchmarks/run.py

@@ -0,0 +1,19 @@
+#!/bin/env python
+import sys
+import os
+import subprocess
+import argparse
+parser = argparse.ArgumentParser(
+                    prog='Profiler for soil')
+parser.add_argument('--suffix', default=None)
+parser.add_argument('files', nargs="+")
+
+args = parser.parse_args()
+
+for fname in args.files:
+    suffix = ("_" + args.suffix) if args.suffix else ""
+    simname = f"{fname.replace('/', '-')}{suffix}"
+    profpath = os.path.join("profs", simname + ".prof")
+
+    print(f"Running {fname} and saving profile to {profpath}")
+    subprocess.call(["python", "-m", "cProfile", "-o", profpath, fname])

benchmarks/virusonnetwork.csv

@@ -0,0 +1,4 @@
+command,mean,stddev,median,user,system,min,max,parameter_sim
+python virusonnetwork/mesa_basic.py,3.8381473157,0.0518143371442526,3.8475315791,3.873109219999999,0.55102658,3.7523016936,3.9095182436,mesa_basic.py
+python virusonnetwork/soil_step.py,3.2167258977000004,0.02337131987357665,3.2257620261,3.28374132,0.51343958,3.1792271306,3.2511521286000002,soil_step.py
+python virusonnetwork/soil_states.py,3.4908183217,0.03726734070349347,3.4912775086,3.5684004200000006,0.50416068,3.4272087936,3.5529207346000002,soil_states.py

benchmarks/virusonnetwork/_config.py

@@ -0,0 +1,32 @@
+import os
+
+NUM_AGENTS = int(os.environ.get('NUM_AGENTS', 100))
+NUM_ITERS = int(os.environ.get('NUM_ITERS', 10))
+MAX_STEPS = int(os.environ.get('MAX_STEPS', 1000))
+
+
+def run_sim(model, **kwargs):
+    from soil import Simulation
+    opts = dict(model=model,
+                dump=False,
+                num_processes=1,
+                parameters={'num_nodes': NUM_AGENTS,
+                            "avg_node_degree": 3,
+                            "initial_outbreak_size": 5,
+                            "virus_spread_chance": 0.25,
+                            "virus_check_frequency": 0.25,
+                            "recovery_chance": 0.3,
+                            "gain_resistance_chance": 0.1,
+                            },
+                max_steps=MAX_STEPS,
+                iterations=NUM_ITERS)
+    opts.update(kwargs)
+    its = Simulation(**opts).run()
+
+    assert all(it.schedule.steps == MAX_STEPS for it in its)
+    ratios = list(it.resistant_susceptible_ratio() for it in its)
+    print("Max - Avg - Min ratio:", max(ratios), sum(ratios)/len(ratios), min(ratios))
+    assert all(sum([it.number_susceptible,
+                    it.number_infected,
+                    it.number_resistant]) == NUM_AGENTS for it in its)
+    return its

benchmarks/virusonnetwork/mesa_basic.py

@@ -0,0 +1,180 @@
+# Verbatim copy from mesa
+# https://github.com/projectmesa/mesa/blob/976ddfc8a1e5feaaf8007a7abaa9abc7093881a0/examples/virus_on_network/virus_on_network/model.py
+import math
+from enum import Enum
+import networkx as nx
+
+import mesa
+
+
+class State(Enum):
+    SUSCEPTIBLE = 0
+    INFECTED = 1
+    RESISTANT = 2
+
+
+def number_state(model, state):
+    return sum(1 for a in model.grid.get_all_cell_contents() if a.state is state)
+
+
+def number_infected(model):
+    return number_state(model, State.INFECTED)
+
+
+def number_susceptible(model):
+    return number_state(model, State.SUSCEPTIBLE)
+
+
+def number_resistant(model):
+    return number_state(model, State.RESISTANT)
+
+
+class VirusOnNetwork(mesa.Model):
+    """A virus model with some number of agents"""
+
+    def __init__(
+        self,
+        *args,
+        num_nodes=10,
+        avg_node_degree=3,
+        initial_outbreak_size=1,
+        virus_spread_chance=0.4,
+        virus_check_frequency=0.4,
+        recovery_chance=0.3,
+        gain_resistance_chance=0.5,
+        **kwargs,
+    ):
+
+        self.num_nodes = num_nodes
+        prob = avg_node_degree / self.num_nodes
+        self.G = nx.erdos_renyi_graph(n=self.num_nodes, p=prob)
+        self.grid = mesa.space.NetworkGrid(self.G)
+        self.schedule = mesa.time.RandomActivation(self)
+        self.initial_outbreak_size = (
+            initial_outbreak_size if initial_outbreak_size <= num_nodes else num_nodes
+        )
+        self.virus_spread_chance = virus_spread_chance
+        self.virus_check_frequency = virus_check_frequency
+        self.recovery_chance = recovery_chance
+        self.gain_resistance_chance = gain_resistance_chance
+
+        self.datacollector = mesa.DataCollector(
+            {
+                "Ratio": "resistant_susceptible_ratio",
+                "Infected": number_infected,
+                "Susceptible": number_susceptible,
+                "Resistant": number_resistant,
+            }
+        )
+
+        # Create agents
+        for i, node in enumerate(self.G.nodes()):
+            a = VirusAgent(
+                i,
+                self,
+                State.SUSCEPTIBLE,
+                self.virus_spread_chance,
+                self.virus_check_frequency,
+                self.recovery_chance,
+                self.gain_resistance_chance,
+            )
+            self.schedule.add(a)
+            # Add the agent to the node
+            self.grid.place_agent(a, node)
+
+        # Infect some nodes
+        infected_nodes = self.random.sample(list(self.G), self.initial_outbreak_size)
+        for a in self.grid.get_cell_list_contents(infected_nodes):
+            a.state = State.INFECTED
+
+        self.running = True
+        self.datacollector.collect(self)
+    
+    @property
+    def number_susceptible(self):
+        return number_susceptible(self)
+    @property
+    def number_resistant(self):
+        return number_resistant(self)
+    @property
+    def number_infected(self):
+        return number_infected(self)
+
+    def resistant_susceptible_ratio(self):
+        try:
+            return number_state(self, State.RESISTANT) / number_state(
+                self, State.SUSCEPTIBLE
+            )
+        except ZeroDivisionError:
+            return math.inf
+
+    def step(self):
+        self.schedule.step()
+        # collect data
+        self.datacollector.collect(self)
+
+    def run_model(self, n):
+        for i in range(n):
+            self.step()
+
+
+class VirusAgent(mesa.Agent):
+    def __init__(
+        self,
+        unique_id,
+        model,
+        initial_state,
+        virus_spread_chance,
+        virus_check_frequency,
+        recovery_chance,
+        gain_resistance_chance,
+    ):
+        super().__init__(unique_id, model)
+
+        self.state = initial_state
+
+        self.virus_spread_chance = virus_spread_chance
+        self.virus_check_frequency = virus_check_frequency
+        self.recovery_chance = recovery_chance
+        self.gain_resistance_chance = gain_resistance_chance
+
+    def try_to_infect_neighbors(self):
+        neighbors_nodes = self.model.grid.get_neighbors(self.pos, include_center=False)
+        susceptible_neighbors = [
+            agent
+            for agent in self.model.grid.get_cell_list_contents(neighbors_nodes)
+            if agent.state is State.SUSCEPTIBLE
+        ]
+        for a in susceptible_neighbors:
+            if self.random.random() < self.virus_spread_chance:
+                a.state = State.INFECTED
+
+    def try_gain_resistance(self):
+        if self.random.random() < self.gain_resistance_chance:
+            self.state = State.RESISTANT
+
+    def try_remove_infection(self):
+        # Try to remove
+        if self.random.random() < self.recovery_chance:
+            # Success
+            self.state = State.SUSCEPTIBLE
+            self.try_gain_resistance()
+        else:
+            # Failed
+            self.state = State.INFECTED
+
+    def try_check_situation(self):
+        if self.random.random() < self.virus_check_frequency:
+            # Checking...
+            if self.state is State.INFECTED:
+                self.try_remove_infection()
+
+    def step(self):
+        if self.state is State.INFECTED:
+            self.try_to_infect_neighbors()
+        self.try_check_situation()
+
+
+from _config import run_sim
+
+run_sim(model=VirusOnNetwork)

benchmarks/virusonnetwork/soil_states.py

@@ -0,0 +1,92 @@
+# Verbatim copy from mesa
+# https://github.com/projectmesa/mesa/blob/976ddfc8a1e5feaaf8007a7abaa9abc7093881a0/examples/virus_on_network/virus_on_network/model.py
+import math
+from enum import Enum
+import networkx as nx
+
+from soil import *
+
+
+class VirusOnNetwork(Environment):
+    """A virus model with some number of agents"""
+    num_nodes = 10
+    avg_node_degree = 3
+    initial_outbreak_size = 1
+    virus_spread_chance = 0.4
+    virus_check_frequency = 0.4
+    recovery_chance = 0
+    gain_resistance_chance = 0
+
+    def init(self):
+        prob = self.avg_node_degree / self.num_nodes
+        # Use internal seed with the networkx generator
+        self.create_network(generator=nx.erdos_renyi_graph, n=self.num_nodes, p=prob)
+
+        self.initial_outbreak_size = min(self.initial_outbreak_size, self.num_nodes)
+        self.populate_network(VirusAgent)
+
+        # Infect some nodes
+        infected_nodes = self.random.sample(list(self.G), self.initial_outbreak_size)
+        for a in self.get_agents(node_id=infected_nodes):
+            a.set_state(VirusAgent.infected)
+        assert self.number_infected == self.initial_outbreak_size
+
+    @report
+    def resistant_susceptible_ratio(self):
+        try:
+            return self.number_resistant / self.number_susceptible
+        except ZeroDivisionError:
+            return math.inf
+
+    @report
+    @property
+    def number_infected(self):
+        return self.count_agents(state_id=VirusAgent.infected.id)
+
+    @report
+    @property
+    def number_susceptible(self):
+        return self.count_agents(state_id=VirusAgent.susceptible.id)
+
+    @report
+    @property
+    def number_resistant(self):
+        return self.count_agents(state_id=VirusAgent.resistant.id)
+
+
+class VirusAgent(Agent):
+    virus_spread_chance = None # Inherit from model
+    virus_check_frequency = None # Inherit from model
+    recovery_chance = None # Inherit from model
+    gain_resistance_chance = None # Inherit from model
+    just_been_infected = False
+
+    @state(default=True)
+    def susceptible(self):
+        if self.just_been_infected:
+            self.just_been_infected = False
+            return self.infected
+
+    @state
+    def infected(self):
+        susceptible_neighbors = self.get_neighbors(state_id=self.susceptible.id)
+        for a in susceptible_neighbors:
+            if self.prob(self.virus_spread_chance):
+                a.just_been_infected = True
+        if self.prob(self.virus_check_frequency):
+            if self.prob(self.recovery_chance):
+                if self.prob(self.gain_resistance_chance):
+                    return self.resistant
+                else:
+                    return self.susceptible
+            else:
+                return self.infected
+
+    @state
+    def resistant(self):
+        return self.at(INFINITY)
+
+
+if __name__ == "__main__":
+    from _config import run_sim
+    run_sim(model=VirusOnNetwork)

benchmarks/virusonnetwork/soil_step.py

@@ -0,0 +1,104 @@
+# Verbatim copy from mesa
+# https://github.com/projectmesa/mesa/blob/976ddfc8a1e5feaaf8007a7abaa9abc7093881a0/examples/virus_on_network/virus_on_network/model.py
+import math
+from enum import Enum
+import networkx as nx
+
+from soil import *
+
+
+class State(Enum):
+    SUSCEPTIBLE = 0
+    INFECTED = 1
+    RESISTANT = 2
+
+
+class VirusOnNetwork(Environment):
+    """A virus model with some number of agents"""
+    num_nodes = 10
+    avg_node_degree = 3
+    initial_outbreak_size = 1
+    virus_spread_chance = 0.4
+    virus_check_frequency = 0.4
+    recovery_chance = 0
+    gain_resistance_chance = 0
+
+    def init(self):
+        prob = self.avg_node_degree / self.num_nodes
+        # Use internal seed with the networkx generator
+        self.create_network(generator=nx.erdos_renyi_graph, n=self.num_nodes, p=prob)
+
+        self.initial_outbreak_size = min(self.initial_outbreak_size, self.num_nodes)
+        self.populate_network(VirusAgent)
+
+        # Infect some nodes
+        infected_nodes = self.random.sample(list(self.G), self.initial_outbreak_size)
+        for a in self.get_agents(node_id=infected_nodes):
+            a.status = State.INFECTED
+        assert self.number_infected == self.initial_outbreak_size
+
+    @report
+    def resistant_susceptible_ratio(self):
+        try:
+            return self.number_resistant / self.number_susceptible
+        except ZeroDivisionError:
+            return math.inf
+
+    @report
+    @property
+    def number_infected(self):
+        return self.count_agents(status=State.INFECTED)
+
+    @report
+    @property
+    def number_susceptible(self):
+        return self.count_agents(status=State.SUSCEPTIBLE)
+
+    @report
+    @property
+    def number_resistant(self):
+        return self.count_agents(status=State.RESISTANT)
+
+
+
+class VirusAgent(Agent):
+    status = State.SUSCEPTIBLE
+    virus_spread_chance = None # Inherit from model
+    virus_check_frequency = None # Inherit from model
+    recovery_chance = None # Inherit from model
+    gain_resistance_chance = None # Inherit from model
+
+    def try_to_infect_neighbors(self):
+        susceptible_neighbors = self.get_neighbors(status=State.SUSCEPTIBLE)
+        for a in susceptible_neighbors:
+            if self.prob(self.virus_spread_chance):
+                a.status = State.INFECTED
+
+    def try_gain_resistance(self):
+        if self.prob(self.gain_resistance_chance):
+            self.status = State.RESISTANT
+            return self.at(INFINITY)
+
+    def try_remove_infection(self):
+        # Try to remove
+        if self.prob(self.recovery_chance):
+            # Success
+            self.status = State.SUSCEPTIBLE
+            return self.try_gain_resistance()
+
+    def try_check_situation(self):
+        if self.prob(self.virus_check_frequency):
+            # Checking...
+            if self.status is State.INFECTED:
+                return self.try_remove_infection()
+
+    def step(self):
+        if self.status is State.INFECTED:
+            self.try_to_infect_neighbors()
+        return self.try_check_situation()
+
+
+
+if __name__ == "__main__":
+    from _config import run_sim
+    run_sim(model=VirusOnNetwork)

docs/conf.py

@@ -31,7 +31,10 @@
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = ['IPython.sphinxext.ipython_console_highlighting']
+extensions = [
+    "IPython.sphinxext.ipython_console_highlighting",
+    "nbsphinx",
+]
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
@@ -64,7 +67,7 @@ release = '0.1'
 #
 # This is also used if you do content translation via gettext catalogs.
 # Usually you set "language" from the command line for these cases.
-language = None
+language = "en"
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
@@ -152,6 +155,3 @@ texinfo_documents = [
      author, 'Soil', 'One line description of project.',
      'Miscellaneous'),
 ]
-
-
-

docs/example.yml

@@ -1,40 +0,0 @@
----
-name: MyExampleSimulation
-max_time: 50
-num_trials: 3
-interval: 2
-model_params:
-  topology:
-    params:
-      generator: barabasi_albert_graph
-      n: 100
-      m: 2
-  agents:
-    distribution:
-    - agent_class: SISaModel
-      topology: True
-      ratio: 0.1
-      state:
-        state_id: content
-    - agent_class: SISaModel
-      topology: True
-      ratio: .1
-      state:
-        state_id: discontent
-    - agent_class: SISaModel
-      topology: True
-      ratio: 0.8
-      state:
-        state_id: neutral
-  prob_infect: 0.075
-  neutral_discontent_spon_prob: 0.1
-  neutral_discontent_infected_prob: 0.3
-  neutral_content_spon_prob: 0.3
-  neutral_content_infected_prob: 0.4
-  discontent_neutral: 0.5
-  discontent_content: 0.5
-  variance_d_c: 0.2
-  content_discontent: 0.2
-  variance_c_d: 0.2
-  content_neutral: 0.2
-  standard_variance: 1

docs/index.rst

@@ -1,7 +1,21 @@
 Welcome to Soil's documentation!
 ================================
 
-Soil is an Agent-based Social Simulator in Python focused on Social Networks.
+Soil is an opinionated Agent-based Social Simulator in Python focused on Social Networks.
+To get started developing your own simulations and agent behaviors, check out our :doc:`Tutorial <tutorial/soil_tutorial>` and the `examples on GitHub <https://github.com/gsi-upm/soil/tree/master/examples>`.
+
+Soil can be installed through pip (see more details in the :doc:`installation` page):.
+
+.. image:: soil.png
+  :width: 80%
+  :align: center
+
+
+.. code:: bash
+
+    pip install soil
+
+
 
 If you use Soil in your research, do not forget to cite this paper:
 
@@ -33,9 +47,9 @@ If you use Soil in your research, do not forget to cite this paper:
    :caption: Learn more about soil:
 
    installation
-   quickstart
-   configuration
-   Tutorial <soil_tutorial>
+   Tutorial <tutorial/soil_tutorial>
+   notes_v1.0
+   soil-vs
 
 ..
 

docs/installation.rst

@@ -1,7 +1,10 @@
 Installation
 ------------
 
-The easiest way to install Soil is through pip, with Python >= 3.4:
+Through pip
+===========
+
+The easiest way to install Soil is through pip, with Python >= 3.8:
 
 .. code:: bash
 
@@ -25,4 +28,38 @@ Or, if you're using using soil programmatically:
    import soil
    print(soil.__version__)
 
-The latest version can be installed through `GitHub <https://github.com/gsi-upm/soil>`_ or `GitLab <https://lab.gsi.upm.es/soil/soil.git>`_.
+
+
+Web UI
+======
+
+Soil also includes a web server that allows you to upload your simulations, change parameters, and visualize the results, including a timeline of the network.
+To make it work, you have to install soil like this:
+
+.. code::
+
+  pip install soil[web]
+
+Once installed, the soil web UI can be run in two ways:
+
+.. code::
+
+  soil-web
+
+  # OR
+
+  python -m soil.web
+
+
+Development
+===========
+
+The latest version can be downloaded from `GitHub <https://github.com/gsi-upm/soil>`_ and installed manually:
+
+.. code:: bash
+
+   git clone https://github.com/gsi-upm/soil
+   cd soil
+   python -m venv .venv
+   source .venv/bin/activate
+   pip install --editable .

docs/mesa.rst

@@ -1,22 +0,0 @@
-Mesa compatibility
-------------------
-
-Soil is in the process of becoming fully compatible with MESA.
-The idea is to provide a set of modular classes and functions that extend the functionality of mesa, whilst staying compatible.
-In the end, it should be possible to add regular mesa agents to a soil simulation, or use a soil agent within a mesa simulation/model.
-
-This is a non-exhaustive list of tasks to achieve compatibility:
-
-- [ ] Integrate `soil.Simulation` with mesa's runners:
-  - [ ] `soil.Simulation` could mimic/become a `mesa.batchrunner`
-- [ ] Integrate `soil.Environment` with `mesa.Model`:
-  - [x] `Soil.Environment` inherits from `mesa.Model`
-  - [x] `Soil.Environment` includes a Mesa-like Scheduler (see the `soil.time` module.
-  - [ ] Allow for `mesa.Model` to be used in a simulation.
-- [ ] Integrate `soil.Agent` with `mesa.Agent`:
-  - [x] Rename agent.id to unique_id?
-  - [x] mesa agents can be used in soil simulations (see `examples/mesa`)
-- [ ] Provide examples
-  - [ ] Using mesa modules in a soil simulation
-  - [ ] Using soil modules in a mesa simulation
-- [ ] Document the new APIs and usage

docs/notes_v1.0.rst

@@ -0,0 +1,38 @@
+Upgrading to Soil 1.0
+---------------------
+
+What are the main changes in version 1.0?
+#########################################
+
+Version 1.0 is a major rewrite of the Soil system, focused on simplifying the API, aligning it with Mesa, and making it easier to use.
+Unfortunately, this comes at the cost of backwards compatibility.
+
+We drew several lessons from the previous version of Soil, and tried to address them in this version.
+Mainly:
+
+- The split between simulation configuration and simulation code was overly complicated for most use cases. As a result, most users ended up reusing configuration.
+- Storing **all** the simulation data in a database is costly and unnecessary for most use cases. For most use cases, only a handful of variables need to be stored. This fits nicely with Mesa's data collection system.
+- The API was too complex, and it was difficult to understand how to use it.
+- Most parts of the API were not aligned with Mesa, which made it difficult to use Mesa's features or to integrate Soil modules with Mesa code, especially for newcomers.
+- Many parts of the API were tightly coupled, which made it difficult to find bugs, test the system and add new features.
+
+The 0.30 rewrite should provide a middle ground between Soil's opinionated approach and Mesa's flexibility.
+The new Soil is less configuration-centric.
+It aims to provide more modular and convenient functions, most of which can be used in vanilla Mesa.
+
+How are agents assigned to nodes in the network
+###############################################
+
+The constructor of the `NetworkAgent` class has two arguments: `node_id` and `topology`.
+If `topology` is not provided, it will default to `self.model.topology`.
+This assignment might err if the model does not have a `topology` attribute, but most Soil environments derive from `NetworkEnvironment`, so they include a topology by default.
+If `node_id` is not provided, a random node will be selected from the topology, until a node with no agent is found.
+Then, the `node_id` of that node is assigned to the agent.
+If no node with no agent is found, a new node is automatically added to the topology.
+
+
+Can Soil environments include more than one network / topology?
+###############################################################
+
+Yes, but each network has to be included manually.
+Somewhere between 0.20 and 0.30 we included the ability to include multiple networks, but it was deemed too complex and was removed.

docs/quickstart.rst

@@ -1,93 +0,0 @@
-Quickstart
-----------
-
-This section shows how to run your first simulation with Soil.
-For installation instructions, see :doc:`installation`.
-
-There are mainly two parts in a simulation: agent classes and simulation configuration.
-An agent class defines how the agent will behave throughout the simulation.
-The configuration includes things such as number of agents to use and their type, network topology to use, etc.
-
-
-.. image:: soil.png
-  :width: 80%
-  :align: center
-
-
-Soil includes several agent classes in the ``soil.agents`` module, and we will use them in this quickstart.
-If you are interested in developing your own agents classes, see :doc:`soil_tutorial`.
-
-Configuration
-=============
-To get you started, we will use this configuration (:download:`download the file <quickstart.yml>` directly):
-
-.. literalinclude:: quickstart.yml
-   :language: yaml
-
-The agent type used, SISa, is a very simple model.
-It only has three states (neutral, content and discontent),
-Its parameters are the probabilities to change from one state to another, either spontaneously or because of contagion from neighboring agents.
-
-Running the simulation
-======================
-
-To see the simulation in action, simply point soil to the configuration, and tell it to store the graph and the history of agent states and environment parameters at every point.
-
-.. code::
-
-    ❯ soil --graph --csv quickstart.yml                                                          [13:35:29]
-    INFO:soil:Using config(s): quickstart
-    INFO:soil:Dumping results to soil_output/quickstart : ['csv', 'gexf']
-    INFO:soil:Starting simulation quickstart at 13:35:30.
-    INFO:soil:Starting Simulation quickstart trial 0 at 13:35:30.
-    INFO:soil:Finished Simulation quickstart trial 0 at 13:35:49 in 19.43677067756653 seconds
-    INFO:soil:Starting Dumping simulation quickstart trial 0 at 13:35:49.
-    INFO:soil:Finished Dumping simulation quickstart trial 0 at 13:35:51 in 1.7733407020568848 seconds
-    INFO:soil:Dumping results to soil_output/quickstart
-    INFO:soil:Finished simulation quickstart at 13:35:51 in 21.29862952232361 seconds
-
-
-The ``CSV`` file should look like this:
-
-.. code::
-
-   agent_id,t_step,key,value
-   env,0,neutral_discontent_spon_prob,0.05
-   env,0,neutral_discontent_infected_prob,0.1
-   env,0,neutral_content_spon_prob,0.2
-   env,0,neutral_content_infected_prob,0.4
-   env,0,discontent_neutral,0.2
-   env,0,discontent_content,0.05
-   env,0,content_discontent,0.05
-   env,0,variance_d_c,0.05
-   env,0,variance_c_d,0.1
-
-Results and visualization
-=========================
-
-The environment variables are marked as ``agent_id`` env.
-Th exported values are only stored when they change.
-To find out how to get every key and value at every point in the simulation, check out the :doc:`soil_tutorial`.
-
-The dynamic graph is exported as a .gexf file which could be visualized with
-`Gephi <https://gephi.org/users/download/>`__.
-Now it is your turn to experiment with the simulation.
-Change some of the parameters, such as the number of agents, the probability of becoming content, or the type of network, and see how the results change.
-
-
-Soil also includes a web server that allows you to upload your simulations, change parameters, and visualize the results, including a timeline of the network.
-To make it work, you have to install soil like this:
-
-.. code::
-
-  pip install soil[web]
-
-Once installed, the soil web UI can be run in two ways:
-
-.. code::
-
-  soil-web
-
-  # OR
-
-  python -m soil.web

docs/quickstart.yml

@@ -1,33 +0,0 @@
----
-name: quickstart
-num_trials: 1
-max_time: 1000
-model_params:
-  agents:
-    - agent_class: SISaModel
-      topology: true
-      state:
-        id: neutral
-      weight: 1
-    - agent_class: SISaModel
-      topology: true
-      state:
-        id: content
-      weight: 2
-  topology:
-    params:
-      n: 100
-      k: 5
-      p: 0.2
-      generator: newman_watts_strogatz_graph
-  neutral_discontent_spon_prob: 0.05
-  neutral_discontent_infected_prob: 0.1
-  neutral_content_spon_prob: 0.2
-  neutral_content_infected_prob: 0.4
-  discontent_neutral: 0.2
-  discontent_content: 0.05
-  content_discontent: 0.05
-  variance_d_c: 0.05
-  variance_c_d: 0.1
-  content_neutral: 0.1
-  standard_variance: 0.1

docs/requirements.txt

@@ -1 +1,2 @@
 ipython>=7.31.1
+nbsphinx==0.9.1

docs/soil-vs.rst

@@ -1,4 +1,8 @@
-### MESA
+Soil vs other ABM frameworks
+============================
+
+MESA
+----
 
 Starting with version 0.3, Soil has been redesigned to complement Mesa, while remaining compatible with it.
 That means that every component in Soil (i.e., Models, Environments, etc.) can be mixed with existing mesa components.
@@ -10,3 +14,42 @@ Here are some reasons to use Soil instead of plain mesa:
 - Functions to automatically populate a topology with an agent distribution (i.e., different ratios of agent class and state)
 - The `soil.Simulation` class allows you to run multiple instances of the same experiment (i.e., multiple trials with the same parameters but a different randomness seed)
 - Reporting functions that aggregate multiple
+
+Mesa compatibility
+~~~~~~~~~~~~~~~~~~
+
+Soil is in the process of becoming fully compatible with MESA.
+The idea is to provide a set of modular classes and functions that extend the functionality of mesa, whilst staying compatible.
+In the end, it should be possible to add regular mesa agents to a soil simulation, or use a soil agent within a mesa simulation/model.
+
+This is a non-exhaustive list of tasks to achieve compatibility:
+
+.. |check| raw:: html
+
+    ☑
+
+.. |uncheck| raw:: html
+
+    ☐
+
+- |check| Integrate `soil.Simulation` with mesa's runners:
+
+  - |check| `soil.Simulation` can replace `mesa.batchrunner`
+
+- |check| Integrate `soil.Environment` with `mesa.Model`:
+
+  - |check| `Soil.Environment` inherits from `mesa.Model`
+  - |check| `Soil.Environment` includes a Mesa-like Scheduler (see the `soil.time` module.
+  - |check| Allow for `mesa.Model` to be used in a simulation.
+
+- |check| Integrate `soil.Agent` with `mesa.Agent`:
+
+  - |check| Rename agent.id to unique_id
+  - |check| mesa agents can be used in soil simulations (see `examples/mesa`)
+
+- |check| Provide examples
+
+  - |check| Using mesa modules in a soil simulation (see `examples/mesa`)
+  - |uncheck| Using soil modules in a mesa simulation (see `examples/mesa`)
+
+- |uncheck| Document the new APIs and usage

examples/README.md

@@ -0,0 +1 @@
+Some of these examples are close to real life simulations, whereas some others are only a demonstration of Soil's capatibilities.

examples/cars/README.md

@@ -2,6 +2,8 @@ This example can be run like with command-line options, like this:
 
 ```bash
 python cars.py --level DEBUG  -e summary --csv
+#or
+soil cars.py -e summary
 ```
 
 This will set the `CSV` (save the agent and model data to a CSV) and `summary` (print the a summary of the data to stdout) exporters, and set the log level to DEBUG.

examples/cars/cars_sim.py

@@ -56,41 +56,25 @@ class City(EventedEnvironment):
     :param int height: Height of the internal grid
     :param int width: Width of the internal grid
     """
+    n_cars = 1
+    n_passengers = 10
+    height = 100
+    width = 100
 
-    def __init__(
-        self,
-        *args,
-        n_cars=1,
-        n_passengers=10,
-        height=100,
-        width=100,
-        agents=None,
-        model_reporters=None,
-        **kwargs,
-    ):
-        self.grid = MultiGrid(width=width, height=height, torus=False)
-        if agents is None:
-            agents = []
-            for i in range(n_cars):
-                agents.append({"agent_class": Driver})
-            for i in range(n_passengers):
-                agents.append({"agent_class": Passenger})
-        model_reporters = model_reporters or {
-            "earnings": "total_earnings",
-            "n_passengers": "number_passengers",
-        }
-        print("REPORTERS", model_reporters)
-        super().__init__(
-            *args, agents=agents, model_reporters=model_reporters, **kwargs
-        )
-        for agent in self.agents:
+    def init(self):
+        self.grid = MultiGrid(width=self.width, height=self.height, torus=False)
+        if not self.get_agents():
+            self.add_agents(Driver, k=self.n_cars)
+            self.add_agents(Passenger, k=self.n_passengers)
+
+        for agent in self.get_agents():
             self.grid.place_agent(agent, (0, 0))
             self.grid.move_to_empty(agent)
         
-    @property
-    def total_earnings(self):
-        return sum(d.earnings for d in self.agents(agent_class=Driver))
+        self.total_earnings = 0
+        self.add_model_reporter("total_earnings")
 
+    @report
     @property
     def number_passengers(self):
         return self.count_agents(agent_class=Passenger)
@@ -102,7 +86,7 @@ class Driver(Evented, FSM):
     earnings = 0
 
     def on_receive(self, msg, sender):
-        """This is not a state. It will run (and block) every time check_messages is invoked"""
+        """This is not a state. It will run (and block) every time process_messages is invoked"""
         if self.journey is None and isinstance(msg, Journey) and msg.driver is None:
             msg.driver = self
             self.journey = msg
@@ -110,16 +94,15 @@ class Driver(Evented, FSM):
     def check_passengers(self):
         """If there are no more passengers, stop forever"""
         c = self.count_agents(agent_class=Passenger)
-        self.info(f"Passengers left {c}")
-        if not c:
-            self.die()
+        self.debug(f"Passengers left {c}")
+        return c
 
-    @default_state
-    @state
-    def wandering(self):
+    @state(default=True)
+    async def wandering(self):
         """Move around the city until a journey is accepted"""
         target = None
-        self.check_passengers()
+        if not self.check_passengers():
+            return self.die("No passengers left")
         self.journey = None
         while self.journey is None:  # No potential journeys detected (see on_receive)
             if target is None or not self.move_towards(target):
@@ -127,14 +110,15 @@ class Driver(Evented, FSM):
                     self.model.grid.get_neighborhood(self.pos, moore=False)
                 )
 
-            self.check_passengers()
+            if not self.check_passengers():
+                return self.die("No passengers left")
             # This will call on_receive behind the scenes, and the agent's status will be updated
-            self.check_messages()
-            yield Delta(30)  # Wait at least 30 seconds before checking again
+            self.process_messages()
+            await self.delay(30)  # Wait at least 30 seconds before checking again
 
         try:
             # Re-send the journey to the passenger, to confirm that we have been selected
-            self.journey = yield self.journey.passenger.ask(self.journey, timeout=60)
+            self.journey = await self.journey.passenger.ask(self.journey, timeout=60, delay=5)
         except events.TimedOut:
             # No journey has been accepted. Try again
             self.journey = None
@@ -143,19 +127,24 @@ class Driver(Evented, FSM):
         return self.driving
 
     @state
-    def driving(self):
+    async def driving(self):
         """The journey has been accepted. Pick them up and take them to their destination"""
+        self.info(f"Driving towards Passenger {self.journey.passenger.unique_id}")
         while self.move_towards(self.journey.origin):
-            yield
+            await self.delay()
+        self.info(f"Driving {self.journey.passenger.unique_id} from {self.journey.origin} to {self.journey.destination}")
         while self.move_towards(self.journey.destination, with_passenger=True):
-            yield
+            await self.delay()
+        self.info("Arrived at destination")
         self.earnings += self.journey.tip
-        self.check_passengers()
+        self.model.total_earnings += self.journey.tip
+        if not self.check_passengers():
+            return self.die("No passengers left")
         return self.wandering
 
     def move_towards(self, target, with_passenger=False):
         """Move one cell at a time towards a target"""
-        self.info(f"Moving { self.pos } -> { target }")
+        self.debug(f"Moving { self.pos } -> { target }")
         if target[0] == self.pos[0] and target[1] == self.pos[1]:
             return False
 
@@ -179,7 +168,7 @@ class Passenger(Evented, FSM):
     pos = None
 
     def on_receive(self, msg, sender):
-        """This is not a state. It will be run synchronously every time `check_messages` is run"""
+        """This is not a state. It will be run synchronously every time `process_messages` is run"""
 
         if isinstance(msg, Journey):
             self.journey = msg
@@ -187,10 +176,10 @@ class Passenger(Evented, FSM):
 
     @default_state
     @state
-    def asking(self):
+    async def asking(self):
         destination = (
-            self.random.randint(0, self.model.grid.height),
-            self.random.randint(0, self.model.grid.width),
+            self.random.randint(0, self.model.grid.height-1),
+            self.random.randint(0, self.model.grid.width-1),
         )
         self.journey = None
         journey = Journey(
@@ -202,42 +191,42 @@ class Passenger(Evented, FSM):
 
         timeout = 60
         expiration = self.now + timeout
+        self.info(f"Asking for journey at: { self.pos }")
         self.model.broadcast(journey, ttl=timeout, sender=self, agent_class=Driver)
         while not self.journey:
-            self.info(f"Passenger at: { self.pos }. Checking for responses.")
+            self.debug(f"Waiting for responses at: { self.pos }")
             try:
-                # This will call check_messages behind the scenes, and the agent's status will be updated
+                # This will call process_messages behind the scenes, and the agent's status will be updated
                 # If you want to avoid that, you can call it with: check=False
-                yield self.received(expiration=expiration)
+                await self.received(expiration=expiration, delay=10)
             except events.TimedOut:
-                self.info(f"Passenger at: { self.pos }. Asking for journey.")
+                self.info(f"Still no response. Waiting at: { self.pos }")
                 self.model.broadcast(
                     journey, ttl=timeout, sender=self, agent_class=Driver
                 )
                 expiration = self.now + timeout
+        self.info(f"Got a response! Waiting for driver")
         return self.driving_home
 
     @state
-    def driving_home(self):
+    async def driving_home(self):
         while (
             self.pos[0] != self.journey.destination[0]
             or self.pos[1] != self.journey.destination[1]
         ):
             try:
-                yield self.received(timeout=60)
+                await self.received(timeout=60)
             except events.TimedOut:
                 pass
 
-        self.info("Got home safe!")
-        self.die()
+        self.die("Got home safe!")
 
 
-simulation = Simulation(
-    name="RideHailing",
-    model_class=City,
-    model_params={"n_passengers": 2},
+simulation = Simulation(name="RideHailing",
+                        model=City,
                         seed="carsSeed",
-)
+                        max_time=1000,
+                        parameters=dict(n_passengers=2))
 
 if __name__ == "__main__":
-    simulation.run()
+    easy(simulation)

examples/complete.yml

@@ -1,54 +0,0 @@
----
-version: '2'
-name: simple
-group: tests
-dir_path: "/tmp/"
-num_trials: 3
-max_steps: 100
-interval: 1
-seed: "CompleteSeed!"
-model_class: Environment
-model_params:
-  am_i_complete: true
-  topology:
-    params:
-      generator: complete_graph
-      n: 12
-  environment:
-  agents:
-    agent_class: CounterModel
-    topology: true
-    state:
-      times: 1
-      # In this group we are not specifying any topology
-    fixed:
-      - name: 'Environment Agent 1'
-        agent_class: BaseAgent
-        group: environment
-        topology: false
-        hidden: true
-        state:
-          times: 10
-      - agent_class: CounterModel
-        id: 0
-        group: fixed_counters
-        state:
-          times: 1
-          total: 0
-      - agent_class: CounterModel
-        group: fixed_counters
-        id: 1
-    distribution:
-      - agent_class: CounterModel
-        weight: 1
-        group: distro_counters
-        state:
-          times: 3
-      - agent_class: AggregatedCounter
-        weight: 0.2
-    override:
-      - filter:
-          agent_class: AggregatedCounter
-        n: 2
-        state:
-          times: 5

examples/custom_generator/custom_generator.yml

@@ -1,16 +0,0 @@
----
-name: custom-generator
-description: Using a custom generator for the network
-num_trials: 3
-max_steps: 100
-interval: 1
-network_params:
-  generator: mymodule.mygenerator
-# These are custom parameters
-  n: 10
-  n_edges: 5  
-network_agents:
-  - agent_class: CounterModel
-    weight: 1
-    state:
-      state_id: 0

examples/custom_generator/generator_sim.py

@@ -1,6 +1,7 @@
 from networkx import Graph
 import random
 import networkx as nx
+from soil import Simulation, Environment, CounterModel, parameters
 
 
 def mygenerator(n=5, n_edges=5):
@@ -20,3 +21,19 @@ def mygenerator(n=5, n_edges=5):
         n_out = random.choice(nodes)
         G.add_edge(n_in, n_out)
     return G
+
+
+class GeneratorEnv(Environment):
+    """Using a custom generator for the network"""
+
+    generator: parameters.function = staticmethod(mygenerator)
+
+    def init(self):
+        self.create_network(generator=self.generator, n=10, n_edges=5)
+        self.add_agents(CounterModel)
+
+
+sim = Simulation(model=GeneratorEnv, max_steps=10)
+
+if __name__ == '__main__':
+    sim.run(dump=False)

examples/custom_timeouts/custom_timeouts.py

@@ -1,47 +0,0 @@
-from soil.agents import FSM, state, default_state
-from soil.time import Delta
-
-
-class Fibonacci(FSM):
-    """Agent that only executes in t_steps that are Fibonacci numbers"""
-
-    defaults = {"prev": 1}
-
-    @default_state
-    @state
-    def counting(self):
-        self.log("Stopping at {}".format(self.now))
-        prev, self["prev"] = self["prev"], max([self.now, self["prev"]])
-        return None, Delta(prev)
-
-
-class Odds(FSM):
-    """Agent that only executes in odd t_steps"""
-
-    @default_state
-    @state
-    def odds(self):
-        self.log("Stopping at {}".format(self.now))
-        return None, Delta(1 + self.now % 2)
-
-
-from soil import Simulation
-
-simulation = Simulation(
-    model_params={
-        'agents':[
-            {'agent_class': Fibonacci, 'node_id': 0},
-            {'agent_class': Odds, 'node_id': 1}
-        ],
-        'topology': {
-            'params': {
-                'generator': 'complete_graph',
-                'n': 2
-            }
-        },
-    },
-    max_time=100,
-)
-
-if __name__ == "__main__":
-    simulation.run(dry_run=True)

examples/custom_timeouts/custom_timeouts_sim.py

@@ -0,0 +1,40 @@
+from soil.agents import FSM, state, default_state
+
+
+class Fibonacci(FSM):
+    """Agent that only executes in t_steps that are Fibonacci numbers"""
+    prev = 1
+
+    @default_state
+    @state
+    def counting(self):
+        self.log("Stopping at {}".format(self.now))
+        prev, self["prev"] = self["prev"], max([self.now, self["prev"]])
+        return self.delay(prev)
+    
+
+
+class Odds(FSM):
+    """Agent that only executes in odd t_steps"""
+
+    @state(default=True)
+    def odds(self):
+        self.log("Stopping at {}".format(self.now))
+        return self.delay(1 + (self.now % 2))
+
+
+from soil import Environment, Simulation
+from networkx import complete_graph
+
+
+class TimeoutsEnv(Environment):
+    def init(self):
+        self.create_network(generator=complete_graph, n=2)
+        self.add_agent(agent_class=Fibonacci, node_id=0)
+        self.add_agent(agent_class=Odds, node_id=1)
+
+
+sim = Simulation(model=TimeoutsEnv, max_steps=10)
+
+if __name__ == "__main__":
+    sim.run(dump=False)

examples/markov_chains/MarkovChains.ipynb

@@ -0,0 +1,355 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "7641396c-a602-477e-bf03-09e1191ff549",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%load_ext autoreload"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "4f12285c-78db-4ee8-b9c6-7799d34f10f5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%autoreload 1"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "7710bb03-0cb9-413a-a407-fe48855ff917",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%aimport markov_sim"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "2dffca0f-da9e-4f69-ac43-7afe52ad2d32",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%aimport soil\n",
+    "%aimport soil.visualization"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "12871006-70ca-4c6f-8a3e-0aae1d0bce31",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "G = markov_sim.load_city_graph(\"Chamberi, Madrid\", network_type=\"drive\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "31e96cc5-b703-4d2a-a006-7b9a2cedc365",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# env = markov_sim.CityEnv(G=G, n_assets=20, side=10, max_weight=1, seed=10)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "5e070b36-0ba6-4780-8fd4-3c72fa3bb240",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# for i in range(2):\n",
+    "#     env.step()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 35,
+   "id": "56f8b997-65b0-431d-9517-b93edb1cfcd8",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/home/j/.cache/pypoetry/virtualenvs/soil-cCX5yKRx-py3.10/lib/python3.10/site-packages/osmnx/plot.py:955: UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown\n",
+      "  plt.show()\n",
+      "/home/j/.cache/pypoetry/virtualenvs/soil-cCX5yKRx-py3.10/lib/python3.10/site-packages/osmnx/plot.py:955: UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown\n",
+      "  plt.show()\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "86e45bd44e434674b11805fd94e98414",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/html": [
+       "Cannot show widget. You probably want to rerun the code cell above (<i>Click in the code cell, and press Shift+Enter <kbd>⇧</kbd>+<kbd>↩</kbd></i>)."
+      ],
+      "text/plain": [
+       "Cannot show ipywidgets in text"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "from soil.visualization import JupyterViz, GeoNetworkDrawer, Controller\n",
+    "from soil import visualization\n",
+    "from matplotlib import colors\n",
+    "from matplotlib import colormaps\n",
+    "plasma = colormaps.get_cmap('plasma')\n",
+    "model_params = {\n",
+    "    \"n_assets\": {\n",
+    "        \"type\": \"SliderInt\",\n",
+    "        \"value\": 100,\n",
+    "        \"label\": \"Number of assets:\",\n",
+    "        \"min\": 1,\n",
+    "        \"max\": 1000,\n",
+    "        \"step\": 1,\n",
+    "    },\n",
+    "    \"max_weight\": {\n",
+    "        \"type\": \"SliderInt\",\n",
+    "        \"value\": 3,\n",
+    "        \"label\": \"Maximum edge weight:\",\n",
+    "        \"min\": 1,\n",
+    "        \"max\": 20,\n",
+    "        \"step\": 1,\n",
+    "    },\n",
+    "    \"ratio_lazy\": {\n",
+    "        \"type\": \"SliderFloat\",\n",
+    "        \"value\": 0,\n",
+    "        \"label\": \"Ratio of lazy agents (they prefer shorter streets):\",\n",
+    "        \"min\": 0,\n",
+    "        \"max\": 1,\n",
+    "        \"step\": 0.05,\n",
+    "    },\n",
+    "    \"side\": {\n",
+    "        \"type\": \"SliderInt\",\n",
+    "        \"value\": 10,\n",
+    "        \"label\": \"Size of the side:\",\n",
+    "        \"min\": 2,\n",
+    "        \"max\": 20,\n",
+    "        \"step\": 1,\n",
+    "    },\n",
+    "    \"gradual_move\": {\n",
+    "        \"type\": \"Checkbox\",\n",
+    "        \"value\": True,\n",
+    "        \"label\": \"Use gradual movement\",\n",
+    "    }, \n",
+    "    \"lockstep\": {\n",
+    "        \"type\": \"Checkbox\",\n",
+    "        \"value\": True,\n",
+    "        \"label\": \"Run in locksteps\",\n",
+    "    },\n",
+    "    \"G\": G,\n",
+    "    # \"width\": 10,\n",
+    "}\n",
+    "\n",
+    "def colorize(d):\n",
+    "    # print(d)\n",
+    "    if any(a.waiting for a in d):\n",
+    "        return 'red'\n",
+    "    else:\n",
+    "        return 'blue'\n",
+    "\n",
+    "def network_portrayal(graph, spring=True):\n",
+    "    global pos, l\n",
+    "    node_size = [10*(len(node[1][\"agent\"])) for node in graph.nodes(data=True)]\n",
+    "    node_color = [colorize(d[\"agent\"]) for (k, d) in graph.nodes(data=True)]\n",
+    "    # pos = {node: (d[\"x\"], d[\"y\"]) for node, d in graph.nodes(data=True)}\n",
+    "    edge_width = [graph.edges[k]['travel_time']/100 for k in graph.edges]\n",
+    "    # print(edge_width)\n",
+    "    weights = [graph.edges[k]['occupation'] for k in graph.edges]\n",
+    "    norm = colors.Normalize(vmin=0, vmax=max(weights))\n",
+    "    color = plasma(norm(weights))\n",
+    "    # print(color)\n",
+    "    return dict(node_size=node_size, node_color=node_color, edge_linewidth=edge_width, edge_color=color)\n",
+    "\n",
+    "page = visualization.JupyterViz(\n",
+    "    markov_sim.CityEnv,\n",
+    "    model_params,\n",
+    "    measures=[\"NodeGini\", \"EdgeGini\", \"EdgeOccupation\"],\n",
+    "    name=\"City Environment\",\n",
+    "    space_drawer=GeoNetworkDrawer,\n",
+    "    agent_portrayal=network_portrayal,\n",
+    "    columns=3,\n",
+    ")\n",
+    "# This is required to render the visualization in the Jupyter notebook\n",
+    "page"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 39,
+   "id": "70da18d7-66bd-4710-89a6-aca14707c56e",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>NodeGini</th>\n",
+       "      <th>EdgeGini</th>\n",
+       "      <th>EdgeOccupation</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>0.866567</td>\n",
+       "      <td>0.927276</td>\n",
+       "      <td>0.087624</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>0.866567</td>\n",
+       "      <td>0.933494</td>\n",
+       "      <td>0.081301</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>0.863867</td>\n",
+       "      <td>0.933163</td>\n",
+       "      <td>0.078591</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>0.866567</td>\n",
+       "      <td>0.929943</td>\n",
+       "      <td>0.084914</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>0.869433</td>\n",
+       "      <td>0.934949</td>\n",
+       "      <td>0.076784</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>...</th>\n",
+       "      <td>...</td>\n",
+       "      <td>...</td>\n",
+       "      <td>...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>127</th>\n",
+       "      <td>0.880367</td>\n",
+       "      <td>0.934185</td>\n",
+       "      <td>0.075881</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>128</th>\n",
+       "      <td>0.881400</td>\n",
+       "      <td>0.933038</td>\n",
+       "      <td>0.078591</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>129</th>\n",
+       "      <td>0.881400</td>\n",
+       "      <td>0.936299</td>\n",
+       "      <td>0.078591</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>130</th>\n",
+       "      <td>0.881400</td>\n",
+       "      <td>0.929784</td>\n",
+       "      <td>0.086721</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>131</th>\n",
+       "      <td>0.876733</td>\n",
+       "      <td>0.932746</td>\n",
+       "      <td>0.082204</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "<p>132 rows × 3 columns</p>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "     NodeGini  EdgeGini  EdgeOccupation\n",
+       "0    0.866567  0.927276        0.087624\n",
+       "1    0.866567  0.933494        0.081301\n",
+       "2    0.863867  0.933163        0.078591\n",
+       "3    0.866567  0.929943        0.084914\n",
+       "4    0.869433  0.934949        0.076784\n",
+       "..        ...       ...             ...\n",
+       "127  0.880367  0.934185        0.075881\n",
+       "128  0.881400  0.933038        0.078591\n",
+       "129  0.881400  0.936299        0.078591\n",
+       "130  0.881400  0.929784        0.086721\n",
+       "131  0.876733  0.932746        0.082204\n",
+       "\n",
+       "[132 rows x 3 columns]"
+      ]
+     },
+     "execution_count": 39,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "page.controller.model.datacollector.get_model_vars_dataframe()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d9a7d3c8-2f87-47d5-8d27-a7387ea3457d",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

examples/markov_chains/app.py

@@ -0,0 +1,11 @@
+from flask import Flask
+import solara.server.flask
+
+app = Flask(__name__)
+app.register_blueprint(solara.server.flask.blueprint, url_prefix="/solara/")
+
+
+@app.route("/")
+def hello_world():
+    return "<p>Hello, World!</p>"
+

examples/markov_chains/markov_sim.py

@@ -0,0 +1,159 @@
+'''
+This scenario has drivers driving around a city.
+In this model, drivers can only be at intersections, which are treated as nodes in the City Graph (grid).
+
+At the start of the simulation, drivers are randomly positioned in the city grid.
+
+The following models for agent behavior are included:
+
+* DummyDriver: In each simulation step, this type of driver can instantly move to any of the neighboring nodes in the grid, or stay in its place.
+
+'''
+
+import networkx as nx
+from soil import Environment, BaseAgent, state, time
+from mesa.space import NetworkGrid
+import mesa
+import statistics
+
+
+class CityGrid(NetworkGrid):
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args,  **kwargs)
+
+        for (u, v, d) in self.G.edges(data=True):
+            d["occupation"] = 0
+        # self.dijkstras = dict(nx.all_pairs_dijkstra(self.G, weight="length"))
+
+    # def eta(self, pos1, pos2):
+    #     return self.dijkstras[pos1][0][pos2]
+
+    def travel_time(self, pos1, pos2):
+        return float(min(d["travel_time"] for d in self.G.adj[pos1][pos2].values()))
+
+    
+    def node_occupation(self):
+        return {k: len(v.get("agent", [])) for (k, v) in self.G.nodes(data=True)}
+
+    def edge_occupation(self):
+        return {(u,v): d.get('occupation', 1) for (u, v, d) in self.G.edges(data=True)}
+
+
+class Roamer(BaseAgent):
+    waiting = False
+
+    def step(self):
+        '''
+        A simple driver that just moves to a neighboring cell in the city
+        '''
+        yield from self.move_to(None)
+        return self.delay(0)
+    
+    def choose_next(self):
+        opts = self.model.grid.get_neighborhood(self.pos, include_center=False)
+        pos = self.random.choice(opts)
+        delay = self.model.grid.travel_time(self.pos, pos)
+        return pos, delay
+
+    def move_to(self, pos=None):
+        self.waiting = True
+        if pos is None:
+            pos, delay = self.choose_next()
+        if self.model.gradual_move:
+            # Calculate how long it will take, and wait for that long
+            if pos != self.pos:
+                self.model.grid.G.edges[self.pos,pos,0]["occupation"] += 1
+            yield delay
+        if self.model.gradual_move and pos != self.pos:
+            w1 = self.model.grid.G.edges[self.pos,pos,0]["occupation"] 
+            oldpos = self.pos
+            self.model.grid.G.edges[self.pos,pos,0]["occupation"] = w1 - 1
+            assert self.model.grid.G.edges[self.pos,pos,0]["occupation"] == w1-1
+        self.model.grid.move_agent(self, pos)
+        self.waiting = False
+
+
+class LazyRoamer(Roamer):
+    waiting = False
+    def choose_next(self):
+        opts = self.model.grid.get_neighborhood(self.pos, include_center=False)
+        times = [self.model.grid.travel_time(self.pos, other) for other in opts]
+        idx = self.random.choices(range(len(times)), k=1, weights=[1/time for time in times])[0]
+        return opts[idx], times[idx]
+
+
+
+def gini(values):
+    s = sum(values)
+    
+    N = len(values)
+    if s == 0:
+        return 0
+    x = sorted(values)
+
+    B = sum(xi * (N - i) for i, xi in enumerate(x)) / (N * s)
+    return 1 + (1 / N) - 2 * B
+
+
+class CityEnv(Environment):
+    def __init__(self, *, G, side=20, n_assets=100, ratio_lazy=1, lockstep=True, gradual_move=True, max_weight=1, **kwargs):
+        super().__init__(**kwargs)
+        if lockstep:
+            self.schedule = time.Lockstepper(self.schedule)
+        self.n_assets = n_assets
+        self.side = side
+        self.max_weight = max_weight
+        self.gradual_move = gradual_move
+        self.grid = CityGrid(g=G)
+
+        n_lazy = round(self.n_assets * ratio_lazy)
+        n_other = self.n_assets - n_lazy
+        self.add_agents(Roamer, k=n_other)
+        self.add_agents(LazyRoamer, k=n_lazy)
+
+        positions = list(self.grid.G.nodes)
+        for agent in self.get_agents():
+            pos = self.random.choice(positions)
+            self.grid.place_agent(agent, pos)
+        
+        self.datacollector = mesa.DataCollector(
+            model_reporters={
+                "NodeGini": lambda model: gini(model.grid.node_occupation().values()),
+                "EdgeGini": lambda model: gini(model.grid.edge_occupation().values()),            
+                "EdgeOccupation": lambda model: statistics.mean(model.grid.edge_occupation().values()),            
+            }#, agent_reporters={"Wealth": "wealth"}
+        )
+
+class SquareCityEnv(CityEnv):
+    def __init__(self, *, side=20, **kwargs):
+        self.side = side
+        G = nx.grid_graph(dim=[side, side])
+        for (_, _, d) in G.edges(data=True):
+            d["travel_time"] = self.random.randint(1, self.max_weight)
+
+        for (k, d) in G.nodes(data=True):   
+            d["pos"] = k
+        super().__init__(**kwargs, G=G)
+
+import osmnx as ox
+
+
+class NamedCityEnv(CityEnv):
+    def __init__(self, *, location="Chamberi, Madrid", **kwargs):
+        self.location = location
+        super().__init__(**kwargs, G=load_city_graph(location))
+
+
+def load_city_graph(location='Chamberi, Madrid', **kwargs):
+    G = ox.graph.graph_from_place(location, **kwargs)
+    G = ox.add_edge_speeds(G)
+    G = ox.add_edge_travel_times(G)
+    largest = sorted(nx.strongly_connected_components(G), key=lambda x: len(x))[-1]
+    G = G.subgraph(largest)
+    return G
+
+
+if __name__ == "__main__":
+    env = CityEnv()
+    for i in range(100):
+        env.step()

examples/markov_chains/sol.py

@@ -0,0 +1,26 @@
+import solara
+
+@solara.component
+def MainPage(clicks):
+    color = "green"
+    if clicks.value >= 5:
+        color = "red"
+
+    def increment():
+        clicks.value += 1
+        print("clicks", clicks)  # noqa
+
+    solara.Button(label=f"Clicked: {clicks}", on_click=increment, color=color)
+
+@solara.component
+def Page():
+    v = Visualization()
+    v.viz()
+
+class Visualization:
+    def __init__(self):
+        self.clicks = solara.reactive(0)
+
+    def viz(self):
+        from sol_lib import MainPage
+        return MainPage(self.clicks)

examples/markov_chains/sol_lib.py

@@ -0,0 +1,13 @@
+import solara
+
+@solara.component
+def MainPage(clicks):
+    color = "green"
+    if clicks.value >= 5:
+        color = "red"
+
+    def increment():
+        clicks.value += 1
+        print("clicks", clicks)  # noqa
+
+    solara.Button(label=f"Clicked: {clicks}", on_click=increment, color=color)

examples/mesa/mesa.yml

@@ -1,19 +0,0 @@
----
-name: mesa_sim
-group: tests
-dir_path: "/tmp"
-num_trials: 3
-max_steps: 100
-interval: 1
-seed: '1'
-model_class: social_wealth.MoneyEnv
-model_params:
-  generator: social_wealth.graph_generator
-  agents:
-    topology: true
-    distribution:
-      - agent_class: social_wealth.SocialMoneyAgent
-        weight: 1
-  N: 10
-  width: 50
-  height: 50

examples/mesa/mesa_sim.py

@@ -0,0 +1,7 @@
+from soil import Simulation
+from social_wealth import MoneyEnv, graph_generator
+
+sim = Simulation(name="mesa_sim", dump=False, max_steps=10, model=MoneyEnv, parameters=dict(generator=graph_generator, N=10, width=50, height=50))
+
+if __name__ == "__main__":
+    sim.run()

examples/mesa/server.py

@@ -1,5 +1,5 @@
 from mesa.visualization.ModularVisualization import ModularServer
-from soil.visualization import UserSettableParameter
+from mesa.visualization.UserParam import Slider, Choice
 from mesa.visualization.modules import ChartModule, NetworkModule, CanvasGrid
 from social_wealth import MoneyEnv, graph_generator, SocialMoneyAgent
 import networkx as nx
@@ -63,9 +63,8 @@ chart = ChartModule(
     [{"Label": "Gini", "Color": "Black"}], data_collector_name="datacollector"
 )
 
-model_params = {
-    "N": UserSettableParameter(
-        "slider",
+parameters = {
+    "N": Slider(
         "N",
         5,
         1,
@@ -73,8 +72,7 @@ model_params = {
         1,
         description="Choose how many agents to include in the model",
     ),
-    "height": UserSettableParameter(
-        "slider",
+    "height": Slider(
         "height",
         5,
         5,
@@ -82,8 +80,7 @@ model_params = {
         1,
         description="Grid height",
     ),
-    "width": UserSettableParameter(
-        "slider",
+    "width": Slider(
         "width",
         5,
         5,
@@ -91,8 +88,7 @@ model_params = {
         1,
         description="Grid width",
     ),
-    "agent_class": UserSettableParameter(
-        "choice",
+    "agent_class": Choice(
         "Agent class",
         value="MoneyAgent",
         choices=["MoneyAgent", "SocialMoneyAgent"],
@@ -102,12 +98,12 @@ model_params = {
 
 
 canvas_element = CanvasGrid(
-    gridPortrayal, model_params["width"].value, model_params["height"].value, 500, 500
+    gridPortrayal, parameters["width"].value, parameters["height"].value, 500, 500
 )
 
 
 server = ModularServer(
-    MoneyEnv, [grid, chart, canvas_element], "Money Model", model_params
+    MoneyEnv, [grid, chart, canvas_element], "Money Model", parameters
 )
 server.port = 8521
 

examples/mesa/social_wealth.py

@@ -7,7 +7,7 @@ from mesa.space import MultiGrid
 
 # from mesa.time import RandomActivation
 from mesa.datacollection import DataCollector
-from mesa.batchrunner import BatchRunner
+from mesa.batchrunner import batch_run
 
 import networkx as nx
 
@@ -53,7 +53,7 @@ class MoneyAgent(MesaAgent):
             self.give_money()
 
 
-class SocialMoneyAgent(NetworkAgent, MoneyAgent):
+class SocialMoneyAgent(MoneyAgent, NetworkAgent):
     wealth = 1
 
     def give_money(self):
@@ -101,7 +101,7 @@ class MoneyEnv(Environment):
         self.populate_network(agent_class=agent_class)
 
         # Create agents
-        for agent in self.agents:
+        for agent in self.get_agents():
             x = self.random.randrange(self.grid.width)
             y = self.random.randrange(self.grid.height)
             self.grid.place_agent(agent, (x, y))
@@ -122,16 +122,14 @@ if __name__ == "__main__":
 
     variable_params = {"N": range(10, 100, 10)}
 
-    batch_run = BatchRunner(
+    results = batch_run(
         MoneyEnv,
         variable_parameters=variable_params,
         fixed_parameters=fixed_params,
         iterations=5,
-        max_steps=100,
-        model_reporters={"Gini": compute_gini},
+        max_steps=100
     )  
-    batch_run.run_all()
 
-    run_data = batch_run.get_model_vars_dataframe()
-    run_data.head()
+    run_data = pd.DataFrame(results)
+    print(run_data.head())
     print(run_data.Gini)

examples/newsspread/NewsSpread.ipynb

@@ -2,13 +2,12 @@
  "cells": [
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 1,
    "metadata": {
     "ExecuteTime": {
      "end_time": "2017-11-08T16:22:30.732107Z",
      "start_time": "2017-11-08T17:22:30.059855+01:00"
-    },
-    "collapsed": true
+    }
    },
    "outputs": [],
    "source": [
@@ -28,24 +27,16 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 2,
    "metadata": {
     "ExecuteTime": {
      "end_time": "2017-11-08T16:22:35.580593Z",
      "start_time": "2017-11-08T17:22:35.542745+01:00"
     }
    },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Populating the interactive namespace from numpy and matplotlib\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
-    "%pylab inline\n",
+    "%matplotlib inline\n",
     "\n",
     "from soil import *"
    ]
@@ -66,7 +57,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 3,
    "metadata": {
     "ExecuteTime": {
      "end_time": "2017-11-08T16:22:37.242327Z",
@@ -86,7 +77,7 @@
       "  prob_neighbor_spread: 0.0\r\n",
       "  prob_tv_spread: 0.01\r\n",
       "interval: 1\r\n",
-      "max_time: 30\r\n",
+      "max_time: 300\r\n",
       "name: Sim_all_dumb\r\n",
       "network_agents:\r\n",
       "- agent_class: DumbViewer\r\n",
@@ -110,7 +101,7 @@
       "  prob_neighbor_spread: 0.0\r\n",
       "  prob_tv_spread: 0.01\r\n",
       "interval: 1\r\n",
-      "max_time: 30\r\n",
+      "max_time: 300\r\n",
       "name: Sim_half_herd\r\n",
       "network_agents:\r\n",
       "- agent_class: DumbViewer\r\n",
@@ -142,18 +133,18 @@
       "  prob_neighbor_spread: 0.0\r\n",
       "  prob_tv_spread: 0.01\r\n",
       "interval: 1\r\n",
-      "max_time: 30\r\n",
+      "max_time: 300\r\n",
       "name: Sim_all_herd\r\n",
       "network_agents:\r\n",
       "- agent_class: HerdViewer\r\n",
       "  state:\r\n",
       "    has_tv: true\r\n",
-      "    id: neutral\r\n",
+      "    state_id: neutral\r\n",
       "  weight: 1\r\n",
       "- agent_class: HerdViewer\r\n",
       "  state:\r\n",
       "    has_tv: true\r\n",
-      "    id: neutral\r\n",
+      "    state_id: neutral\r\n",
       "  weight: 1\r\n",
       "network_params:\r\n",
       "  generator: barabasi_albert_graph\r\n",
@@ -169,13 +160,13 @@
       "  prob_tv_spread: 0.01\r\n",
       "  prob_neighbor_cure: 0.1\r\n",
       "interval: 1\r\n",
-      "max_time: 30\r\n",
+      "max_time: 300\r\n",
       "name: Sim_wise_herd\r\n",
       "network_agents:\r\n",
       "- agent_class: HerdViewer\r\n",
       "  state:\r\n",
       "    has_tv: true\r\n",
-      "    id: neutral\r\n",
+      "    state_id: neutral\r\n",
       "  weight: 1\r\n",
       "- agent_class: WiseViewer\r\n",
       "  state:\r\n",
@@ -195,13 +186,13 @@
       "  prob_tv_spread: 0.01\r\n",
       "  prob_neighbor_cure: 0.1\r\n",
       "interval: 1\r\n",
-      "max_time: 30\r\n",
+      "max_time: 300\r\n",
       "name: Sim_all_wise\r\n",
       "network_agents:\r\n",
       "- agent_class: WiseViewer\r\n",
       "  state:\r\n",
       "    has_tv: true\r\n",
-      "    id: neutral\r\n",
+      "    state_id: neutral\r\n",
       "  weight: 1\r\n",
       "- agent_class: WiseViewer\r\n",
       "  state:\r\n",
@@ -225,7 +216,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 22,
+   "execution_count": 4,
    "metadata": {
     "ExecuteTime": {
      "end_time": "2017-11-08T18:07:46.781745Z",
@@ -233,7 +224,24 @@
     },
     "scrolled": true
    },
-   "outputs": [],
+   "outputs": [
+    {
+     "ename": "ValueError",
+     "evalue": "No objects to concatenate",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
+      "Cell \u001b[0;32mIn[4], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m evodumb \u001b[38;5;241m=\u001b[39m \u001b[43manalysis\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mread_data\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43msoil_output/Sim_all_dumb/\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mprocess\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43manalysis\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget_count\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mgroup\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mkeys\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m[\u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mid\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m]\u001b[49m\u001b[43m)\u001b[49m;\n",
+      "File \u001b[0;32m/mnt/data/home/j/git/lab.gsi/soil/soil/soil/analysis.py:14\u001b[0m, in \u001b[0;36mread_data\u001b[0;34m(group, *args, **kwargs)\u001b[0m\n\u001b[1;32m     12\u001b[0m iterable \u001b[38;5;241m=\u001b[39m _read_data(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[1;32m     13\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m group:\n\u001b[0;32m---> 14\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mgroup_trials\u001b[49m\u001b[43m(\u001b[49m\u001b[43miterable\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     15\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m     16\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mlist\u001b[39m(iterable)\n",
+      "File \u001b[0;32m/mnt/data/home/j/git/lab.gsi/soil/soil/soil/analysis.py:201\u001b[0m, in \u001b[0;36mgroup_trials\u001b[0;34m(trials, aggfunc)\u001b[0m\n\u001b[1;32m    199\u001b[0m trials \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mlist\u001b[39m(trials)\n\u001b[1;32m    200\u001b[0m trials \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mlist\u001b[39m(\u001b[38;5;28mmap\u001b[39m(\u001b[38;5;28;01mlambda\u001b[39;00m x: x[\u001b[38;5;241m1\u001b[39m] \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(x, \u001b[38;5;28mtuple\u001b[39m) \u001b[38;5;28;01melse\u001b[39;00m x, trials))\n\u001b[0;32m--> 201\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mpd\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mconcat\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtrials\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241m.\u001b[39mgroupby(level\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m0\u001b[39m)\u001b[38;5;241m.\u001b[39magg(aggfunc)\u001b[38;5;241m.\u001b[39mreorder_levels([\u001b[38;5;241m2\u001b[39m, \u001b[38;5;241m0\u001b[39m,\u001b[38;5;241m1\u001b[39m] ,axis\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1\u001b[39m)\n",
+      "File \u001b[0;32m/mnt/data/home/j/git/lab.gsi/soil/soil/.env-v0.20/lib/python3.8/site-packages/pandas/util/_decorators.py:331\u001b[0m, in \u001b[0;36mdeprecate_nonkeyword_arguments.<locals>.decorate.<locals>.wrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    325\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(args) \u001b[38;5;241m>\u001b[39m num_allow_args:\n\u001b[1;32m    326\u001b[0m     warnings\u001b[38;5;241m.\u001b[39mwarn(\n\u001b[1;32m    327\u001b[0m         msg\u001b[38;5;241m.\u001b[39mformat(arguments\u001b[38;5;241m=\u001b[39m_format_argument_list(allow_args)),\n\u001b[1;32m    328\u001b[0m         \u001b[38;5;167;01mFutureWarning\u001b[39;00m,\n\u001b[1;32m    329\u001b[0m         stacklevel\u001b[38;5;241m=\u001b[39mfind_stack_level(),\n\u001b[1;32m    330\u001b[0m     )\n\u001b[0;32m--> 331\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfunc\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
+      "File \u001b[0;32m/mnt/data/home/j/git/lab.gsi/soil/soil/.env-v0.20/lib/python3.8/site-packages/pandas/core/reshape/concat.py:368\u001b[0m, in \u001b[0;36mconcat\u001b[0;34m(objs, axis, join, ignore_index, keys, levels, names, verify_integrity, sort, copy)\u001b[0m\n\u001b[1;32m    146\u001b[0m \u001b[38;5;129m@deprecate_nonkeyword_arguments\u001b[39m(version\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mNone\u001b[39;00m, allowed_args\u001b[38;5;241m=\u001b[39m[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mobjs\u001b[39m\u001b[38;5;124m\"\u001b[39m])\n\u001b[1;32m    147\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mconcat\u001b[39m(\n\u001b[1;32m    148\u001b[0m     objs: Iterable[NDFrame] \u001b[38;5;241m|\u001b[39m Mapping[HashableT, NDFrame],\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    157\u001b[0m     copy: \u001b[38;5;28mbool\u001b[39m \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mTrue\u001b[39;00m,\n\u001b[1;32m    158\u001b[0m ) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m DataFrame \u001b[38;5;241m|\u001b[39m Series:\n\u001b[1;32m    159\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m    160\u001b[0m \u001b[38;5;124;03m    Concatenate pandas objects along a particular axis.\u001b[39;00m\n\u001b[1;32m    161\u001b[0m \n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    366\u001b[0m \u001b[38;5;124;03m    1   3   4\u001b[39;00m\n\u001b[1;32m    367\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[0;32m--> 368\u001b[0m     op \u001b[38;5;241m=\u001b[39m \u001b[43m_Concatenator\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    369\u001b[0m \u001b[43m        \u001b[49m\u001b[43mobjs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    370\u001b[0m \u001b[43m        \u001b[49m\u001b[43maxis\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43maxis\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    371\u001b[0m \u001b[43m        \u001b[49m\u001b[43mignore_index\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mignore_index\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    372\u001b[0m \u001b[43m        \u001b[49m\u001b[43mjoin\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mjoin\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    373\u001b[0m \u001b[43m        \u001b[49m\u001b[43mkeys\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mkeys\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    374\u001b[0m \u001b[43m        \u001b[49m\u001b[43mlevels\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mlevels\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    375\u001b[0m \u001b[43m        \u001b[49m\u001b[43mnames\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mnames\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    376\u001b[0m \u001b[43m        \u001b[49m\u001b[43mverify_integrity\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mverify_integrity\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    377\u001b[0m \u001b[43m        \u001b[49m\u001b[43mcopy\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcopy\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    378\u001b[0m \u001b[43m        \u001b[49m\u001b[43msort\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43msort\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    379\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    381\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m op\u001b[38;5;241m.\u001b[39mget_result()\n",
+      "File \u001b[0;32m/mnt/data/home/j/git/lab.gsi/soil/soil/.env-v0.20/lib/python3.8/site-packages/pandas/core/reshape/concat.py:425\u001b[0m, in \u001b[0;36m_Concatenator.__init__\u001b[0;34m(self, objs, axis, join, keys, levels, names, ignore_index, verify_integrity, copy, sort)\u001b[0m\n\u001b[1;32m    422\u001b[0m     objs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mlist\u001b[39m(objs)\n\u001b[1;32m    424\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(objs) \u001b[38;5;241m==\u001b[39m \u001b[38;5;241m0\u001b[39m:\n\u001b[0;32m--> 425\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mNo objects to concatenate\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m    427\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m keys \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m    428\u001b[0m     objs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mlist\u001b[39m(com\u001b[38;5;241m.\u001b[39mnot_none(\u001b[38;5;241m*\u001b[39mobjs))\n",
+      "\u001b[0;31mValueError\u001b[0m: No objects to concatenate"
+     ]
+    }
+   ],
    "source": [
     "evodumb = analysis.read_data('soil_output/Sim_all_dumb/', process=analysis.get_count, group=True, keys=['id']);"
    ]
@@ -721,9 +729,9 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "venv-soil",
    "language": "python",
-   "name": "python3"
+   "name": "venv-soil"
   },
   "language_info": {
    "codemirror_mode": {
@@ -735,7 +743,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.6.2"
+   "version": "3.8.10"
   },
   "toc": {
    "colors": {