mirror of
https://github.com/gsi-upm/soil
synced 2024-12-21 15:58:13 +00:00
Pre-release version of v1.0
This commit is contained in:
J. Fernando Sánchez
parent
be65592055
commit
cc238d84ec
26
CHANGELOG.md
26
CHANGELOG.md
@ -3,19 +3,31 @@ All notable changes to this project will be documented in this file.
|
||||
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [0.30 UNRELEASED]
|
||||
## [1.0 UNRELEASED]
|
||||
|
||||
Version 1.0 introduced multiple changes, especially on the `Simulation` class and anything related to how configuration is handled.
|
||||
For an explanation of the general changes in version 1.0, please refer to the file `docs/notes_v1.0.rst`.
|
||||
|
||||
### Added
|
||||
* Simple debugging capabilities in `soil.debugging`, with a custom `pdb.Debugger` subclass that exposes commands to list agents and their status and set breakpoints on states (for FSM agents). Try it with `soil --debug <simulation file>`
|
||||
* Ability to run mesa simulations
|
||||
* The `soil.exporters` module to export the results of datacollectors (model.datacollector) into files at the end of trials/simulations
|
||||
* A modular set of classes for environments/models. Now the ability to configure the agents through an agent definition and a topology through a network configuration is split into two classes (`soil.agents.BaseEnvironment` for agents, `soil.agents.NetworkEnvironment` to add topology).
|
||||
* FSM agents can now have generators as states. They work similar to normal states, with one caveat. Only `time` values can be yielded, not a state. This is because the state will not change, it will be resumed after the yield, at the appropriate time. The return value *can* be a state, or a `(state, time)` tuple, just like in normal states.
|
||||
* Environments now have a class method to make them easier to use without a simulation`.run`. Notice that this is different from `run_model`, which is an instance method.
|
||||
* Ability to run simulations using mesa models
|
||||
* The `soil.exporters` module to export the results of datacollectors (`model.datacollector`) into files at the end of trials/simulations
|
||||
* Agents can now have generators as a step function or a state. They work similar to normal functions, with one caveat in the case of `FSM`: only `time` values (or None) can be yielded, not a state. This is because the state will not change, it will be resumed after the yield, at the appropriate time. The return value *can* be a state, or a `(state, time)` tuple, just like in normal states.
|
||||
* Simulations can now specify a `matrix` with possible values for every simulation parameter. The final parameters will be calculated based on the `parameters` used and a cartesian product (i.e., all possible combinations) of each parameter.
|
||||
* Simple debugging capabilities in `soil.debugging`, with a custom `pdb.Debugger` subclass that exposes commands to list agents and their status and set breakpoints on states (for FSM agents). Try it with `soil --debug <simulation file>`
|
||||
### Changed
|
||||
* Configuration schema is very simplified
|
||||
* Configuration schema (`Simulation`) is very simplified. All simulations should be checked
|
||||
* Model / environment variables are expected (but not enforced) to be a single value. This is done to more closely align with mesa
|
||||
* `Exporter.iteration_end` now takes two parameters: `env` (same as before) and `params` (specific parameters for this environment). We considered including a `parameters` attribute in the environment, but this would not be compatible with mesa.
|
||||
* `num_trials` renamed to `iterations`
|
||||
* General renaming of `trial` to `iteration`, to work better with `mesa`
|
||||
* `model_parameters` renamed to `parameters` in simulation
|
||||
* Simulation results for every iteration of a simulation with the same name are stored in a single `sqlite` database
|
||||
|
||||
### Removed
|
||||
* Any `tsih` and `History` integration in the main classes. To record the state of environments/agents, just use a datacollector. In some cases this may be slower or consume more memory than the previous system. However, few cases actually used the full potential of the history, and it came at the cost of unnecessary complexity and worse performance for the majority of cases.
|
||||
|
||||
|
||||
## [0.20.7]
|
||||
### Changed
|
||||
* Creating a `time.When` from another `time.When` does not nest them anymore (it returns the argument)
|
||||
|
||||
@ -7,7 +7,7 @@ Learn how to run your own simulations with our [documentation](http://soilsim.re
|
||||
Follow our [tutorial](examples/tutorial/soil_tutorial.ipynb) to develop your own agent models.
|
||||
|
||||
> **Warning**
|
||||
> Mesa 0.30 introduced many fundamental changes. Check the [documention on how to update your simulations to work with newer versions](docs/notes_v0.30.rst)
|
||||
> Soil 1.0 introduced many fundamental changes. Check the [documention on how to update your simulations to work with newer versions](docs/notes_v1.0.rst)
|
||||
|
||||
## Features
|
||||
|
||||
|
||||
@ -1,7 +1,20 @@
|
||||
Welcome to Soil's documentation!
|
||||
================================
|
||||
|
||||
Soil is an Agent-based Social Simulator in Python focused on Social Networks.
|
||||
Soil is an opinionated Agent-based Social Simulator in Python focused on Social Networks.
|
||||
|
||||
.. image:: soil.png
|
||||
:width: 80%
|
||||
:align: center
|
||||
|
||||
Soil can be installed through pip (see more details in the :doc:`installation` page):
|
||||
|
||||
.. code:: bash
|
||||
|
||||
pip install soil
|
||||
|
||||
|
||||
To get started developing your own simulations and agent behaviors, check out our :doc:`Tutorial <soil_tutorial>` and the `examples on GitHub <https://github.com/gsi-upm/soil/tree/master/examples>.
|
||||
|
||||
If you use Soil in your research, do not forget to cite this paper:
|
||||
|
||||
@ -33,8 +46,6 @@ If you use Soil in your research, do not forget to cite this paper:
|
||||
:caption: Learn more about soil:
|
||||
|
||||
installation
|
||||
quickstart
|
||||
configuration
|
||||
Tutorial <soil_tutorial>
|
||||
|
||||
..
|
||||
|
||||
@ -1,7 +1,10 @@
|
||||
Installation
|
||||
------------
|
||||
|
||||
The easiest way to install Soil is through pip, with Python >= 3.4:
|
||||
Through pip
|
||||
===========
|
||||
|
||||
The easiest way to install Soil is through pip, with Python >= 3.8:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
@ -25,4 +28,38 @@ Or, if you're using using soil programmatically:
|
||||
import soil
|
||||
print(soil.__version__)
|
||||
|
||||
The latest version can be installed through `GitHub <https://github.com/gsi-upm/soil>`_ or `GitLab <https://lab.gsi.upm.es/soil/soil.git>`_.
|
||||
|
||||
|
||||
Web UI
|
||||
======
|
||||
|
||||
Soil also includes a web server that allows you to upload your simulations, change parameters, and visualize the results, including a timeline of the network.
|
||||
To make it work, you have to install soil like this:
|
||||
|
||||
.. code::
|
||||
|
||||
pip install soil[web]
|
||||
|
||||
Once installed, the soil web UI can be run in two ways:
|
||||
|
||||
.. code::
|
||||
|
||||
soil-web
|
||||
|
||||
# OR
|
||||
|
||||
python -m soil.web
|
||||
|
||||
|
||||
Development
|
||||
===========
|
||||
|
||||
The latest version can be downloaded from `GitHub <https://github.com/gsi-upm/soil>`_ and installed manually:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
git clone https://github.com/gsi-upm/soil
|
||||
cd soil
|
||||
python -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install --editable .
|
||||
@ -1,7 +1,7 @@
|
||||
What are the main changes between version 0.3 and 0.2?
|
||||
######################################################
|
||||
What are the main changes in version 1.0?
|
||||
#########################################
|
||||
|
||||
Version 0.3 is a major rewrite of the Soil system, focused on simplifying the API, aligning it with Mesa, and making it easier to use.
|
||||
Version 1.0 is a major rewrite of the Soil system, focused on simplifying the API, aligning it with Mesa, and making it easier to use.
|
||||
Unfortunately, this comes at the cost of backwards compatibility.
|
||||
|
||||
We drew several lessons from the previous version of Soil, and tried to address them in this version.
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 7.0 KiB |
BIN
docs/output_30_0.png
Normal file
BIN
docs/output_30_0.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 4.7 KiB |
BIN
docs/output_34_0.png
Normal file
BIN
docs/output_34_0.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
BIN
docs/output_49_0.png
Normal file
BIN
docs/output_49_0.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 26 KiB |
BIN
docs/output_50_0.png
Normal file
BIN
docs/output_50_0.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 31 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 14 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 14 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 14 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 14 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 15 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 15 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 15 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 14 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 14 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 5.3 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 17 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 17 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 11 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 19 KiB |
@ -1,93 +0,0 @@
|
||||
Quickstart
|
||||
----------
|
||||
|
||||
This section shows how to run your first simulation with Soil.
|
||||
For installation instructions, see :doc:`installation`.
|
||||
|
||||
There are mainly two parts in a simulation: agent classes and simulation configuration.
|
||||
An agent class defines how the agent will behave throughout the simulation.
|
||||
The configuration includes things such as number of agents to use and their type, network topology to use, etc.
|
||||
|
||||
|
||||
.. image:: soil.png
|
||||
:width: 80%
|
||||
:align: center
|
||||
|
||||
|
||||
Soil includes several agent classes in the ``soil.agents`` module, and we will use them in this quickstart.
|
||||
If you are interested in developing your own agents classes, see :doc:`soil_tutorial`.
|
||||
|
||||
Configuration
|
||||
=============
|
||||
To get you started, we will use this configuration (:download:`download the file <quickstart.yml>` directly):
|
||||
|
||||
.. literalinclude:: quickstart.yml
|
||||
:language: yaml
|
||||
|
||||
The agent type used, SISa, is a very simple model.
|
||||
It only has three states (neutral, content and discontent),
|
||||
Its parameters are the probabilities to change from one state to another, either spontaneously or because of contagion from neighboring agents.
|
||||
|
||||
Running the simulation
|
||||
======================
|
||||
|
||||
To see the simulation in action, simply point soil to the configuration, and tell it to store the graph and the history of agent states and environment parameters at every point.
|
||||
|
||||
.. code::
|
||||
|
||||
❯ soil --graph --csv quickstart.yml [13:35:29]
|
||||
INFO:soil:Using config(s): quickstart
|
||||
INFO:soil:Dumping results to soil_output/quickstart : ['csv', 'gexf']
|
||||
INFO:soil:Starting simulation quickstart at 13:35:30.
|
||||
INFO:soil:Starting Simulation quickstart trial 0 at 13:35:30.
|
||||
INFO:soil:Finished Simulation quickstart trial 0 at 13:35:49 in 19.43677067756653 seconds
|
||||
INFO:soil:Starting Dumping simulation quickstart trial 0 at 13:35:49.
|
||||
INFO:soil:Finished Dumping simulation quickstart trial 0 at 13:35:51 in 1.7733407020568848 seconds
|
||||
INFO:soil:Dumping results to soil_output/quickstart
|
||||
INFO:soil:Finished simulation quickstart at 13:35:51 in 21.29862952232361 seconds
|
||||
|
||||
|
||||
The ``CSV`` file should look like this:
|
||||
|
||||
.. code::
|
||||
|
||||
agent_id,t_step,key,value
|
||||
env,0,neutral_discontent_spon_prob,0.05
|
||||
env,0,neutral_discontent_infected_prob,0.1
|
||||
env,0,neutral_content_spon_prob,0.2
|
||||
env,0,neutral_content_infected_prob,0.4
|
||||
env,0,discontent_neutral,0.2
|
||||
env,0,discontent_content,0.05
|
||||
env,0,content_discontent,0.05
|
||||
env,0,variance_d_c,0.05
|
||||
env,0,variance_c_d,0.1
|
||||
|
||||
Results and visualization
|
||||
=========================
|
||||
|
||||
The environment variables are marked as ``agent_id`` env.
|
||||
Th exported values are only stored when they change.
|
||||
To find out how to get every key and value at every point in the simulation, check out the :doc:`soil_tutorial`.
|
||||
|
||||
The dynamic graph is exported as a .gexf file which could be visualized with
|
||||
`Gephi <https://gephi.org/users/download/>`__.
|
||||
Now it is your turn to experiment with the simulation.
|
||||
Change some of the parameters, such as the number of agents, the probability of becoming content, or the type of network, and see how the results change.
|
||||
|
||||
|
||||
Soil also includes a web server that allows you to upload your simulations, change parameters, and visualize the results, including a timeline of the network.
|
||||
To make it work, you have to install soil like this:
|
||||
|
||||
.. code::
|
||||
|
||||
pip install soil[web]
|
||||
|
||||
Once installed, the soil web UI can be run in two ways:
|
||||
|
||||
.. code::
|
||||
|
||||
soil-web
|
||||
|
||||
# OR
|
||||
|
||||
python -m soil.web
|
||||
@ -1,33 +0,0 @@
|
||||
---
|
||||
name: quickstart
|
||||
num_trials: 1
|
||||
max_time: 1000
|
||||
model_params:
|
||||
agents:
|
||||
- agent_class: SISaModel
|
||||
topology: true
|
||||
state:
|
||||
id: neutral
|
||||
weight: 1
|
||||
- agent_class: SISaModel
|
||||
topology: true
|
||||
state:
|
||||
id: content
|
||||
weight: 2
|
||||
topology:
|
||||
params:
|
||||
n: 100
|
||||
k: 5
|
||||
p: 0.2
|
||||
generator: newman_watts_strogatz_graph
|
||||
neutral_discontent_spon_prob: 0.05
|
||||
neutral_discontent_infected_prob: 0.1
|
||||
neutral_content_spon_prob: 0.2
|
||||
neutral_content_infected_prob: 0.4
|
||||
discontent_neutral: 0.2
|
||||
discontent_content: 0.05
|
||||
content_discontent: 0.05
|
||||
variance_d_c: 0.05
|
||||
variance_c_d: 0.1
|
||||
content_neutral: 0.1
|
||||
standard_variance: 0.1
|
||||
File diff suppressed because it is too large
Load Diff
@ -94,9 +94,9 @@ class Driver(Evented, FSM):
|
||||
def check_passengers(self):
|
||||
"""If there are no more passengers, stop forever"""
|
||||
c = self.count_agents(agent_class=Passenger)
|
||||
self.info(f"Passengers left {c}")
|
||||
self.debug(f"Passengers left {c}")
|
||||
if not c:
|
||||
self.die()
|
||||
self.die("No more passengers")
|
||||
|
||||
@default_state
|
||||
@state
|
||||
@ -129,10 +129,13 @@ class Driver(Evented, FSM):
|
||||
@state
|
||||
def driving(self):
|
||||
"""The journey has been accepted. Pick them up and take them to their destination"""
|
||||
self.info(f"Driving towards Passenger {self.journey.passenger.unique_id}")
|
||||
while self.move_towards(self.journey.origin):
|
||||
yield
|
||||
self.info(f"Driving {self.journey.passenger.unique_id} from {self.journey.origin} to {self.journey.destination}")
|
||||
while self.move_towards(self.journey.destination, with_passenger=True):
|
||||
yield
|
||||
self.info("Arrived at destination")
|
||||
self.earnings += self.journey.tip
|
||||
self.model.total_earnings += self.journey.tip
|
||||
self.check_passengers()
|
||||
@ -140,7 +143,7 @@ class Driver(Evented, FSM):
|
||||
|
||||
def move_towards(self, target, with_passenger=False):
|
||||
"""Move one cell at a time towards a target"""
|
||||
self.info(f"Moving { self.pos } -> { target }")
|
||||
self.debug(f"Moving { self.pos } -> { target }")
|
||||
if target[0] == self.pos[0] and target[1] == self.pos[1]:
|
||||
return False
|
||||
|
||||
@ -174,8 +177,8 @@ class Passenger(Evented, FSM):
|
||||
@state
|
||||
def asking(self):
|
||||
destination = (
|
||||
self.random.randint(0, self.model.grid.height),
|
||||
self.random.randint(0, self.model.grid.width),
|
||||
self.random.randint(0, self.model.grid.height-1),
|
||||
self.random.randint(0, self.model.grid.width-1),
|
||||
)
|
||||
self.journey = None
|
||||
journey = Journey(
|
||||
@ -187,19 +190,21 @@ class Passenger(Evented, FSM):
|
||||
|
||||
timeout = 60
|
||||
expiration = self.now + timeout
|
||||
self.info(f"Asking for journey at: { self.pos }")
|
||||
self.model.broadcast(journey, ttl=timeout, sender=self, agent_class=Driver)
|
||||
while not self.journey:
|
||||
self.info(f"Passenger at: { self.pos }. Checking for responses.")
|
||||
self.debug(f"Waiting for responses at: { self.pos }")
|
||||
try:
|
||||
# This will call check_messages behind the scenes, and the agent's status will be updated
|
||||
# If you want to avoid that, you can call it with: check=False
|
||||
yield self.received(expiration=expiration)
|
||||
except events.TimedOut:
|
||||
self.info(f"Passenger at: { self.pos }. Asking for journey.")
|
||||
self.info(f"Still no response. Waiting at: { self.pos }")
|
||||
self.model.broadcast(
|
||||
journey, ttl=timeout, sender=self, agent_class=Driver
|
||||
)
|
||||
expiration = self.now + timeout
|
||||
self.info(f"Got a response! Waiting for driver")
|
||||
return self.driving_home
|
||||
|
||||
@state
|
||||
@ -220,7 +225,7 @@ simulation = Simulation(name="RideHailing",
|
||||
model=City,
|
||||
seed="carsSeed",
|
||||
max_time=1000,
|
||||
model_params=dict(n_passengers=2))
|
||||
parameters=dict(n_passengers=2))
|
||||
|
||||
if __name__ == "__main__":
|
||||
easy(simulation)
|
||||
@ -1,7 +1,7 @@
|
||||
from soil import Simulation
|
||||
from social_wealth import MoneyEnv, graph_generator
|
||||
|
||||
sim = Simulation(name="mesa_sim", dump=False, max_steps=10, interval=2, model=MoneyEnv, model_params=dict(generator=graph_generator, N=10, width=50, height=50))
|
||||
sim = Simulation(name="mesa_sim", dump=False, max_steps=10, interval=2, model=MoneyEnv, parameters=dict(generator=graph_generator, N=10, width=50, height=50))
|
||||
|
||||
if __name__ == "__main__":
|
||||
sim.run()
|
||||
|
||||
@ -63,7 +63,7 @@ chart = ChartModule(
|
||||
[{"Label": "Gini", "Color": "Black"}], data_collector_name="datacollector"
|
||||
)
|
||||
|
||||
model_params = {
|
||||
parameters = {
|
||||
"N": Slider(
|
||||
"N",
|
||||
5,
|
||||
@ -98,12 +98,12 @@ model_params = {
|
||||
|
||||
|
||||
canvas_element = CanvasGrid(
|
||||
gridPortrayal, model_params["width"].value, model_params["height"].value, 500, 500
|
||||
gridPortrayal, parameters["width"].value, parameters["height"].value, 500, 500
|
||||
)
|
||||
|
||||
|
||||
server = ModularServer(
|
||||
MoneyEnv, [grid, chart, canvas_element], "Money Model", model_params
|
||||
MoneyEnv, [grid, chart, canvas_element], "Money Model", parameters
|
||||
)
|
||||
server.port = 8521
|
||||
|
||||
|
||||
@ -116,7 +116,7 @@ for [r1, r2] in product([0, 0.5, 1.0], repeat=2):
|
||||
Simulation(
|
||||
name='newspread_sim',
|
||||
model=NewsSpread,
|
||||
model_params=dict(
|
||||
parameters=dict(
|
||||
ratio_dumb=r1,
|
||||
ratio_herd=r2,
|
||||
ratio_wise=1-r1-r2,
|
||||
@ -124,7 +124,7 @@ for [r1, r2] in product([0, 0.5, 1.0], repeat=2):
|
||||
network_params=netparams,
|
||||
prob_neighbor_spread=0,
|
||||
),
|
||||
num_trials=5,
|
||||
iterations=5,
|
||||
max_steps=300,
|
||||
dump=False,
|
||||
).run()
|
||||
|
||||
@ -38,7 +38,7 @@ simulation = Simulation(
|
||||
name="Programmatic",
|
||||
model=ProgrammaticEnv,
|
||||
seed='Program',
|
||||
num_trials=1,
|
||||
iterations=1,
|
||||
max_time=100,
|
||||
dump=False,
|
||||
)
|
||||
|
||||
@ -178,10 +178,10 @@ class Police(FSM):
|
||||
sim = Simulation(
|
||||
model=CityPubs,
|
||||
name="pubcrawl",
|
||||
num_trials=3,
|
||||
iterations=3,
|
||||
max_steps=10,
|
||||
dump=False,
|
||||
model_params=dict(
|
||||
parameters=dict(
|
||||
network_generator=nx.empty_graph,
|
||||
network_params={"n": 30},
|
||||
model=CityPubs,
|
||||
|
||||
@ -147,7 +147,7 @@ class RandomAccident(BaseAgent):
|
||||
self.debug("Rabbits alive: {}".format(rabbits_alive))
|
||||
|
||||
|
||||
sim = Simulation(model=RabbitsImprovedEnv, max_time=100, seed="MySeed", num_trials=1)
|
||||
sim = Simulation(model=RabbitsImprovedEnv, max_time=100, seed="MySeed", iterations=1)
|
||||
|
||||
if __name__ == "__main__":
|
||||
sim.run()
|
||||
|
||||
@ -155,7 +155,7 @@ class RandomAccident(BaseAgent):
|
||||
|
||||
|
||||
|
||||
sim = Simulation(model=RabbitEnv, max_time=100, seed="MySeed", num_trials=1)
|
||||
sim = Simulation(model=RabbitEnv, max_time=100, seed="MySeed", iterations=1)
|
||||
|
||||
if __name__ == "__main__":
|
||||
sim.run()
|
||||
@ -38,7 +38,7 @@ class RandomEnv(Environment):
|
||||
s = Simulation(
|
||||
name="Programmatic",
|
||||
model=RandomEnv,
|
||||
num_trials=1,
|
||||
iterations=1,
|
||||
max_time=100,
|
||||
dump=False,
|
||||
)
|
||||
|
||||
@ -2,6 +2,7 @@ import networkx as nx
|
||||
from soil.agents import Geo, NetworkAgent, FSM, custom, state, default_state
|
||||
from soil import Environment, Simulation
|
||||
from soil.parameters import *
|
||||
from soil.utils import int_seed
|
||||
|
||||
|
||||
class TerroristEnvironment(Environment):
|
||||
@ -38,9 +39,8 @@ class TerroristEnvironment(Environment):
|
||||
HavenModel
|
||||
], [self.ratio_civil, self.ratio_leader, self.ratio_training, self.ratio_haven])
|
||||
|
||||
@staticmethod
|
||||
def generator(*args, **kwargs):
|
||||
return nx.random_geometric_graph(*args, **kwargs)
|
||||
def generator(self, *args, **kwargs):
|
||||
return nx.random_geometric_graph(*args, **kwargs, seed=int_seed(self._seed))
|
||||
|
||||
class TerroristSpreadModel(FSM, Geo):
|
||||
"""
|
||||
@ -137,7 +137,7 @@ class TerroristSpreadModel(FSM, Geo):
|
||||
|
||||
def ego_search(self, steps=1, center=False, agent=None, **kwargs):
|
||||
"""Get a list of nodes in the ego network of *node* of radius *steps*"""
|
||||
node = agent.node_id
|
||||
node = agent.node_id if agent else self.node_id
|
||||
G = self.subgraph(**kwargs)
|
||||
return nx.ego_graph(G, node, center=center, radius=steps).nodes()
|
||||
|
||||
@ -279,26 +279,26 @@ class TerroristNetworkModel(TerroristSpreadModel):
|
||||
)
|
||||
)
|
||||
neighbours = set(
|
||||
agent.id
|
||||
agent.unique_id
|
||||
for agent in self.get_neighbors(agent_class=TerroristNetworkModel)
|
||||
)
|
||||
search = (close_ups | step_neighbours) - neighbours
|
||||
for agent in self.get_agents(search):
|
||||
social_distance = 1 / self.shortest_path_length(agent.id)
|
||||
spatial_proximity = 1 - self.get_distance(agent.id)
|
||||
social_distance = 1 / self.shortest_path_length(agent.unique_id)
|
||||
spatial_proximity = 1 - self.get_distance(agent.unique_id)
|
||||
prob_new_interaction = (
|
||||
self.weight_social_distance * social_distance
|
||||
+ self.weight_link_distance * spatial_proximity
|
||||
)
|
||||
if (
|
||||
agent["id"] == agent.civilian.id
|
||||
agent.state_id == "civilian"
|
||||
and self.random.random() < prob_new_interaction
|
||||
):
|
||||
self.add_edge(agent)
|
||||
break
|
||||
|
||||
def get_distance(self, target):
|
||||
source_x, source_y = nx.get_node_attributes(self.G, "pos")[self.id]
|
||||
source_x, source_y = nx.get_node_attributes(self.G, "pos")[self.unique_id]
|
||||
target_x, target_y = nx.get_node_attributes(self.G, "pos")[target]
|
||||
dx = abs(source_x - target_x)
|
||||
dy = abs(source_y - target_y)
|
||||
@ -306,16 +306,17 @@ class TerroristNetworkModel(TerroristSpreadModel):
|
||||
|
||||
def shortest_path_length(self, target):
|
||||
try:
|
||||
return nx.shortest_path_length(self.G, self.id, target)
|
||||
return nx.shortest_path_length(self.G, self.unique_id, target)
|
||||
except nx.NetworkXNoPath:
|
||||
return float("inf")
|
||||
|
||||
|
||||
sim = Simulation(
|
||||
model=TerroristEnvironment,
|
||||
num_trials=1,
|
||||
iterations=1,
|
||||
name="TerroristNetworkModel_sim",
|
||||
max_steps=150,
|
||||
seed="default2",
|
||||
skip_test=False,
|
||||
dump=False,
|
||||
)
|
||||
|
||||
File diff suppressed because one or more lines are too long
@ -9,4 +9,5 @@ Mesa>=1.2
|
||||
pydantic>=1.9
|
||||
sqlalchemy>=1.4
|
||||
typing-extensions>=4.4
|
||||
annotated-types>=0.4
|
||||
annotated-types>=0.4
|
||||
tqdm>=4.64
|
||||
|
||||
@ -1 +1 @@
|
||||
0.30.0rc4
|
||||
0.1.0rc1
|
||||
@ -16,6 +16,7 @@ except NameError:
|
||||
basestring = str
|
||||
|
||||
from pathlib import Path
|
||||
from .analysis import *
|
||||
from .agents import *
|
||||
from . import agents
|
||||
from .simulation import *
|
||||
@ -87,7 +88,7 @@ def main(
|
||||
"--graph",
|
||||
"-g",
|
||||
action="store_true",
|
||||
help="Dump each trial's network topology as a GEXF graph. Defaults to false.",
|
||||
help="Dump each iteration's network topology as a GEXF graph. Defaults to false.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--csv",
|
||||
@ -116,11 +117,23 @@ def main(
|
||||
help="Export environment and/or simulations using this exporter",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--until",
|
||||
default="",
|
||||
"--max_time",
|
||||
default="-1",
|
||||
help="Set maximum time for the simulation to run. ",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--max_steps",
|
||||
default="-1",
|
||||
help="Set maximum number of steps for the simulation to run.",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--iterations",
|
||||
default="",
|
||||
help="Set maximum number of iterations (runs) for the simulation.",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--seed",
|
||||
default=None,
|
||||
@ -147,7 +160,8 @@ def main(
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
logger.setLevel(getattr(logging, (args.level or "INFO").upper()))
|
||||
level = getattr(logging, (args.level or "INFO").upper())
|
||||
logger.setLevel(level)
|
||||
|
||||
if args.version:
|
||||
return
|
||||
@ -185,11 +199,14 @@ def main(
|
||||
debug=debug,
|
||||
exporters=exporters,
|
||||
num_processes=args.num_processes,
|
||||
level=level,
|
||||
outdir=output,
|
||||
exporter_params=exp_params,
|
||||
**kwargs)
|
||||
if args.seed is not None:
|
||||
opts["seed"] = args.seed
|
||||
if args.iterations:
|
||||
opts["iterations"] =int(args.iterations)
|
||||
|
||||
if sim:
|
||||
logger.info("Loading simulation instance")
|
||||
@ -218,7 +235,7 @@ def main(
|
||||
k, v = s.split("=", 1)[:2]
|
||||
v = eval(v)
|
||||
tail, *head = k.rsplit(".", 1)[::-1]
|
||||
target = sim.model_params
|
||||
target = sim.parameters
|
||||
if head:
|
||||
for part in head[0].split("."):
|
||||
try:
|
||||
@ -233,7 +250,9 @@ def main(
|
||||
if args.only_convert:
|
||||
print(sim.to_yaml())
|
||||
continue
|
||||
res.append(sim.run(until=args.until))
|
||||
max_time = float(args.max_time) if args.max_time != "-1" else None
|
||||
max_steps = float(args.max_steps) if args.max_steps != "-1" else None
|
||||
res.append(sim.run(max_time=max_time, max_steps=max_steps))
|
||||
|
||||
except Exception as ex:
|
||||
if args.pdb:
|
||||
|
||||
@ -6,9 +6,9 @@ from . import NetworkAgent
|
||||
class Geo(NetworkAgent):
|
||||
"""In this type of network, nodes have a "pos" attribute."""
|
||||
|
||||
def geo_search(self, radius, agent=None, center=False, **kwargs):
|
||||
def geo_search(self, radius, center=False, **kwargs):
|
||||
"""Get a list of nodes whose coordinates are closer than *radius* to *node*."""
|
||||
node = agent.node
|
||||
node = self.node_id
|
||||
|
||||
G = self.subgraph(**kwargs)
|
||||
|
||||
|
||||
@ -220,7 +220,7 @@ class BaseAgent(MesaAgent, MutableMapping, metaclass=MetaAgent):
|
||||
def _check_alive(self):
|
||||
if not self.alive:
|
||||
raise time.DeadAgent(self.unique_id)
|
||||
|
||||
|
||||
def log(self, *message, level=logging.INFO, **kwargs):
|
||||
if not self.logger.isEnabledFor(level):
|
||||
return
|
||||
@ -669,4 +669,4 @@ except ImportError:
|
||||
|
||||
def custom(cls, **kwargs):
|
||||
"""Create a new class from a template class and keyword arguments"""
|
||||
return type(cls.__name__, (cls,), kwargs)
|
||||
return type(cls.__name__, (cls,), kwargs)
|
||||
|
||||
@ -5,7 +5,7 @@ from functools import partial, wraps
|
||||
import inspect
|
||||
|
||||
|
||||
def state(name=None):
|
||||
def state(name=None, default=False):
|
||||
def decorator(func, name=None):
|
||||
"""
|
||||
A state function should return either a state id, or a tuple (state_id, when)
|
||||
@ -40,7 +40,7 @@ def state(name=None):
|
||||
self._last_except = None
|
||||
|
||||
func.id = name or func.__name__
|
||||
func.is_default = False
|
||||
func.is_default = default
|
||||
return func
|
||||
|
||||
if callable(name):
|
||||
@ -101,6 +101,10 @@ class FSM(BaseAgent, metaclass=MetaFSM):
|
||||
if init:
|
||||
self.init()
|
||||
|
||||
@classmethod
|
||||
def states(cls):
|
||||
return list(cls._states.keys())
|
||||
|
||||
def step(self):
|
||||
self.debug(f"Agent {self.unique_id} @ state {self.state_id}")
|
||||
|
||||
|
||||
@ -40,14 +40,11 @@ class NetworkAgent(BaseAgent):
|
||||
|
||||
def iter_agents(self, unique_id=None, *, limit_neighbors=False, **kwargs):
|
||||
unique_ids = None
|
||||
if isinstance(unique_id, list):
|
||||
unique_ids = set(unique_id)
|
||||
elif unique_id is not None:
|
||||
unique_ids = set(
|
||||
[
|
||||
unique_id,
|
||||
]
|
||||
)
|
||||
if unique_ids is not None:
|
||||
try:
|
||||
unique_ids = set(unique_id)
|
||||
except TypeError:
|
||||
unique_ids = set([unique_id])
|
||||
|
||||
if limit_neighbors:
|
||||
neighbor_ids = set()
|
||||
|
||||
49
soil/analysis.py
Normal file
49
soil/analysis.py
Normal file
@ -0,0 +1,49 @@
|
||||
import os
|
||||
import sqlalchemy
|
||||
import pandas as pd
|
||||
from collections import namedtuple
|
||||
|
||||
def plot(env, agent_df=None, model_df=None, steps=False, ignore=["agent_count", ]):
|
||||
"""Plot the model dataframe and agent dataframe together."""
|
||||
if agent_df is None:
|
||||
agent_df = env.agent_df()
|
||||
if model_df is None:
|
||||
model_df = env.model_df()
|
||||
ignore = list(ignore)
|
||||
if not steps:
|
||||
ignore.append("step")
|
||||
else:
|
||||
ignore.append("time")
|
||||
|
||||
ax = model_df.drop(ignore, axis='columns').plot();
|
||||
if not agent_df.empty:
|
||||
agent_df.unstack().apply(lambda x: x.value_counts(),
|
||||
axis=1).fillna(0).plot(ax=ax, secondary_y=True);
|
||||
|
||||
Results = namedtuple("Results", ["config", "parameters", "env", "agents"])
|
||||
#TODO implement reading from CSV and SQLITE
|
||||
def read_sql(fpath=None, name=None, include_agents=False):
|
||||
if not (fpath is None) ^ (name is None):
|
||||
raise ValueError("Specify either a path or a simulation name")
|
||||
if name:
|
||||
fpath = os.path.join("soil_output", name, f"{name}.sqlite")
|
||||
fpath = os.path.abspath(fpath)
|
||||
# TODO: improve url parsing. This is a hacky way to check we weren't given a URL
|
||||
if "://" not in fpath:
|
||||
fpath = f"sqlite:///{fpath}"
|
||||
engine = sqlalchemy.create_engine(fpath)
|
||||
with engine.connect() as conn:
|
||||
env = pd.read_sql_table("env", con=conn,
|
||||
index_col="step").reset_index().set_index([
|
||||
"simulation_id", "params_id",
|
||||
"iteration_id", "step"
|
||||
])
|
||||
agents = pd.read_sql_table("agents", con=conn, index_col=["simulation_id", "params_id", "iteration_id", "step", "agent_id"])
|
||||
config = pd.read_sql_table("configuration", con=conn, index_col="simulation_id")
|
||||
parameters = pd.read_sql_table("parameters", con=conn, index_col=["iteration_id", "params_id", "simulation_id"])
|
||||
try:
|
||||
parameters = parameters.pivot(columns="key", values="value")
|
||||
except Exception as e:
|
||||
print(f"warning: coult not pivot parameters: {e}")
|
||||
|
||||
return Results(config, parameters, env, agents)
|
||||
@ -8,8 +8,10 @@ class SoilCollector(MDC):
|
||||
tables = tables or {}
|
||||
if 'agent_count' not in model_reporters:
|
||||
model_reporters['agent_count'] = lambda m: m.schedule.get_agent_count()
|
||||
if 'state_id' not in agent_reporters:
|
||||
agent_reporters['agent_id'] = lambda agent: getattr(agent, 'state_id', None)
|
||||
if 'time' not in model_reporters:
|
||||
model_reporters['time'] = lambda m: m.now
|
||||
# if 'state_id' not in agent_reporters:
|
||||
# agent_reporters['state_id'] = lambda agent: getattr(agent, 'state_id', None)
|
||||
|
||||
super().__init__(model_reporters=model_reporters,
|
||||
agent_reporters=agent_reporters,
|
||||
|
||||
@ -34,11 +34,13 @@ class BaseEnvironment(Model):
|
||||
:meth:`soil.environment.Environment.get` method.
|
||||
"""
|
||||
|
||||
collector_class = datacollection.SoilCollector
|
||||
|
||||
def __new__(cls,
|
||||
*args: Any,
|
||||
seed="default",
|
||||
dir_path=None,
|
||||
collector_class: type = datacollection.SoilCollector,
|
||||
collector_class: type = None,
|
||||
agent_reporters: Optional[Any] = None,
|
||||
model_reporters: Optional[Any] = None,
|
||||
tables: Optional[Any] = None,
|
||||
@ -46,6 +48,7 @@ class BaseEnvironment(Model):
|
||||
"""Create a new model with a default seed value"""
|
||||
self = super().__new__(cls, *args, seed=seed, **kwargs)
|
||||
self.dir_path = dir_path or os.getcwd()
|
||||
collector_class = collector_class or cls.collector_class
|
||||
collector_class = serialization.deserialize(collector_class)
|
||||
self.datacollector = collector_class(
|
||||
model_reporters=model_reporters,
|
||||
@ -69,6 +72,7 @@ class BaseEnvironment(Model):
|
||||
dir_path=None,
|
||||
schedule_class=time.TimedActivation,
|
||||
interval=1,
|
||||
logger = None,
|
||||
agents: Optional[Dict] = None,
|
||||
collector_class: type = datacollection.SoilCollector,
|
||||
agent_reporters: Optional[Any] = None,
|
||||
@ -80,10 +84,15 @@ class BaseEnvironment(Model):
|
||||
|
||||
super().__init__()
|
||||
|
||||
|
||||
self.current_id = -1
|
||||
|
||||
self.id = id
|
||||
|
||||
if logger:
|
||||
self.logger = logger
|
||||
else:
|
||||
self.logger = utils.logger.getChild(self.id)
|
||||
|
||||
if schedule_class is None:
|
||||
schedule_class = time.TimedActivation
|
||||
@ -93,8 +102,6 @@ class BaseEnvironment(Model):
|
||||
self.interval = interval
|
||||
self.schedule = schedule_class(self)
|
||||
|
||||
self.logger = utils.logger.getChild(self.id)
|
||||
|
||||
for (k, v) in env_params.items():
|
||||
self[k] = v
|
||||
|
||||
@ -102,6 +109,7 @@ class BaseEnvironment(Model):
|
||||
self.add_agents(**agents)
|
||||
if init:
|
||||
self.init()
|
||||
self.datacollector.collect(self)
|
||||
|
||||
def init(self):
|
||||
pass
|
||||
@ -115,6 +123,22 @@ class BaseEnvironment(Model):
|
||||
|
||||
def count_agents(self, *args, **kwargs):
|
||||
return sum(1 for i in self.agents(*args, **kwargs))
|
||||
|
||||
def agent_df(self, steps=False):
|
||||
df = self.datacollector.get_agent_vars_dataframe()
|
||||
if steps:
|
||||
df.index.rename(["step", "agent_id"], inplace=True)
|
||||
return df
|
||||
model_df = self.datacollector.get_model_vars_dataframe()
|
||||
df.index = df.index.set_levels(model_df.time, level=0).rename(["time", "agent_id"])
|
||||
return df
|
||||
|
||||
def model_df(self, steps=False):
|
||||
df = self.datacollector.get_model_vars_dataframe()
|
||||
if steps:
|
||||
return df
|
||||
df.index.rename("step", inplace=True)
|
||||
return df.reset_index().set_index("time")
|
||||
|
||||
@property
|
||||
def now(self):
|
||||
@ -171,11 +195,12 @@ class BaseEnvironment(Model):
|
||||
self.schedule.step()
|
||||
self.datacollector.collect(self)
|
||||
|
||||
msg = "Model data:\n"
|
||||
max_width = max(len(k) for k in self.datacollector.model_vars.keys())
|
||||
for (k, v) in self.datacollector.model_vars.items():
|
||||
msg += f"\t{k:<{max_width}}: {v[-1]:>6}\n"
|
||||
self.logger.info(f"--- Steps: {self.schedule.steps:^5} - Time: {self.now:^5} --- " + msg)
|
||||
if self.logger.isEnabledFor(logging.DEBUG):
|
||||
msg = "Model data:\n"
|
||||
max_width = max(len(k) for k in self.datacollector.model_vars.keys())
|
||||
for (k, v) in self.datacollector.model_vars.items():
|
||||
msg += f"\t{k:<{max_width}}: {v[-1]:>6}\n"
|
||||
self.logger.debug(f"--- Steps: {self.schedule.steps:^5} - Time: {self.now:^5} --- " + msg)
|
||||
|
||||
def add_model_reporter(self, name, func=None):
|
||||
if not func:
|
||||
@ -186,9 +211,18 @@ class BaseEnvironment(Model):
|
||||
if agent_type:
|
||||
reporter = lambda a: getattr(a, name) if isinstance(a, agent_type) else None
|
||||
else:
|
||||
reporter = name
|
||||
reporter = lambda a: getattr(a, name, None)
|
||||
self.datacollector._new_agent_reporter(name, reporter)
|
||||
|
||||
@classmethod
|
||||
def run(cls, *,
|
||||
iterations=1,
|
||||
num_processes=1, **kwargs):
|
||||
from .simulation import Simulation
|
||||
return Simulation(name=cls.__name__,
|
||||
model=cls, iterations=iterations,
|
||||
num_processes=num_processes, **kwargs).run()
|
||||
|
||||
def __getitem__(self, key):
|
||||
try:
|
||||
return getattr(self, key)
|
||||
@ -250,6 +284,7 @@ class NetworkEnvironment(BaseEnvironment):
|
||||
self._check_agent_nodes()
|
||||
if init:
|
||||
self.init()
|
||||
self.datacollector.collect(self)
|
||||
|
||||
def add_agent(self, agent_class, *args, node_id=None, topology=None, **kwargs):
|
||||
if node_id is None and topology is None:
|
||||
@ -373,7 +408,7 @@ class EventedEnvironment(BaseEnvironment):
|
||||
for agent in self.agents(**kwargs):
|
||||
if agent == sender:
|
||||
continue
|
||||
self.logger.info(f"Telling {repr(agent)}: {msg} ttl={ttl}")
|
||||
self.logger.debug(f"Telling {repr(agent)}: {msg} ttl={ttl}")
|
||||
try:
|
||||
inbox = agent._inbox
|
||||
except AttributeError:
|
||||
|
||||
@ -8,9 +8,10 @@ from textwrap import dedent, indent
|
||||
|
||||
import matplotlib.pyplot as plt
|
||||
import networkx as nx
|
||||
import pandas as pd
|
||||
|
||||
|
||||
from .serialization import deserialize
|
||||
from .serialization import deserialize, serialize
|
||||
from .utils import try_backup, open_or_reuse, logger, timer
|
||||
|
||||
|
||||
@ -68,12 +69,12 @@ class Exporter:
|
||||
"""Method to call when the simulation ends"""
|
||||
pass
|
||||
|
||||
def trial_start(self, env):
|
||||
"""Method to call when a trial start"""
|
||||
def iteration_start(self, env):
|
||||
"""Method to call when a iteration start"""
|
||||
pass
|
||||
|
||||
def trial_end(self, env):
|
||||
"""Method to call when a trial ends"""
|
||||
def iteration_end(self, env, params, params_id):
|
||||
"""Method to call when a iteration ends"""
|
||||
pass
|
||||
|
||||
def output(self, f, mode="w", **kwargs):
|
||||
@ -85,27 +86,39 @@ class Exporter:
|
||||
f = os.path.join(self.outdir, f)
|
||||
except TypeError:
|
||||
pass
|
||||
return open_or_reuse(f, mode=mode, **kwargs)
|
||||
return open_or_reuse(f, mode=mode, backup=self.simulation.backup, **kwargs)
|
||||
|
||||
def get_dfs(self, env):
|
||||
yield from get_dc_dfs(env.datacollector, trial_id=env.id)
|
||||
def get_dfs(self, env, **kwargs):
|
||||
yield from get_dc_dfs(env.datacollector,
|
||||
simulation_id=self.simulation.id,
|
||||
iteration_id=env.id,
|
||||
**kwargs)
|
||||
|
||||
|
||||
def get_dc_dfs(dc, trial_id=None):
|
||||
dfs = {
|
||||
"env": dc.get_model_vars_dataframe(),
|
||||
"agents": dc.get_agent_vars_dataframe(),
|
||||
}
|
||||
def get_dc_dfs(dc, **kwargs):
|
||||
dfs = {}
|
||||
dfe = dc.get_model_vars_dataframe()
|
||||
dfe.index.rename("step", inplace=True)
|
||||
dfs["env"] = dfe
|
||||
try:
|
||||
dfa = dc.get_agent_vars_dataframe()
|
||||
dfa.index.rename(["step", "agent_id"], inplace=True)
|
||||
dfs["agents"] = dfa
|
||||
except UserWarning:
|
||||
pass
|
||||
for table_name in dc.tables:
|
||||
dfs[table_name] = dc.get_table_dataframe(table_name)
|
||||
if trial_id:
|
||||
for (name, df) in dfs.items():
|
||||
df["trial_id"] = trial_id
|
||||
for (name, df) in dfs.items():
|
||||
for (k, v) in kwargs.items():
|
||||
df[k] = v
|
||||
df.set_index(["simulation_id", "iteration_id"], append=True, inplace=True)
|
||||
|
||||
yield from dfs.items()
|
||||
|
||||
|
||||
class SQLite(Exporter):
|
||||
"""Writes sqlite results"""
|
||||
sim_started = False
|
||||
|
||||
def sim_start(self):
|
||||
if not self.dump:
|
||||
@ -113,46 +126,64 @@ class SQLite(Exporter):
|
||||
return
|
||||
self.dbpath = os.path.join(self.outdir, f"{self.simulation.name}.sqlite")
|
||||
logger.info("Dumping results to %s", self.dbpath)
|
||||
try_backup(self.dbpath, remove=True)
|
||||
if self.simulation.backup:
|
||||
try_backup(self.dbpath, remove=True)
|
||||
|
||||
if self.simulation.overwrite:
|
||||
if os.path.exists(self.dbpath):
|
||||
os.remove(self.dbpath)
|
||||
|
||||
self.engine = create_engine(f"sqlite:///{self.dbpath}", echo=False)
|
||||
|
||||
def trial_end(self, env):
|
||||
sim_dict = {k: serialize(v)[0] for (k,v) in self.simulation.to_dict().items()}
|
||||
sim_dict["simulation_id"] = self.simulation.id
|
||||
df = pd.DataFrame([sim_dict])
|
||||
df.to_sql("configuration", con=self.engine, if_exists="append")
|
||||
|
||||
def iteration_end(self, env, params, params_id, *args, **kwargs):
|
||||
if not self.dump:
|
||||
logger.info("Running in NO DUMP mode, the database will NOT be created")
|
||||
logger.info("Running in NO DUMP mode. Results will NOT be saved to a DB.")
|
||||
return
|
||||
|
||||
with timer(
|
||||
"Dumping simulation {} trial {}".format(self.simulation.name, env.id)
|
||||
"Dumping simulation {} iteration {}".format(self.simulation.name, env.id)
|
||||
):
|
||||
|
||||
engine = create_engine(f"sqlite:///{self.dbpath}", echo=False)
|
||||
pd.DataFrame([{"simulation_id": self.simulation.id,
|
||||
"params_id": params_id,
|
||||
"iteration_id": env.id,
|
||||
"key": k,
|
||||
"value": serialize(v)[0]} for (k,v) in params.items()]).to_sql("parameters", con=self.engine, if_exists="append")
|
||||
|
||||
for (t, df) in self.get_dfs(env):
|
||||
df.to_sql(t, con=engine, if_exists="append")
|
||||
for (t, df) in self.get_dfs(env, params_id=params_id):
|
||||
df.to_sql(t, con=self.engine, if_exists="append")
|
||||
|
||||
class csv(Exporter):
|
||||
"""Export the state of each environment (and its agents) a CSV file for the simulation"""
|
||||
|
||||
"""Export the state of each environment (and its agents) in a separate CSV file"""
|
||||
def sim_start(self):
|
||||
super().sim_start()
|
||||
|
||||
def trial_end(self, env):
|
||||
def iteration_end(self, env, params, params_id, *args, **kwargs):
|
||||
with timer(
|
||||
"[CSV] Dumping simulation {} trial {} @ dir {}".format(
|
||||
"[CSV] Dumping simulation {} iteration {} @ dir {}".format(
|
||||
self.simulation.name, env.id, self.outdir
|
||||
)
|
||||
):
|
||||
for (df_name, df) in self.get_dfs(env):
|
||||
with self.output("{}.{}.csv".format(env.id, df_name)) as f:
|
||||
for (df_name, df) in self.get_dfs(env, params_id=params_id):
|
||||
with self.output("{}.{}.csv".format(env.id, df_name), mode="a") as f:
|
||||
df.to_csv(f)
|
||||
|
||||
|
||||
# TODO: reimplement GEXF exporting without history
|
||||
class gexf(Exporter):
|
||||
def trial_end(self, env):
|
||||
def iteration_end(self, env, *args, **kwargs):
|
||||
if not self.dump:
|
||||
logger.info("Not dumping GEXF (NO_DUMP mode)")
|
||||
return
|
||||
|
||||
with timer(
|
||||
"[GEXF] Dumping simulation {} trial {}".format(self.simulation.name, env.id)
|
||||
"[GEXF] Dumping simulation {} iteration {}".format(self.simulation.name, env.id)
|
||||
):
|
||||
with self.output("{}.gexf".format(env.id), mode="wb") as f:
|
||||
network.dump_gexf(env.history_to_graph(), f)
|
||||
@ -164,13 +195,13 @@ class dummy(Exporter):
|
||||
with self.output("dummy", "w") as f:
|
||||
f.write("simulation started @ {}\n".format(current_time()))
|
||||
|
||||
def trial_start(self, env):
|
||||
def iteration_start(self, env):
|
||||
with self.output("dummy", "w") as f:
|
||||
f.write("trial started@ {}\n".format(current_time()))
|
||||
f.write("iteration started@ {}\n".format(current_time()))
|
||||
|
||||
def trial_end(self, env):
|
||||
def iteration_end(self, env, *args, **kwargs):
|
||||
with self.output("dummy", "w") as f:
|
||||
f.write("trial ended@ {}\n".format(current_time()))
|
||||
f.write("iteration ended@ {}\n".format(current_time()))
|
||||
|
||||
def sim_end(self):
|
||||
with self.output("dummy", "a") as f:
|
||||
@ -178,7 +209,7 @@ class dummy(Exporter):
|
||||
|
||||
|
||||
class graphdrawing(Exporter):
|
||||
def trial_end(self, env):
|
||||
def iteration_end(self, env, *args, **kwargs):
|
||||
# Outside effects
|
||||
f = plt.figure()
|
||||
nx.draw(
|
||||
@ -193,9 +224,9 @@ class graphdrawing(Exporter):
|
||||
|
||||
|
||||
class summary(Exporter):
|
||||
"""Print a summary of each trial to sys.stdout"""
|
||||
"""Print a summary of each iteration to sys.stdout"""
|
||||
|
||||
def trial_end(self, env):
|
||||
def iteration_end(self, env, *args, **kwargs):
|
||||
msg = ""
|
||||
for (t, df) in self.get_dfs(env):
|
||||
if not len(df):
|
||||
@ -227,7 +258,7 @@ class YAML(Exporter):
|
||||
if not self.dump:
|
||||
logger.debug("NOT dumping results")
|
||||
return
|
||||
with self.output(self.simulation.name + ".dumped.yml") as f:
|
||||
with self.output(self.simulation.id + ".dumped.yml") as f:
|
||||
logger.info(f"Dumping simulation configuration to {self.outdir}")
|
||||
f.write(self.simulation.to_yaml())
|
||||
|
||||
@ -238,19 +269,14 @@ class default(Exporter):
|
||||
exporter_cls = exporter_cls or [YAML, SQLite]
|
||||
self.inner = [cls(*args, **kwargs) for cls in exporter_cls]
|
||||
|
||||
def sim_start(self):
|
||||
def sim_start(self, *args, **kwargs):
|
||||
for exporter in self.inner:
|
||||
exporter.sim_start()
|
||||
exporter.sim_start(*args, **kwargs)
|
||||
|
||||
def sim_end(self):
|
||||
def sim_end(self, *args, **kwargs):
|
||||
for exporter in self.inner:
|
||||
exporter.sim_end()
|
||||
exporter.sim_end(*args, **kwargs)
|
||||
|
||||
def trial_start(self, env):
|
||||
def iteration_end(self, *args, **kwargs):
|
||||
for exporter in self.inner:
|
||||
exporter.trial_start(env)
|
||||
|
||||
|
||||
def trial_end(self, env):
|
||||
for exporter in self.inner:
|
||||
exporter.trial_end(env)
|
||||
exporter.iteration_end(*args, **kwargs)
|
||||
|
||||
@ -140,7 +140,7 @@ def get_module(modname):
|
||||
module = importlib.import_module(modname)
|
||||
KNOWN_MODULES[modname] = module
|
||||
return KNOWN_MODULES[modname]
|
||||
|
||||
|
||||
|
||||
def name(value, known_modules=KNOWN_MODULES):
|
||||
"""Return a name that can be imported, to serialize/deserialize an object"""
|
||||
@ -181,7 +181,7 @@ def serialize_dict(d, known_modules=KNOWN_MODULES):
|
||||
d = dict(d)
|
||||
except (ValueError, TypeError) as ex:
|
||||
return serialize(d)[0]
|
||||
for (k, v) in d.items():
|
||||
for (k, v) in reversed(list(d.items())):
|
||||
if isinstance(v, dict):
|
||||
d[k] = serialize_dict(v, known_modules=known_modules)
|
||||
elif isinstance(v, list):
|
||||
|
||||
@ -1,23 +1,26 @@
|
||||
import os
|
||||
from time import time as current_time, strftime
|
||||
import importlib
|
||||
import sys
|
||||
import yaml
|
||||
import traceback
|
||||
import hashlib
|
||||
|
||||
import inspect
|
||||
import logging
|
||||
import networkx as nx
|
||||
|
||||
from tqdm.auto import tqdm
|
||||
|
||||
from textwrap import dedent
|
||||
|
||||
from dataclasses import dataclass, field, asdict, replace
|
||||
from typing import Any, Dict, Union, Optional, List
|
||||
|
||||
|
||||
from networkx.readwrite import json_graph
|
||||
from functools import partial
|
||||
from contextlib import contextmanager
|
||||
import pickle
|
||||
from itertools import product
|
||||
import json
|
||||
|
||||
|
||||
from . import serialization, exporters, utils, basestring, agents
|
||||
from .environment import Environment
|
||||
@ -41,11 +44,13 @@ def do_not_run():
|
||||
|
||||
def _iter_queued():
|
||||
while _QUEUED:
|
||||
(cls, args, kwargs) = _QUEUED.pop(0)
|
||||
yield replace(cls, **kwargs)
|
||||
(cls, params) = _QUEUED.pop(0)
|
||||
yield replace(cls, parameters=params)
|
||||
|
||||
|
||||
# TODO: change documentation for simulation
|
||||
# TODO: rename iterations to iterations
|
||||
# TODO: make parameters a dict of iterable/any
|
||||
@dataclass
|
||||
class Simulation:
|
||||
"""
|
||||
@ -57,18 +62,21 @@ class Simulation:
|
||||
description: A description of the simulation.
|
||||
group: The group that the simulation belongs to.
|
||||
model: The model to use for the simulation. This can be a string or a class.
|
||||
model_params: The parameters to pass to the model.
|
||||
parameters: The parameters to pass to the model.
|
||||
matrix: A matrix of values for each parameter.
|
||||
seed: The seed to use for the simulation.
|
||||
dir_path: The directory path to use for the simulation.
|
||||
max_time: The maximum time to run the simulation.
|
||||
max_steps: The maximum number of steps to run the simulation.
|
||||
interval: The interval to use for the simulation.
|
||||
num_trials: The number of trials (times) to run the simulation.
|
||||
iterations: The number of iterations (times) to run the simulation.
|
||||
num_processes: The number of processes to use for the simulation. If greater than one, simulations will be performed in parallel. This may make debugging and error handling difficult.
|
||||
tables: The tables to use in the simulation datacollector
|
||||
agent_reporters: The agent reporters to use in the datacollector
|
||||
model_reporters: The model reporters to use in the datacollector
|
||||
dry_run: Whether or not to run the simulation. If True, the simulation will not be run.
|
||||
backup: Whether or not to backup the simulation. If True, the simulation files will be backed up to a different directory.
|
||||
overwrite: Whether or not to replace existing simulation data.
|
||||
source_file: Python file to use to find additional classes.
|
||||
"""
|
||||
|
||||
@ -77,24 +85,28 @@ class Simulation:
|
||||
name: Optional[str] = None
|
||||
description: Optional[str] = ""
|
||||
group: str = None
|
||||
backup: bool = False
|
||||
overwrite: bool = False
|
||||
dry_run: bool = False
|
||||
dump: bool = False
|
||||
model: Union[str, type] = "soil.Environment"
|
||||
model_params: dict = field(default_factory=dict)
|
||||
seed: str = field(default_factory=lambda: current_time())
|
||||
parameters: dict = field(default_factory=dict)
|
||||
matrix: dict = field(default_factory=dict)
|
||||
seed: str = "default"
|
||||
dir_path: str = field(default_factory=lambda: os.getcwd())
|
||||
max_time: float = float("inf")
|
||||
max_steps: int = -1
|
||||
max_time: float = None
|
||||
max_steps: int = None
|
||||
interval: int = 1
|
||||
num_trials: int = 1
|
||||
iterations: int = 1
|
||||
num_processes: Optional[int] = 1
|
||||
exporters: Optional[List[str]] = field(default_factory=lambda: [exporters.default])
|
||||
model_reporters: Optional[Dict[str, Any]] = field(default_factory=dict)
|
||||
agent_reporters: Optional[Dict[str, Any]] = field(default_factory=dict)
|
||||
tables: Optional[Dict[str, Any]] = field(default_factory=dict)
|
||||
outdir: Optional[str] = None
|
||||
outdir: str = field(default_factory=lambda: os.path.join(os.getcwd(), "soil_output"))
|
||||
# outdir: Optional[str] = None
|
||||
exporter_params: Optional[Dict[str, Any]] = field(default_factory=dict)
|
||||
dry_run: bool = False
|
||||
dump: bool = False
|
||||
extra: Dict[str, Any] = field(default_factory=dict)
|
||||
level: int = logging.INFO
|
||||
skip_test: Optional[bool] = False
|
||||
debug: Optional[bool] = False
|
||||
|
||||
@ -103,14 +115,39 @@ class Simulation:
|
||||
if isinstance(self.model, str):
|
||||
self.name = self.model
|
||||
else:
|
||||
self.name = self.model.__class__.__name__
|
||||
self.name = self.model.__name__
|
||||
self.logger = logger.getChild(self.name)
|
||||
self.logger.setLevel(self.level)
|
||||
|
||||
def run_simulation(self, *args, **kwargs):
|
||||
return self.run(*args, **kwargs)
|
||||
if self.source_file:
|
||||
source_file = self.source_file
|
||||
if not os.path.isabs(source_file):
|
||||
source_file = os.path.abspath(os.path.join(self.dir_path, source_file))
|
||||
serialization.add_source_file(source_file)
|
||||
self.source_file = source_file
|
||||
|
||||
def run(self, *args, **kwargs):
|
||||
if isinstance(self.model, str):
|
||||
self.model = serialization.deserialize(self.model)
|
||||
|
||||
def deserialize_reporters(reporters):
|
||||
for (k, v) in reporters.items():
|
||||
if isinstance(v, str) and v.startswith("py:"):
|
||||
reporters[k] = serialization.deserialize(v.split(":", 1)[1])
|
||||
return reporters
|
||||
|
||||
self.agent_reporters = deserialize_reporters(self.agent_reporters)
|
||||
self.model_reporters = deserialize_reporters(self.model_reporters)
|
||||
self.tables = deserialize_reporters(self.tables)
|
||||
if self.source_file:
|
||||
serialization.remove_source_file(self.source_file)
|
||||
self.id = f"{self.name}_{current_time()}"
|
||||
|
||||
def run(self, **kwargs):
|
||||
"""Run the simulation and return the list of resulting environments"""
|
||||
logger.info(
|
||||
if kwargs:
|
||||
return replace(self, **kwargs).run()
|
||||
|
||||
self.logger.debug(
|
||||
dedent(
|
||||
"""
|
||||
Simulation:
|
||||
@ -119,179 +156,156 @@ class Simulation:
|
||||
)
|
||||
+ self.to_yaml()
|
||||
)
|
||||
param_combinations = self._collect_params(**kwargs)
|
||||
if _AVOID_RUNNING:
|
||||
_QUEUED.append((self, args, kwargs))
|
||||
_QUEUED.extend((self, param) for param in param_combinations)
|
||||
return []
|
||||
return list(self._run_gen(*args, **kwargs))
|
||||
|
||||
def _run_gen(
|
||||
self,
|
||||
num_processes=1,
|
||||
dry_run=None,
|
||||
dump=None,
|
||||
exporters=None,
|
||||
outdir=None,
|
||||
exporter_params={},
|
||||
log_level=None,
|
||||
**kwargs,
|
||||
):
|
||||
"""Run the simulation and yield the resulting environments."""
|
||||
if log_level:
|
||||
logger.setLevel(log_level)
|
||||
outdir = outdir or self.outdir
|
||||
logger.info("Using exporters: %s", exporters or [])
|
||||
logger.info("Output directory: %s", outdir)
|
||||
if dry_run is None:
|
||||
dry_run = self.dry_run
|
||||
if dump is None:
|
||||
dump = self.dump
|
||||
if exporters is None:
|
||||
exporters = self.exporters
|
||||
if not exporter_params:
|
||||
exporter_params = self.exporter_params
|
||||
self.logger.debug("Using exporters: %s", self.exporters or [])
|
||||
|
||||
exporters = serialization.deserialize_all(
|
||||
exporters,
|
||||
self.exporters,
|
||||
simulation=self,
|
||||
known_modules=[
|
||||
"soil.exporters",
|
||||
],
|
||||
dump=dump and not dry_run,
|
||||
outdir=outdir,
|
||||
**exporter_params,
|
||||
dump=self.dump and not self.dry_run,
|
||||
outdir=self.outdir,
|
||||
**self.exporter_params,
|
||||
)
|
||||
|
||||
if self.source_file:
|
||||
source_file = self.source_file
|
||||
if not os.path.isabs(source_file):
|
||||
source_file = os.path.abspath(os.path.join(self.dir_path, source_file))
|
||||
serialization.add_source_file(source_file)
|
||||
try:
|
||||
results = []
|
||||
for exporter in exporters:
|
||||
exporter.sim_start()
|
||||
|
||||
with utils.timer("simulation {}".format(self.name)):
|
||||
for params in tqdm(param_combinations, desc=self.name, unit="configuration"):
|
||||
for (k, v) in params.items():
|
||||
tqdm.write(f"{k} = {v}")
|
||||
sha = hashlib.sha256()
|
||||
sha.update(repr(sorted(params.items())).encode())
|
||||
params_id = sha.hexdigest()[:7]
|
||||
for env in self._run_iters_for_params(params):
|
||||
for exporter in exporters:
|
||||
exporter.sim_start()
|
||||
exporter.iteration_end(env, params, params_id)
|
||||
results.append(env)
|
||||
|
||||
if dry_run:
|
||||
def func(*args, **kwargs):
|
||||
return None
|
||||
else:
|
||||
func = self.run_trial
|
||||
for exporter in exporters:
|
||||
exporter.sim_end()
|
||||
|
||||
for env in utils.run_parallel(
|
||||
func=self.run_trial,
|
||||
iterable=range(int(self.num_trials)),
|
||||
num_processes=num_processes,
|
||||
log_level=log_level,
|
||||
**kwargs,
|
||||
):
|
||||
if env is None and dry_run:
|
||||
return results
|
||||
|
||||
def _collect_params(self):
|
||||
|
||||
parameters = []
|
||||
if self.parameters:
|
||||
parameters.append(self.parameters)
|
||||
if self.matrix:
|
||||
assert isinstance(self.matrix, dict)
|
||||
for values in product(*(self.matrix.values())):
|
||||
parameters.append(dict(zip(self.matrix.keys(), values)))
|
||||
|
||||
if not parameters:
|
||||
parameters = [{}]
|
||||
if self.dump:
|
||||
self.logger.info("Output directory: %s", self.outdir)
|
||||
|
||||
return parameters
|
||||
|
||||
def _run_iters_for_params(
|
||||
self,
|
||||
params
|
||||
):
|
||||
"""Run the simulation and yield the resulting environments."""
|
||||
|
||||
try:
|
||||
if self.source_file:
|
||||
serialization.add_source_file(self.source_file)
|
||||
|
||||
with utils.timer(f"running for config {params}"):
|
||||
if self.dry_run:
|
||||
def func(*args, **kwargs):
|
||||
return None
|
||||
else:
|
||||
func = self._run_model
|
||||
|
||||
for env in tqdm(utils.run_parallel(
|
||||
func=func,
|
||||
iterable=range(self.iterations),
|
||||
**params,
|
||||
), total=self.iterations, leave=False):
|
||||
if env is None and self.dry_run:
|
||||
continue
|
||||
|
||||
for exporter in exporters:
|
||||
exporter.trial_end(env)
|
||||
|
||||
yield env
|
||||
|
||||
for exporter in exporters:
|
||||
exporter.sim_end()
|
||||
finally:
|
||||
pass
|
||||
# TODO: reintroduce
|
||||
# if self.source_file:
|
||||
# serialization.remove_source_file(self.source_file)
|
||||
if self.source_file:
|
||||
serialization.remove_source_file(self.source_file)
|
||||
|
||||
def get_env(self, trial_id=0, model_params=None, **kwargs):
|
||||
"""Create an environment for a trial of the simulation"""
|
||||
def _get_env(self, iteration_id, params):
|
||||
"""Create an environment for a iteration of the simulation"""
|
||||
|
||||
def deserialize_reporters(reporters):
|
||||
for (k, v) in reporters.items():
|
||||
if isinstance(v, str) and v.startswith("py:"):
|
||||
reporters[k] = serialization.deserialize(v.split(":", 1)[1])
|
||||
return reporters
|
||||
iteration_id = str(iteration_id)
|
||||
|
||||
params = self.model_params.copy()
|
||||
if model_params:
|
||||
params.update(model_params)
|
||||
params.update(kwargs)
|
||||
agent_reporters = self.agent_reporters
|
||||
agent_reporters.update(params.pop("agent_reporters", {}))
|
||||
model_reporters = self.model_reporters
|
||||
model_reporters.update(params.pop("model_reporters", {}))
|
||||
|
||||
agent_reporters = self.agent_reporters.copy()
|
||||
agent_reporters.update(deserialize_reporters(params.pop("agent_reporters", {})))
|
||||
model_reporters = self.model_reporters.copy()
|
||||
model_reporters.update(deserialize_reporters(params.pop("model_reporters", {})))
|
||||
tables = self.tables.copy()
|
||||
tables.update(deserialize_reporters(params.pop("tables", {})))
|
||||
|
||||
env = serialization.deserialize(self.model)
|
||||
return env(
|
||||
id=f"{self.name}_trial_{trial_id}",
|
||||
seed=f"{self.seed}_trial_{trial_id}",
|
||||
return self.model(
|
||||
id=iteration_id,
|
||||
seed=f"{self.seed}_iteration_{iteration_id}",
|
||||
dir_path=self.dir_path,
|
||||
interval=self.interval,
|
||||
logger=self.logger.getChild(iteration_id),
|
||||
agent_reporters=agent_reporters,
|
||||
model_reporters=model_reporters,
|
||||
tables=tables,
|
||||
tables=self.tables,
|
||||
**params,
|
||||
)
|
||||
|
||||
def run_trial(
|
||||
self, trial_id=None, until=None, log_file=False, log_level=logging.INFO, **opts
|
||||
):
|
||||
def _run_model(self, iteration_id, **params):
|
||||
"""
|
||||
Run a single trial of the simulation
|
||||
Run a single iteration of the simulation
|
||||
|
||||
"""
|
||||
if log_level:
|
||||
logger.setLevel(log_level)
|
||||
model = self.get_env(trial_id, **opts)
|
||||
trial_id = trial_id if trial_id is not None else current_time()
|
||||
with utils.timer("Simulation {} trial {}".format(self.name, trial_id)):
|
||||
return self.run_model(
|
||||
model=model, trial_id=trial_id, until=until, log_level=log_level
|
||||
)
|
||||
# Set-up iteration environment and graph
|
||||
model = self._get_env(iteration_id, params)
|
||||
with utils.timer("Simulation {} iteration {}".format(self.name, iteration_id)):
|
||||
|
||||
def run_model(self, model, until=None, **opts):
|
||||
# Set-up trial environment and graph
|
||||
until = float(until or self.max_time or "inf")
|
||||
max_time = self.max_time
|
||||
max_steps = self.max_steps
|
||||
|
||||
# Set up agents on nodes
|
||||
def is_done():
|
||||
return not model.running
|
||||
|
||||
if until and hasattr(model.schedule, "time"):
|
||||
prev = is_done
|
||||
|
||||
def is_done():
|
||||
return prev() or model.schedule.time >= until
|
||||
if (max_time is not None) and (max_steps is not None):
|
||||
is_done = lambda model: (not model.running) or (model.schedule.time >= max_time) or (model.schedule.steps >= max_steps)
|
||||
elif max_time is not None:
|
||||
is_done = lambda model: (not model.running) or (model.schedule.time >= max_time)
|
||||
elif max_steps is not None:
|
||||
is_done = lambda model: (not model.running) or (model.schedule.steps >= max_steps)
|
||||
else:
|
||||
is_done = lambda model: not model.running
|
||||
|
||||
if not model.schedule.agents:
|
||||
raise Exception("No agents in model. This is probably a bug. Make sure that the model has agents scheduled after its initialization.")
|
||||
if not model.schedule.agents:
|
||||
raise Exception("No agents in model. This is probably a bug. Make sure that the model has agents scheduled after its initialization.")
|
||||
|
||||
if self.max_steps and self.max_steps > 0 and hasattr(model.schedule, "steps"):
|
||||
prev_steps = is_done
|
||||
|
||||
def is_done():
|
||||
return prev_steps() or model.schedule.steps >= self.max_steps
|
||||
|
||||
newline = "\n"
|
||||
logger.info(
|
||||
dedent(
|
||||
f"""
|
||||
Model stats:
|
||||
Agent count: { model.schedule.get_agent_count() }):
|
||||
Topology size: { len(model.G) if hasattr(model, "G") else 0 }
|
||||
"""
|
||||
newline = "\n"
|
||||
self.logger.debug(
|
||||
dedent(
|
||||
f"""
|
||||
Model stats:
|
||||
Agent count: { model.schedule.get_agent_count() }):
|
||||
Topology size: { len(model.G) if hasattr(model, "G") else 0 }
|
||||
"""
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
if self.debug:
|
||||
set_trace()
|
||||
if self.debug:
|
||||
set_trace()
|
||||
|
||||
while not is_done():
|
||||
utils.logger.debug(
|
||||
f'Simulation time {model.schedule.time}/{until}.'
|
||||
)
|
||||
model.step()
|
||||
while not is_done(model):
|
||||
self.logger.debug(
|
||||
f'Simulation time {model.schedule.time}/{max_time}.'
|
||||
)
|
||||
model.step()
|
||||
|
||||
return model
|
||||
|
||||
@ -333,10 +347,9 @@ def from_config(conf_or_path):
|
||||
return lst[0]
|
||||
|
||||
|
||||
def iter_from_py(pyfile, module_name='custom_simulation', **kwargs):
|
||||
def iter_from_py(pyfile, module_name='imported_file', **kwargs):
|
||||
"""Try to load every Simulation instance in a given Python file"""
|
||||
import importlib
|
||||
import inspect
|
||||
added = False
|
||||
sims = []
|
||||
assert not _AVOID_RUNNING
|
||||
@ -377,3 +390,6 @@ def run_from_file(*files, **kwargs):
|
||||
for sim in iter_from_file(*files):
|
||||
logger.info(f"Using config(s): {sim.name}")
|
||||
sim.run_simulation(**kwargs)
|
||||
|
||||
def run(env, iterations=1, num_processes=1, dump=False, name="test", **kwargs):
|
||||
return Simulation(model=env, iterations=iterations, name=name, dump=dump, num_processes=num_processes, **kwargs).run()
|
||||
29
soil/time.py
29
soil/time.py
@ -1,6 +1,6 @@
|
||||
from mesa.time import BaseScheduler
|
||||
from queue import Empty
|
||||
from heapq import heappush, heappop
|
||||
from heapq import heappush, heappop, heapreplace
|
||||
import math
|
||||
|
||||
from inspect import getsource
|
||||
@ -99,7 +99,8 @@ class TimedActivation(BaseScheduler):
|
||||
self._shuffle = shuffle
|
||||
# self.step_interval = getattr(self.model, "interval", 1)
|
||||
self.step_interval = self.model.interval
|
||||
self.logger = logger.getChild(f"time_{ self.model }")
|
||||
self.logger = getattr(self.model, "logger", logger).getChild(f"time_{ self.model }")
|
||||
self.next_time = self.time
|
||||
|
||||
def add(self, agent: MesaAgent, when=None):
|
||||
if when is None:
|
||||
@ -110,7 +111,7 @@ class TimedActivation(BaseScheduler):
|
||||
self._schedule(agent, None, when)
|
||||
super().add(agent)
|
||||
|
||||
def _schedule(self, agent, condition=None, when=None):
|
||||
def _schedule(self, agent, condition=None, when=None, replace=False):
|
||||
if condition:
|
||||
if not when:
|
||||
when, condition = condition.schedule_next(
|
||||
@ -125,7 +126,10 @@ class TimedActivation(BaseScheduler):
|
||||
else:
|
||||
key = (when, agent.unique_id, condition)
|
||||
self._next[agent.unique_id] = key
|
||||
heappush(self._queue, (key, agent))
|
||||
if replace:
|
||||
heapreplace(self._queue, (key, agent))
|
||||
else:
|
||||
heappush(self._queue, (key, agent))
|
||||
|
||||
def step(self) -> None:
|
||||
"""
|
||||
@ -144,10 +148,9 @@ class TimedActivation(BaseScheduler):
|
||||
if when > self.time:
|
||||
break
|
||||
|
||||
heappop(self._queue)
|
||||
if cond:
|
||||
if not cond.ready(agent, self.time):
|
||||
self._schedule(agent, cond)
|
||||
self._schedule(agent, cond, replace=True)
|
||||
continue
|
||||
try:
|
||||
agent._last_return = cond.return_value(agent)
|
||||
@ -164,36 +167,38 @@ class TimedActivation(BaseScheduler):
|
||||
returned = agent.step()
|
||||
except DeadAgent:
|
||||
agent.alive = False
|
||||
heappop(self._queue)
|
||||
continue
|
||||
|
||||
# Check status for MESA agents
|
||||
if not getattr(agent, "alive", True):
|
||||
heappop(self._queue)
|
||||
continue
|
||||
|
||||
if returned:
|
||||
next_check = returned.schedule_next(
|
||||
self.time, self.step_interval, first=True
|
||||
)
|
||||
self._schedule(agent, when=next_check[0], condition=next_check[1])
|
||||
self._schedule(agent, when=next_check[0], condition=next_check[1], replace=True)
|
||||
else:
|
||||
next_check = (self.time + self.step_interval, None)
|
||||
|
||||
self._schedule(agent)
|
||||
self._schedule(agent, replace=True)
|
||||
|
||||
self.steps += 1
|
||||
|
||||
if not self._queue:
|
||||
self.time = INFINITY
|
||||
self.model.running = False
|
||||
return self.time
|
||||
self.time = INFINITY
|
||||
return
|
||||
|
||||
next_time = self._queue[0][0][0]
|
||||
|
||||
|
||||
if next_time < self.time:
|
||||
raise Exception(
|
||||
f"An agent has been scheduled for a time in the past, there is probably an error ({when} < {self.time})"
|
||||
)
|
||||
self.logger.debug(f"Updating time step: {self.time} -> {next_time}")
|
||||
self.logger.debug("Updating time step: %s -> %s ", self.time, next_time)
|
||||
|
||||
self.time = next_time
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@ from multiprocessing import Pool, cpu_count
|
||||
from contextlib import contextmanager
|
||||
|
||||
logger = logging.getLogger("soil")
|
||||
logger.setLevel(logging.INFO)
|
||||
logger.setLevel(logging.WARNING)
|
||||
|
||||
timeformat = "%H:%M:%S"
|
||||
|
||||
@ -24,7 +24,7 @@ consoleHandler = logging.StreamHandler()
|
||||
consoleHandler.setFormatter(logFormatter)
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.DEBUG,
|
||||
level=logging.INFO,
|
||||
handlers=[
|
||||
consoleHandler,
|
||||
],
|
||||
@ -60,7 +60,7 @@ def try_backup(path, remove=False):
|
||||
if not os.path.exists(backup_dir):
|
||||
os.makedirs(backup_dir)
|
||||
newpath = os.path.join(backup_dir, "{}@{}".format(os.path.basename(path), stamp))
|
||||
if move:
|
||||
if remove:
|
||||
move(path, newpath)
|
||||
else:
|
||||
copyfile(path, newpath)
|
||||
@ -126,7 +126,7 @@ def unflatten_dict(d):
|
||||
|
||||
def run_and_return_exceptions(func, *args, **kwargs):
|
||||
"""
|
||||
A wrapper for run_trial that catches exceptions and returns them.
|
||||
A wrapper for a function that catches exceptions and returns them.
|
||||
It is meant for async simulations.
|
||||
"""
|
||||
try:
|
||||
@ -154,3 +154,7 @@ def run_parallel(func, iterable, num_processes=1, **kwargs):
|
||||
else:
|
||||
for i in iterable:
|
||||
yield func(i, **kwargs)
|
||||
|
||||
|
||||
def int_seed(seed: str):
|
||||
return int.from_bytes(seed.encode(), "little")
|
||||
@ -54,8 +54,7 @@ class TestAgents(TestCase):
|
||||
class MyAgent(agents.FSM):
|
||||
run = 0
|
||||
|
||||
@agents.default_state
|
||||
@agents.state("original")
|
||||
@agents.state("original", default=True)
|
||||
def root(self):
|
||||
self.run += 1
|
||||
return self.other
|
||||
@ -65,10 +64,11 @@ class TestAgents(TestCase):
|
||||
self.run += 1
|
||||
|
||||
e = environment.Environment()
|
||||
a = MyAgent(model=e, unique_id=e.next_id())
|
||||
a.step()
|
||||
a = e.add_agent(MyAgent)
|
||||
e.step()
|
||||
assert a.run == 1
|
||||
a.step()
|
||||
print("DONE")
|
||||
|
||||
def test_broadcast(self):
|
||||
"""
|
||||
|
||||
@ -28,13 +28,16 @@ class TestConfig(TestCase):
|
||||
|
||||
def test_torvalds_config(self):
|
||||
sim = simulation.from_config(os.path.join(ROOT, "test_config.yml"))
|
||||
assert sim.interval == 2
|
||||
MAX_STEPS = 10
|
||||
INTERVAL = 2
|
||||
assert sim.interval == INTERVAL
|
||||
assert sim.max_steps == MAX_STEPS
|
||||
envs = sim.run()
|
||||
assert len(envs) == 1
|
||||
env = envs[0]
|
||||
assert env.interval == 2
|
||||
assert env.count_agents() == 3
|
||||
assert env.now == 20
|
||||
assert env.now == INTERVAL * MAX_STEPS
|
||||
|
||||
|
||||
def make_example_test(path, cfg):
|
||||
@ -42,10 +45,10 @@ def make_example_test(path, cfg):
|
||||
root = os.getcwd()
|
||||
print(path)
|
||||
s = simulation.from_config(cfg)
|
||||
iterations = s.max_time * s.num_trials
|
||||
iterations = s.max_time * s.iterations
|
||||
if iterations > 1000:
|
||||
s.max_time = 100
|
||||
s.num_trials = 1
|
||||
s.iterations = 1
|
||||
if cfg.skip_test and not FORCE_TESTS:
|
||||
self.skipTest('Example ignored.')
|
||||
envs = s.run_simulation(dump=False)
|
||||
@ -53,7 +56,7 @@ def make_example_test(path, cfg):
|
||||
for env in envs:
|
||||
assert env
|
||||
try:
|
||||
n = cfg.model_params['topology']['params']['n']
|
||||
n = cfg.parameters['topology']['params']['n']
|
||||
assert len(list(env.network_agents)) == n
|
||||
assert env.now > 0 # It has run
|
||||
assert env.now <= cfg.max_time # But not further than allowed
|
||||
|
||||
@ -28,17 +28,22 @@ def get_test_for_sims(sims, path):
|
||||
if sim.skip_test and not FORCE_TESTS:
|
||||
continue
|
||||
run = True
|
||||
iterations = sim.max_steps * sim.num_trials
|
||||
|
||||
if sim.max_steps is None:
|
||||
sim.max_steps = 100
|
||||
|
||||
iterations = sim.max_steps * sim.iterations
|
||||
if iterations < 0 or iterations > 1000:
|
||||
sim.max_steps = 100
|
||||
sim.num_trials = 1
|
||||
envs = sim.run_simulation(dump=False)
|
||||
sim.iterations = 1
|
||||
|
||||
envs = sim.run(dump=False)
|
||||
assert envs
|
||||
for env in envs:
|
||||
assert env
|
||||
assert env.now > 0
|
||||
try:
|
||||
n = sim.model_params["network_params"]["n"]
|
||||
n = sim.parameters["network_params"]["n"]
|
||||
assert len(list(env.network_agents)) == n
|
||||
except KeyError:
|
||||
pass
|
||||
|
||||
@ -9,28 +9,29 @@ from soil import exporters
|
||||
from soil import environment
|
||||
from soil import simulation
|
||||
from soil import agents
|
||||
from soil import decorators
|
||||
|
||||
from mesa import Agent as MesaAgent
|
||||
|
||||
|
||||
class Dummy(exporters.Exporter):
|
||||
started = False
|
||||
trials = 0
|
||||
iterations = 0
|
||||
ended = False
|
||||
total_time = 0
|
||||
called_start = 0
|
||||
called_trial = 0
|
||||
called_iteration = 0
|
||||
called_end = 0
|
||||
|
||||
def sim_start(self):
|
||||
self.__class__.called_start += 1
|
||||
self.__class__.started = True
|
||||
|
||||
def trial_end(self, env):
|
||||
def iteration_end(self, env, *args, **kwargs):
|
||||
assert env
|
||||
self.__class__.trials += 1
|
||||
self.__class__.iterations += 1
|
||||
self.__class__.total_time += env.now
|
||||
self.__class__.called_trial += 1
|
||||
self.__class__.called_iteration += 1
|
||||
|
||||
def sim_end(self):
|
||||
self.__class__.ended = True
|
||||
@ -44,77 +45,78 @@ class Exporters(TestCase):
|
||||
class SimpleEnv(environment.Environment):
|
||||
def init(self):
|
||||
self.add_agent(agent_class=MesaAgent)
|
||||
|
||||
|
||||
num_trials = 5
|
||||
iterations = 5
|
||||
max_time = 2
|
||||
s = simulation.Simulation(num_trials=num_trials, max_time=max_time, name="exporter_sim",
|
||||
dump=False, model=SimpleEnv)
|
||||
s = simulation.Simulation(iterations=iterations,
|
||||
max_time=max_time, name="exporter_sim",
|
||||
exporters=[Dummy], dump=False, model=SimpleEnv)
|
||||
|
||||
for env in s.run_simulation(exporters=[Dummy], dump=False):
|
||||
for env in s.run():
|
||||
assert len(env.agents) == 1
|
||||
|
||||
assert Dummy.started
|
||||
assert Dummy.ended
|
||||
assert Dummy.called_start == 1
|
||||
assert Dummy.called_end == 1
|
||||
assert Dummy.called_trial == num_trials
|
||||
assert Dummy.trials == num_trials
|
||||
assert Dummy.total_time == max_time * num_trials
|
||||
assert Dummy.called_iteration == iterations
|
||||
assert Dummy.iterations == iterations
|
||||
assert Dummy.total_time == max_time * iterations
|
||||
|
||||
def test_writing(self):
|
||||
"""Try to write CSV, sqlite and YAML (without no_dump)"""
|
||||
n_trials = 5
|
||||
n_iterations = 5
|
||||
n_nodes = 4
|
||||
max_time = 2
|
||||
config = {
|
||||
"name": "exporter_sim",
|
||||
"model_params": {
|
||||
"network_generator": "complete_graph",
|
||||
"network_params": {"n": n_nodes},
|
||||
"agent_class": "CounterModel",
|
||||
},
|
||||
"max_time": max_time,
|
||||
"num_trials": n_trials,
|
||||
"dump": True,
|
||||
}
|
||||
output = io.StringIO()
|
||||
s = simulation.from_config(config)
|
||||
tmpdir = tempfile.mkdtemp()
|
||||
envs = s.run_simulation(
|
||||
|
||||
class ConstantEnv(environment.Environment):
|
||||
@decorators.report
|
||||
@property
|
||||
def constant(self):
|
||||
return 1
|
||||
|
||||
s = simulation.Simulation(
|
||||
model=ConstantEnv,
|
||||
name="exporter_sim",
|
||||
exporters=[
|
||||
exporters.default,
|
||||
exporters.csv,
|
||||
],
|
||||
model_params={
|
||||
"agent_reporters": {"times": "times"},
|
||||
"model_reporters": {
|
||||
"constant": lambda x: 1,
|
||||
},
|
||||
},
|
||||
dump=True,
|
||||
outdir=tmpdir,
|
||||
exporter_params={"copy_to": output},
|
||||
parameters=dict(
|
||||
network_generator="complete_graph",
|
||||
network_params={"n": n_nodes},
|
||||
agent_class="CounterModel",
|
||||
agent_reporters={"times": "times"},
|
||||
),
|
||||
max_time=max_time,
|
||||
outdir=tmpdir,
|
||||
iterations=n_iterations,
|
||||
dump=True,
|
||||
)
|
||||
envs = s.run()
|
||||
result = output.getvalue()
|
||||
|
||||
simdir = os.path.join(tmpdir, s.group or "", s.name)
|
||||
with open(os.path.join(simdir, "{}.dumped.yml".format(s.name))) as f:
|
||||
with open(os.path.join(simdir, "{}.dumped.yml".format(s.id))) as f:
|
||||
result = f.read()
|
||||
assert result
|
||||
|
||||
try:
|
||||
for e in envs:
|
||||
dbpath = os.path.join(simdir, f"{s.name}.sqlite")
|
||||
db = sqlite3.connect(dbpath)
|
||||
cur = db.cursor()
|
||||
agent_entries = cur.execute("SELECT times FROM agents WHERE times > 0").fetchall()
|
||||
env_entries = cur.execute("SELECT constant from env WHERE constant == 1").fetchall()
|
||||
assert len(agent_entries) == n_nodes * n_trials * max_time
|
||||
assert len(env_entries) == n_trials * max_time
|
||||
dbpath = os.path.join(simdir, f"{s.name}.sqlite")
|
||||
db = sqlite3.connect(dbpath)
|
||||
cur = db.cursor()
|
||||
agent_entries = cur.execute("SELECT times FROM agents WHERE times > 0").fetchall()
|
||||
env_entries = cur.execute("SELECT constant from env WHERE constant == 1").fetchall()
|
||||
assert len(agent_entries) == n_nodes * n_iterations * max_time
|
||||
assert len(env_entries) == n_iterations * (max_time + 1) # +1 for the initial state
|
||||
|
||||
for e in envs:
|
||||
with open(os.path.join(simdir, "{}.env.csv".format(e.id))) as f:
|
||||
result = f.read()
|
||||
assert result
|
||||
|
||||
finally:
|
||||
shutil.rmtree(tmpdir)
|
||||
|
||||
@ -30,14 +30,14 @@ class TestMain(TestCase):
|
||||
def test_empty_simulation(self):
|
||||
"""A simulation with a base behaviour should do nothing"""
|
||||
config = {
|
||||
"model_params": {
|
||||
"parameters": {
|
||||
"topology": join(ROOT, "test.gexf"),
|
||||
"agent_class": MesaAgent,
|
||||
},
|
||||
"max_time": 1
|
||||
}
|
||||
s = simulation.from_config(config)
|
||||
s.run_simulation(dump=False)
|
||||
s.run(dump=False)
|
||||
|
||||
def test_network_agent(self):
|
||||
"""
|
||||
@ -45,9 +45,9 @@ class TestMain(TestCase):
|
||||
agent should be able to update its state."""
|
||||
config = {
|
||||
"name": "CounterAgent",
|
||||
"num_trials": 1,
|
||||
"iterations": 1,
|
||||
"max_time": 2,
|
||||
"model_params": {
|
||||
"parameters": {
|
||||
"network_params": {
|
||||
"generator": nx.complete_graph,
|
||||
"n": 2,
|
||||
@ -93,7 +93,7 @@ class TestMain(TestCase):
|
||||
try:
|
||||
os.chdir(os.path.dirname(pyfile))
|
||||
s = simulation.from_py(pyfile)
|
||||
env = s.run_simulation(dump=False)[0]
|
||||
env = s.run(dump=False)[0]
|
||||
for a in env.network_agents:
|
||||
skill_level = a["skill_level"]
|
||||
if a.node_id == "Torvalds":
|
||||
@ -157,11 +157,11 @@ class TestMain(TestCase):
|
||||
n_trials = 50
|
||||
max_time = 2
|
||||
s = simulation.Simulation(
|
||||
model_params=dict(agents=dict(agent_classes=[CheckRun], k=1)),
|
||||
num_trials=n_trials,
|
||||
parameters=dict(agents=dict(agent_classes=[CheckRun], k=1)),
|
||||
iterations=n_trials,
|
||||
max_time=max_time,
|
||||
)
|
||||
runs = list(s.run_simulation(dump=False))
|
||||
runs = list(s.run(dump=False))
|
||||
over = list(x.now for x in runs if x.now > 2)
|
||||
assert len(runs) == n_trials
|
||||
assert len(over) == 0
|
||||
@ -212,13 +212,24 @@ class TestMain(TestCase):
|
||||
for sim in sims:
|
||||
assert sim
|
||||
assert sim.name == "newspread_sim"
|
||||
assert sim.num_trials == 5
|
||||
assert sim.iterations == 5
|
||||
assert sim.max_steps == 300
|
||||
assert not sim.dump
|
||||
assert sim.model_params
|
||||
assert "ratio_dumb" in sim.model_params
|
||||
assert "ratio_herd" in sim.model_params
|
||||
assert "ratio_wise" in sim.model_params
|
||||
assert "network_generator" in sim.model_params
|
||||
assert "network_params" in sim.model_params
|
||||
assert "prob_neighbor_spread" in sim.model_params
|
||||
assert sim.parameters
|
||||
assert "ratio_dumb" in sim.parameters
|
||||
assert "ratio_herd" in sim.parameters
|
||||
assert "ratio_wise" in sim.parameters
|
||||
assert "network_generator" in sim.parameters
|
||||
assert "network_params" in sim.parameters
|
||||
assert "prob_neighbor_spread" in sim.parameters
|
||||
|
||||
def test_config_matrix(self):
|
||||
"""It should be possible to specify a matrix of parameters"""
|
||||
a = [1, 2]
|
||||
b = [3, 4]
|
||||
sim = simulation.Simulation(matrix=dict(a=a, b=b))
|
||||
configs = sim._collect_params()
|
||||
assert len(configs) == len(a) * len(b)
|
||||
for i in a:
|
||||
for j in b:
|
||||
assert {"a": i, "b": j} in configs
|
||||
Loading…
Reference in New Issue
Block a user