Improved docs

Fixed several bugs
Added convenience methods in soil.analysis

Dockerfile

@@ -0,0 +1,3 @@
+FROM python:3.4-onbuild
+
+ENTRYPOINT ["python", "-m", "soil"]

README.md

@@ -3,7 +3,7 @@
 Soil is an extensible and user-friendly Agent-based Social Simulator for Social Networks.
 Learn how to run your own simulations with our [documentation](http://soilsim.readthedocs.io).
 
-Follow our [tutorial](notebooks/soil_tutorial.ipynb) to develop your own agent models.
+Follow our [tutorial](examples/tutorial/soil_tutorial.ipynb) to develop your own agent models.
 
 If you use Soil in your research, don't forget to cite this paper:
 

docs/index.rst

@@ -34,13 +34,14 @@ If you use Soil in your research, do not forget to cite this paper:
 
 
 .. toctree::
-   :maxdepth: 2
+   :maxdepth: 0
    :caption: Learn more about soil:
 
    installation
    quickstart
-   Tutorial - Spreading news
+   Tutorial <soil_tutorial>
 
+..
 
 
 .. Indices and tables

docs/installation.rst

@@ -1,7 +1,7 @@
 Installation
 ------------
 
-The easiest way to install Soil is through pip:
+The easiest way to install Soil is through pip, with Python >= 3.4:
 
 .. code:: bash
 

docs/quickstart.rst

@@ -13,7 +13,7 @@ Here's an example (``example.yml``).
     name: MyExampleSimulation
     max_time: 50
     num_trials: 3
-    timeout: 2
+    interval: 2
     network_params:
         network_type: barabasi_albert_graph
         n: 100
@@ -34,6 +34,12 @@ Here's an example (``example.yml``).
     environment_params:
         prob_infect: 0.075
 
+
+This example configuration will run three trials of a simulation containing a randomly generated network.
+The 100 nodes in the network will be SISaModel agents, 10% of them will start in the content state, 10% in the discontent state, and the remaining 80% in the neutral state.
+All agents will have access to the environment, which only contains one variable, ``prob_infected``.
+The state of the agents will be updated every 2 seconds (``interval``).
+
 Now run the simulation with the command line tool:
 
 .. code:: bash
@@ -41,7 +47,7 @@ Now run the simulation with the command line tool:
    soil example.yml
 
 Once the simulation finishes, its results will be stored in a folder named ``MyExampleSimulation``.
-Four types of objects are saved by default: a pickle of the simulation, a ``YAML`` representation of the simulation (to re-launch it), for every trial, a csv file with the content of the state of every network node and the environment parameters at every step of the simulation as well as the network in gephi format (``gexf``).
+Four types of objects are saved by default: a pickle of the simulation; a ``YAML`` representation of the simulation (which can be used to re-launch it); and for every trial, a csv file with the content of the state of every network node and the environment parameters at every step of the simulation, as well as the network in gephi format (``gexf``).
 
 
 .. code::
@@ -54,12 +60,6 @@ Four types of objects are saved by default: a pickle of the simulation, a ``YAML
     │   └── Sim_prob_0_trial_0.gexf
 
 
-This example configuration will run three trials of a simulation containing a randomly generated network.
-The 100 nodes in the network will be SISaModel agents, 10% of them will start in the content state, 10% in the discontent state, and the remaining 80% in the neutral state.
-All agents will have access to the environment, which only contains one variable, ``prob_infected``.
-The state of the agents will be updated every 2 seconds (``timeout``).
-
-
 Network
 =======
 
@@ -94,7 +94,7 @@ For example, the following configuration is equivalent to :code:`nx.complete_gra
 Environment
 ============
 The environment is the place where the shared state of the simulation is stored.
-For instance, the probability of certain events.
+For instance, the probability of disease outbreak.
 The configuration file may specify the initial value of the environment parameters:
 
 .. code:: yaml
@@ -103,14 +103,17 @@ The configuration file may specify the initial value of the environment paramete
         daily_probability_of_earthquake: 0.001
         number_of_earthquakes: 0
 
+Any agent has unrestricted access to the environment.
+However, for the sake of simplicity, we recommend limiting environment updates to environment agents.
+
 Agents
 ======
 Agents are a way of modelling behavior.
 Agents can be characterized with two variables: an agent type (``agent_type``) and its state.
-Only one agent is executed at a time (generally, every ``timeout`` seconds), and it has access to its state and the environment parameters.
+Only one agent is executed at a time (generally, every ``interval`` seconds), and it has access to its state and the environment parameters.
 Through the environment, it can access the network topology and the state of other agents.
 
-There are three three types of agents according to how they are added to the simulation: network agents, environment agent, and other agents.
+There are three three types of agents according to how they are added to the simulation: network agents and environment agent.
 
 Network Agents
 ##############
@@ -118,13 +121,13 @@ Network agents are attached to a node in the topology.
 The configuration file allows you to specify how agents will be mapped to topology nodes.
 
 The simplest way is to specify a single type of agent.
-Hence, every node in the network will have an associated agent of that type.
+Hence, every node in the network will be associated to an agent of that type.
 
 .. code:: yaml
 
    agent_type: SISaModel
 
-It is also possible to add more than one type of agent to the simulation, and to control the ratio of each type (``weight``).
+It is also possible to add more than one type of agent to the simulation, and to control the ratio of each type (using the ``weight`` property).
 For instance, with following configuration, it is five times more likely for a node to be assigned a CounterModel type than a SISaModel type.
 
 .. code:: yaml

examples/complete.yml

@@ -4,6 +4,8 @@ dir_path: "/tmp/"
 num_trials: 3
 max_time: 100
 interval: 1
+seed: "CompleteSeed!"
+dump: false
 network_params:
   generator: complete_graph
   n: 10
@@ -21,4 +23,4 @@ default_state:
   incidents: 0
 states:
   - name: 'The first node'
-  - name: 'The second node'
+  - name: 'The second node'

examples/custom-agents/UnnamedSimulation.dumped.yml

@@ -1,17 +0,0 @@
-default_state: {}
-environment_agents: []
-environment_params: {prob_neighbor_spread: 0.0, prob_tv_spread: 0.01}
-interval: 1
-max_time: 20
-name: Sim_prob_0
-network_agents:
-- agent_type: NewsSpread
-  state: {has_tv: false}
-  weight: 1
-- agent_type: NewsSpread
-  state: {has_tv: true}
-  weight: 2
-network_params: {generator: erdos_renyi_graph, n: 500, p: 0.1}
-num_trials: 1
-states:
-- {has_tv: true}

examples/custom-agents/agent.py

@@ -1,20 +0,0 @@
-import soil
-import random
-
-class NewsSpread(soil.agents.FSM):
-    @soil.agents.default_state
-    @soil.agents.state
-    def neutral(self):
-        r = random.random()
-        if self['has_tv'] and r < self.env['prob_tv_spread']:
-                return self.infected
-        return
-
-    @soil.agents.state
-    def infected(self):
-        prob_infect = self.env['prob_neighbor_spread']
-        for neighbor in self.get_neighboring_agents(state_id=self.neutral.id):
-            r = random.random()
-            if r < prob_infect:
-                neighbor.state['id'] = self.infected.id
-        return

examples/newsspread/NewsSpread.yml

@@ -0,0 +1,138 @@
+---
+default_state: {}
+load_module: newsspread
+environment_agents: []
+environment_params:
+  prob_neighbor_spread: 0.0
+  prob_tv_spread: 0.01
+interval: 1
+max_time: 30
+name: Sim_all_dumb
+network_agents:
+- agent_type: DumbViewer
+  state:
+    has_tv: false
+  weight: 1
+- agent_type: DumbViewer
+  state:
+    has_tv: true
+  weight: 1
+network_params:
+  generator: barabasi_albert_graph
+  n: 500
+  m: 5
+num_trials: 50
+---
+default_state: {}
+load_module: newsspread
+environment_agents: []
+environment_params:
+  prob_neighbor_spread: 0.0
+  prob_tv_spread: 0.01
+interval: 1
+max_time: 30
+name: Sim_half_herd
+network_agents:
+- agent_type: DumbViewer
+  state:
+    has_tv: false
+  weight: 1
+- agent_type: DumbViewer
+  state:
+    has_tv: true
+  weight: 1
+- agent_type: HerdViewer
+  state:
+    has_tv: false
+  weight: 1
+- agent_type: HerdViewer
+  state:
+    has_tv: true
+  weight: 1
+network_params:
+  generator: barabasi_albert_graph
+  n: 500
+  m: 5
+num_trials: 50
+---
+default_state: {}
+load_module: newsspread
+environment_agents: []
+environment_params:
+  prob_neighbor_spread: 0.0
+  prob_tv_spread: 0.01
+interval: 1
+max_time: 30
+name: Sim_all_herd
+network_agents:
+- agent_type: HerdViewer
+  state:
+    has_tv: true
+    id: infected
+  weight: 1
+- agent_type: HerdViewer
+  state:
+    has_tv: true
+    id: neutral
+  weight: 1
+network_params:
+  generator: barabasi_albert_graph
+  n: 500
+  m: 5
+num_trials: 50
+---
+default_state: {}
+load_module: newsspread
+environment_agents: []
+environment_params:
+  prob_neighbor_spread: 0.0
+  prob_tv_spread: 0.01
+  prob_neighbor_cure: 0.1
+interval: 1
+max_time: 30
+name: Sim_wise_herd
+network_agents:
+- agent_type: HerdViewer
+  state:
+    has_tv: true
+    id: infected
+  weight: 1
+- agent_type: WiseViewer
+  state:
+    has_tv: true
+  weight: 1
+network_params:
+  generator: barabasi_albert_graph
+  n: 500
+  m: 5
+num_trials: 50
+---
+default_state: {}
+load_module: newsspread
+environment_agents: []
+environment_params:
+  prob_neighbor_spread: 0.0
+  prob_tv_spread: 0.01
+  prob_neighbor_cure: 0.1
+interval: 1
+max_time: 30
+name: Sim_all_wise
+network_agents:
+- agent_type: WiseViewer
+  state:
+    has_tv: true
+    id: infected
+  weight: 1
+- agent_type: WiseViewer
+  state:
+    has_tv: true
+  weight: 1
+network_params:
+  generator: barabasi_albert_graph
+  n: 500
+  m: 5
+network_params:
+  generator: barabasi_albert_graph
+  n: 500
+  m: 5
+num_trials: 50

examples/newsspread/newsspread.py

@@ -0,0 +1,79 @@
+from soil.agents import BaseAgent,FSM, state, default_state
+import random
+import logging
+
+
+class DumbViewer(FSM):
+    '''
+    A viewer that gets infected via TV (if it has one) and tries to infect
+    its neighbors once it's infected.
+    '''
+    defaults = {
+        'prob_neighbor_spread': 0.5,
+        'prob_neighbor_cure': 0.25,
+    }
+
+    @default_state
+    @state
+    def neutral(self):
+        r = random.random()
+        if self['has_tv'] and r < self.env['prob_tv_spread']:
+            self.infect()
+        return
+
+    @state
+    def infected(self):
+        for neighbor in self.get_neighboring_agents(state_id=self.neutral.id):
+            prob_infect = self.env['prob_neighbor_spread']
+            r = random.random()
+            if r < prob_infect:
+                self.set_state(self.infected.id)
+                neighbor.infect()
+        return
+
+    def infect(self):
+        self.set_state(self.infected)
+
+class HerdViewer(DumbViewer):
+    '''
+    A viewer whose probability of infection depends on the state of its neighbors.
+    '''
+
+    level = logging.DEBUG
+ 
+    def infect(self):
+        infected = self.count_neighboring_agents(state_id=self.infected.id)
+        total = self.count_neighboring_agents()
+        prob_infect = self.env['prob_neighbor_spread'] * infected/total
+        self.debug('prob_infect', prob_infect)
+        r = random.random()
+        if r < prob_infect:
+            self.set_state(self.infected.id)
+
+class WiseViewer(HerdViewer):
+    '''
+    A viewer that can change its mind.
+    '''
+    @state
+    def cured(self):
+        prob_cure = self.env['prob_neighbor_cure']
+        for neighbor in self.get_neighboring_agents(state_id=self.infected.id):
+            r = random.random()
+            if r < prob_cure:
+                try:
+                    neighbor.cure()
+                except AttributeError:
+                    self.debug('Viewer {} cannot be cured'.format(neighbor.id))
+        return
+
+    def cure(self):
+        self.set_state(self.cured.id)
+
+    @state
+    def infected(self):
+        prob_cure = self.env['prob_neighbor_cure']
+        r = random.random()
+        if r < prob_cure:
+            self.cure()
+            return
+        return super().infected()

examples/rabbits/rabbit_agents.py

@@ -0,0 +1,120 @@
+from soil.agents import FSM, state, default_state, BaseAgent
+from enum import Enum
+from random import random, choice
+from itertools import islice
+import logging
+import math
+
+
+class Genders(Enum):
+    male = 'male'
+    female = 'female'
+
+
+class RabbitModel(FSM):
+
+    level = logging.INFO
+
+    defaults = {
+        'age': 0,
+        'gender': Genders.male.value,
+        'mating_prob': 0.001,
+        'offspring': 0,
+    }
+
+    sexual_maturity = 4*30
+    life_expectancy = 365 * 3
+    gestation = 33
+    pregnancy = -1
+    max_females = 5
+
+    @default_state
+    @state
+    def newborn(self):
+        self['age'] += 1
+
+        if self['age'] >= self.sexual_maturity:
+            return self.fertile
+
+    @state
+    def fertile(self):
+        self['age'] += 1
+        if self['age'] > self.life_expectancy:
+            return self.dead
+
+        if self['gender'] == Genders.female.value:
+            return
+
+        # Males try to mate
+        females = self.get_agents(state_id=self.fertile.id, gender=Genders.female.value, limit_neighbors=False)
+        for f in islice(females, self.max_females):
+            r = random()
+            if r < self['mating_prob']:
+                self.impregnate(f)
+                break # Take a break
+
+    def impregnate(self, whom):
+        if self['gender'] == Genders.female.value:
+            raise NotImplementedError('Females cannot impregnate')
+        whom['pregnancy'] = 0
+        whom['mate'] = self.id
+        whom.set_state(whom.pregnant)
+        self.debug('{} impregnating: {}. {}'.format(self.id, whom.id, whom.state))
+
+    @state
+    def pregnant(self):
+        self['age'] += 1
+        if self['age'] > self.life_expectancy:
+            return self.dead
+
+        self['pregnancy'] += 1
+        self.debug('Pregnancy: {}'.format(self['pregnancy']))
+        if self['pregnancy'] >= self.gestation:
+            number_of_babies = int(8+4*random())
+            self.info('Having {} babies'.format(number_of_babies))
+            for i in range(number_of_babies):
+                state = {}
+                state['gender'] = choice(list(Genders)).value
+                child = self.env.add_node(self.__class__, state)
+                self.env.add_edge(self.id, child.id)
+                self.env.add_edge(self['mate'], child.id)
+                # self.add_edge()
+                self.debug('A BABY IS COMING TO LIFE')
+                self.env['rabbits_alive'] = self.env.get('rabbits_alive', self.global_topology.number_of_nodes())+1
+                self.debug('Rabbits alive: {}'.format(self.env['rabbits_alive']))
+                self['offspring'] += 1
+                self.env.get_agent(self['mate'])['offspring'] += 1
+            del self['mate']
+            self['pregnancy'] = -1
+            return self.fertile
+
+    @state
+    def dead(self):
+        self.info('Agent {} is dying'.format(self.id))
+        if 'pregnancy' in self and self['pregnancy'] > -1:
+            self.info('A mother has died carrying a baby!!')
+        self.die()
+        return
+
+
+class RandomAccident(BaseAgent):
+
+    level = logging.DEBUG
+
+    def step(self):
+        rabbits_total = self.global_topology.number_of_nodes()
+        rabbits_alive = self.env.get('rabbits_alive', rabbits_total)
+        prob_death = self.env.get('prob_death', 1e-100)*math.floor(math.log10(max(1, rabbits_alive)))
+        self.debug('Killing some rabbits with prob={}!'.format(prob_death))
+        for i in self.env.network_agents:
+            if i.state['id'] == i.dead.id:
+                continue
+            r = random()
+            if r < prob_death:
+                self.debug('I killed a rabbit: {}'.format(i.id))
+                rabbits_alive = self.env['rabbits_alive'] = rabbits_alive -1
+                self.log('Rabbits alive: {}'.format(self.env['rabbits_alive']))
+                i.set_state(i.dead)
+        self.log('Rabbits alive: {}/{}'.format(rabbits_alive, rabbits_total))
+        if self.count_agents(state_id=RabbitModel.dead.id) == self.global_topology.number_of_nodes():
+            self.die()

examples/rabbits/rabbits.yml

@@ -0,0 +1,23 @@
+---
+load_module: rabbit_agents
+name: rabbits_example
+max_time: 1200
+interval: 1
+seed: MySeed
+agent_type: RabbitModel
+environment_agents:
+    - agent_type: RandomAccident
+environment_params:
+  prob_death: 0.001
+default_state:
+  mating_prob: 0.01
+topology:
+  nodes:
+    - id: 1
+      state:
+        gender: female
+    - id: 0
+      state:
+        gender: male
+  directed: true
+  links: []

examples/torvalds.yml

@@ -1,6 +1,6 @@
 ---
 name: torvalds_example
-max_time: 1
+max_time: 10
 interval: 2
 agent_type: CounterModel
 default_state:
@@ -11,4 +11,4 @@ states:
   Torvalds:
     skill_level: 'God'
   balkian:
-    skill_level: 'developer'
+    skill_level: 'developer'

models_org.py

@@ -1,596 +0,0 @@
-from nxsim import BaseNetworkAgent
-import numpy as np
-import random
-import settings
-
-settings.init()
-
-##############################
-# Variables initialization #
-##############################
-def init():
-    global networkStatus
-    networkStatus = {}  # Dict that will contain the status of every agent in the network
-
-sentimentCorrelationNodeArray=[]
-for x in range(0, settings.number_of_nodes):
-    sentimentCorrelationNodeArray.append({'id':x})
-# Initialize agent states. Let's assume everyone is normal.
-init_states = [{'id': 0, } for _ in range(settings.number_of_nodes)]  # add keys as as necessary, but "id" must always refer to that state category
-
-
-####################
-# Available models #
-####################
-
-class BaseBehaviour(BaseNetworkAgent):
-    def __init__(self, environment=None, agent_id=0, state=()):
-        super().__init__(environment=environment, agent_id=agent_id, state=state)
-        self._attrs = {}
-
-    @property
-    def attrs(self):
-        now = self.env.now
-        if now not in self._attrs:
-            self._attrs[now] = {}
-        return self._attrs[now]
-
-    @attrs.setter
-    def attrs(self, value):
-        self._attrs[self.env.now] = value
-
-    def run(self):
-        while True:
-            self.step(self.env.now)
-            yield self.env.timeout(settings.timeout)
-
-    def step(self, now):
-        networkStatus['agent_%s'% self.id] = self.to_json()
-
-    def to_json(self):
-        final = {}
-        for stamp, attrs in self._attrs.items():
-            for a in attrs:
-                if a not in final:
-                   final[a] = {}
-                final[a][stamp] = attrs[a]
-        return final
-
-class ControlModelM2(BaseBehaviour):
-    #Init infected
-    init_states[random.randint(0,settings.number_of_nodes-1)] = {'id':1}
-    init_states[random.randint(0,settings.number_of_nodes-1)] = {'id':1}
-
-    # Init beacons
-    init_states[random.randint(0, settings.number_of_nodes-1)] = {'id': 4}
-    init_states[random.randint(0, settings.number_of_nodes-1)] = {'id': 4}
-    def __init__(self, environment=None, agent_id=0, state=()):
-        super().__init__(environment=environment, agent_id=agent_id, state=state)
-
-        self.prob_neutral_making_denier = np.random.normal(settings.prob_neutral_making_denier, settings.standard_variance)
-
-        self.prob_infect = np.random.normal(settings.prob_infect, settings.standard_variance)
-
-        self.prob_cured_healing_infected = np.random.normal(settings.prob_cured_healing_infected, settings.standard_variance)
-        self.prob_cured_vaccinate_neutral = np.random.normal(settings.prob_cured_vaccinate_neutral, settings.standard_variance)
-
-        self.prob_vaccinated_healing_infected = np.random.normal(settings.prob_vaccinated_healing_infected, settings.standard_variance)
-        self.prob_vaccinated_vaccinate_neutral = np.random.normal(settings.prob_vaccinated_vaccinate_neutral, settings.standard_variance)
-        self.prob_generate_anti_rumor = np.random.normal(settings.prob_generate_anti_rumor, settings.standard_variance)
-
-    def step(self, now):
-
-        if self.state['id'] == 0:  #Neutral
-            self.neutral_behaviour()
-        elif self.state['id'] == 1:  #Infected
-            self.infected_behaviour()
-        elif self.state['id'] == 2:  #Cured
-            self.cured_behaviour()
-        elif self.state['id'] == 3:  #Vaccinated
-            self.vaccinated_behaviour()
-        elif self.state['id'] == 4:  #Beacon-off
-            self.beacon_off_behaviour()
-        elif self.state['id'] == 5:  #Beacon-on
-            self.beacon_on_behaviour()
-
-        self.attrs['status'] = self.state['id']
-        super().step(now)
-
-
-    def neutral_behaviour(self):
-
-        # Infected
-        infected_neighbors = self.get_neighboring_agents(state_id=1)
-        if len(infected_neighbors)>0:
-            if random.random() < self.prob_neutral_making_denier:
-                self.state['id'] = 3   # Vaccinated making denier
-
-    def infected_behaviour(self):
-
-        # Neutral
-        neutral_neighbors = self.get_neighboring_agents(state_id=0)
-        for neighbor in neutral_neighbors:
-            if random.random() < self.prob_infect:
-                neighbor.state['id'] = 1  # Infected
-
-    def cured_behaviour(self):
-
-        # Vaccinate
-        neutral_neighbors = self.get_neighboring_agents(state_id=0)
-        for neighbor in neutral_neighbors:
-            if random.random() < self.prob_cured_vaccinate_neutral:
-                neighbor.state['id'] = 3  # Vaccinated
-
-        # Cure
-        infected_neighbors = self.get_neighboring_agents(state_id=1)
-        for neighbor in infected_neighbors:
-            if random.random() < self.prob_cured_healing_infected:
-                neighbor.state['id'] = 2  # Cured
-
-
-    def vaccinated_behaviour(self):
-
-        # Cure
-        infected_neighbors = self.get_neighboring_agents(state_id=1)
-        for neighbor in infected_neighbors:
-            if random.random() < self.prob_cured_healing_infected:
-                neighbor.state['id'] = 2  # Cured
-
-
-        # Vaccinate
-        neutral_neighbors = self.get_neighboring_agents(state_id=0)
-        for neighbor in neutral_neighbors:
-            if random.random() < self.prob_cured_vaccinate_neutral:
-                neighbor.state['id'] = 3  # Vaccinated
-
-        # Generate anti-rumor
-        infected_neighbors_2 = self.get_neighboring_agents(state_id=1)
-        for neighbor in infected_neighbors_2:
-            if random.random() < self.prob_generate_anti_rumor:
-                neighbor.state['id'] = 2  # Cured
-
-    def beacon_off_behaviour(self):
-        infected_neighbors = self.get_neighboring_agents(state_id=1)
-        if len(infected_neighbors) > 0:
-            self.state['id'] == 5  #Beacon on
-
-    def beacon_on_behaviour(self):
-
-        # Cure (M2 feature added)
-        infected_neighbors = self.get_neighboring_agents(state_id=1)
-        for neighbor in infected_neighbors:
-            if random.random() < self.prob_generate_anti_rumor:
-                neighbor.state['id'] = 2  # Cured
-            neutral_neighbors_infected = neighbor.get_neighboring_agents(state_id=0)
-            for neighbor in neutral_neighbors_infected:
-                if random.random() < self.prob_generate_anti_rumor:
-                    neighbor.state['id'] = 3  # Vaccinated
-            infected_neighbors_infected = neighbor.get_neighboring_agents(state_id=1)
-            for neighbor in infected_neighbors_infected:
-                if random.random() < self.prob_generate_anti_rumor:
-                    neighbor.state['id'] = 2  # Cured
-
-
-        # Vaccinate
-        neutral_neighbors = self.get_neighboring_agents(state_id=0)
-        for neighbor in neutral_neighbors:
-            if random.random() < self.prob_cured_vaccinate_neutral:
-                neighbor.state['id'] = 3  # Vaccinated
-
-
-class SpreadModelM2(BaseBehaviour):
-    init_states[random.randint(0,settings.number_of_nodes)] = {'id':1}
-    init_states[random.randint(0,settings.number_of_nodes)] = {'id':1}
-    def __init__(self, environment=None, agent_id=0, state=()):
-        super().__init__(environment=environment, agent_id=agent_id, state=state)
-
-        self.prob_neutral_making_denier = np.random.normal(settings.prob_neutral_making_denier, settings.standard_variance)
-
-        self.prob_infect = np.random.normal(settings.prob_infect, settings.standard_variance)
-
-        self.prob_cured_healing_infected = np.random.normal(settings.prob_cured_healing_infected, settings.standard_variance)
-        self.prob_cured_vaccinate_neutral = np.random.normal(settings.prob_cured_vaccinate_neutral, settings.standard_variance)
-
-        self.prob_vaccinated_healing_infected = np.random.normal(settings.prob_vaccinated_healing_infected, settings.standard_variance)
-        self.prob_vaccinated_vaccinate_neutral = np.random.normal(settings.prob_vaccinated_vaccinate_neutral, settings.standard_variance)
-        self.prob_generate_anti_rumor = np.random.normal(settings.prob_generate_anti_rumor, settings.standard_variance)
-
-    def step(self, now):
-
-        if self.state['id'] == 0:  #Neutral
-            self.neutral_behaviour()
-        elif self.state['id'] == 1:  #Infected
-            self.infected_behaviour()
-        elif self.state['id'] == 2:  #Cured
-            self.cured_behaviour()
-        elif self.state['id'] == 3:  #Vaccinated
-            self.vaccinated_behaviour()
-
-        self.attrs['status'] = self.state['id']
-        super().step(now)
-
-
-    def neutral_behaviour(self):
-
-        # Infected
-        infected_neighbors = self.get_neighboring_agents(state_id=1)
-        if len(infected_neighbors)>0:
-            if random.random() < self.prob_neutral_making_denier:
-                self.state['id'] = 3   # Vaccinated making denier
-
-    def infected_behaviour(self):
-
-        # Neutral
-        neutral_neighbors = self.get_neighboring_agents(state_id=0)
-        for neighbor in neutral_neighbors:
-            if random.random() < self.prob_infect:
-                neighbor.state['id'] = 1  # Infected
-
-    def cured_behaviour(self):
-
-        # Vaccinate
-        neutral_neighbors = self.get_neighboring_agents(state_id=0)
-        for neighbor in neutral_neighbors:
-            if random.random() < self.prob_cured_vaccinate_neutral:
-                neighbor.state['id'] = 3  # Vaccinated
-
-        # Cure
-        infected_neighbors = self.get_neighboring_agents(state_id=1)
-        for neighbor in infected_neighbors:
-            if random.random() < self.prob_cured_healing_infected:
-                neighbor.state['id'] = 2  # Cured
-
-
-    def vaccinated_behaviour(self):
-
-        # Cure
-        infected_neighbors = self.get_neighboring_agents(state_id=1)
-        for neighbor in infected_neighbors:
-            if random.random() < self.prob_cured_healing_infected:
-                neighbor.state['id'] = 2  # Cured
-
-
-        # Vaccinate
-        neutral_neighbors = self.get_neighboring_agents(state_id=0)
-        for neighbor in neutral_neighbors:
-            if random.random() < self.prob_cured_vaccinate_neutral:
-                neighbor.state['id'] = 3  # Vaccinated
-
-        # Generate anti-rumor
-        infected_neighbors_2 = self.get_neighboring_agents(state_id=1)
-        for neighbor in infected_neighbors_2:
-            if random.random() < self.prob_generate_anti_rumor:
-                neighbor.state['id'] = 2  # Cured
-
-
-class SISaModel(BaseBehaviour):
-    def __init__(self, environment=None, agent_id=0, state=()):
-        super().__init__(environment=environment, agent_id=agent_id, state=state)
-
-        self.neutral_discontent_spon_prob = np.random.normal(settings.neutral_discontent_spon_prob, settings.standard_variance)
-        self.neutral_discontent_infected_prob = np.random.normal(settings.neutral_discontent_infected_prob,settings.standard_variance)
-        self.neutral_content_spon_prob = np.random.normal(settings.neutral_content_spon_prob,settings.standard_variance)
-        self.neutral_content_infected_prob = np.random.normal(settings.neutral_content_infected_prob,settings.standard_variance)
-
-        self.discontent_neutral = np.random.normal(settings.discontent_neutral,settings.standard_variance)
-        self.discontent_content = np.random.normal(settings.discontent_content,settings.variance_d_c)
-
-        self.content_discontent = np.random.normal(settings.content_discontent,settings.variance_c_d)
-        self.content_neutral = np.random.normal(settings.content_neutral,settings.standard_variance)
-
-    def step(self, now):
-
-        if self.state['id'] == 0:
-            self.neutral_behaviour()
-        if self.state['id'] == 1:
-            self.discontent_behaviour()
-        if self.state['id'] == 2:
-            self.content_behaviour()
-
-        self.attrs['status'] = self.state['id']
-        super().step(now)
-
-
-    def neutral_behaviour(self):
-
-        #Spontaneus effects
-        if random.random() < self.neutral_discontent_spon_prob:
-            self.state['id'] = 1
-        if random.random() < self.neutral_content_spon_prob:
-            self.state['id'] = 2
-
-        #Infected
-        discontent_neighbors = self.get_neighboring_agents(state_id=1)
-        if random.random() < len(discontent_neighbors)*self.neutral_discontent_infected_prob:
-            self.state['id'] = 1
-        content_neighbors = self.get_neighboring_agents(state_id=2)
-        if random.random() < len(content_neighbors)*self.neutral_content_infected_prob:
-            self.state['id'] = 2
-
-    def discontent_behaviour(self):
-
-        #Healing
-        if random.random() < self.discontent_neutral:
-            self.state['id'] = 0
-
-        #Superinfected
-        content_neighbors = self.get_neighboring_agents(state_id=2)
-        if random.random() < len(content_neighbors)*self.discontent_content:
-            self.state['id'] = 2
-
-    def content_behaviour(self):
-
-        #Healing
-        if random.random() < self.content_neutral:
-            self.state['id'] = 0
-
-        #Superinfected
-        discontent_neighbors = self.get_neighboring_agents(state_id=1)
-        if random.random() < len(discontent_neighbors)*self.content_discontent:
-            self.state['id'] = 1
-
-
-class BigMarketModel(BaseBehaviour):
-
-    def __init__(self, environment=None, agent_id=0, state=()):
-        super().__init__(environment=environment, agent_id=agent_id, state=state)
-        self.enterprises = settings.enterprises
-        self.type = ""
-        self.number_of_enterprises = len(settings.enterprises)
-
-        if self.id < self.number_of_enterprises: #Enterprises
-            self.state['id']=self.id
-            self.type="Enterprise"
-            self.tweet_probability = settings.tweet_probability_enterprises[self.id]
-        else:                       #normal users
-            self.state['id']=self.number_of_enterprises
-            self.type="User"
-            self.tweet_probability = settings.tweet_probability_users
-            self.tweet_relevant_probability = settings.tweet_relevant_probability
-            self.tweet_probability_about = settings.tweet_probability_about #List
-            self.sentiment_about = settings.sentiment_about #List
-
-    def step(self, now):
-
-        if(self.id < self.number_of_enterprises): # Ennterprise
-            self.enterpriseBehaviour()
-        else:  # Usuario
-            self.userBehaviour()
-            for i in range(self.number_of_enterprises):       # So that it never is set to 0 if there are not changes (logs)
-                self.attrs['sentiment_enterprise_%s'% self.enterprises[i]] = self.sentiment_about[i]
-
-        super().step(now)
-
-    def enterpriseBehaviour(self):
-
-        if random.random()< self.tweet_probability: #Tweets
-            aware_neighbors = self.get_neighboring_agents(state_id=self.number_of_enterprises) #Nodes neighbour users
-            for x in aware_neighbors:
-                if random.uniform(0,10) < 5:
-                    x.sentiment_about[self.id] += 0.1 #Increments for enterprise
-                else:
-                    x.sentiment_about[self.id] -= 0.1 #Decrements for enterprise
-
-                # Establecemos limites
-                if x.sentiment_about[self.id] > 1:
-                    x.sentiment_about[self.id] = 1
-                if x.sentiment_about[self.id]< -1:
-                    x.sentiment_about[self.id] = -1
-
-                x.attrs['sentiment_enterprise_%s'% self.enterprises[self.id]] = x.sentiment_about[self.id]
-
-
-    def userBehaviour(self):
-
-        if random.random() < self.tweet_probability: #Tweets
-            if random.random() < self.tweet_relevant_probability: #Tweets something relevant
-                #Tweet probability per enterprise
-                for i in range(self.number_of_enterprises):
-                    random_num = random.random()
-                    if random_num < self.tweet_probability_about[i]:
-                        #The condition is fulfilled, sentiments are evaluated towards that enterprise
-                        if self.sentiment_about[i] < 0:
-                            #NEGATIVO
-                            self.userTweets("negative",i)
-                        elif self.sentiment_about[i] == 0:
-                            #NEUTRO
-                            pass
-                        else:
-                            #POSITIVO
-                            self.userTweets("positive",i)
-
-    def userTweets(self,sentiment,enterprise):
-        aware_neighbors = self.get_neighboring_agents(state_id=self.number_of_enterprises) #Nodes neighbours users
-        for x in aware_neighbors:
-            if sentiment == "positive":
-                x.sentiment_about[enterprise] +=0.003
-            elif sentiment == "negative":
-                x.sentiment_about[enterprise] -=0.003
-            else:
-                pass
-
-            # Establecemos limites
-            if x.sentiment_about[enterprise] > 1:
-                x.sentiment_about[enterprise] = 1
-            if x.sentiment_about[enterprise] < -1:
-                x.sentiment_about[enterprise] = -1
-
-            x.attrs['sentiment_enterprise_%s'% self.enterprises[enterprise]] = x.sentiment_about[enterprise]
-
-class SentimentCorrelationModel(BaseBehaviour):
-
-    def __init__(self, environment=None, agent_id=0, state=()):
-        super().__init__(environment=environment, agent_id=agent_id, state=state)
-        self.outside_effects_prob = settings.outside_effects_prob
-        self.anger_prob = settings.anger_prob
-        self.joy_prob = settings.joy_prob
-        self.sadness_prob = settings.sadness_prob
-        self.disgust_prob = settings.disgust_prob
-        self.time_awareness=[]
-        for i in range(4):  #In this model we have 4 sentiments
-            self.time_awareness.append(0)     #0-> Anger, 1-> joy, 2->sadness, 3 -> disgust
-        sentimentCorrelationNodeArray[self.id][self.env.now]=0
-
-
-    def step(self, now):
-        self.behaviour()
-        super().step(now)
-
-    def behaviour(self):
-
-        angry_neighbors_1_time_step=[]
-        joyful_neighbors_1_time_step=[]
-        sad_neighbors_1_time_step=[]
-        disgusted_neighbors_1_time_step=[]
-
-
-        angry_neighbors = self.get_neighboring_agents(state_id=1)
-        for x in angry_neighbors:
-            if x.time_awareness[0] > (self.env.now-500):
-                angry_neighbors_1_time_step.append(x)
-        num_neighbors_angry = len(angry_neighbors_1_time_step)
-
-
-        joyful_neighbors = self.get_neighboring_agents(state_id=2)
-        for x in joyful_neighbors:
-            if x.time_awareness[1] > (self.env.now-500):
-                joyful_neighbors_1_time_step.append(x)
-        num_neighbors_joyful = len(joyful_neighbors_1_time_step)
-
-
-        sad_neighbors = self.get_neighboring_agents(state_id=3)
-        for x in sad_neighbors:
-            if x.time_awareness[2] > (self.env.now-500):
-                sad_neighbors_1_time_step.append(x)
-        num_neighbors_sad = len(sad_neighbors_1_time_step)
-
-
-        disgusted_neighbors = self.get_neighboring_agents(state_id=4)
-        for x in disgusted_neighbors:
-            if x.time_awareness[3] > (self.env.now-500):
-                disgusted_neighbors_1_time_step.append(x)
-        num_neighbors_disgusted = len(disgusted_neighbors_1_time_step)
-
-
-        anger_prob= settings.anger_prob+(len(angry_neighbors_1_time_step)*settings.anger_prob)
-        joy_prob= settings.joy_prob+(len(joyful_neighbors_1_time_step)*settings.joy_prob)
-        sadness_prob = settings.sadness_prob+(len(sad_neighbors_1_time_step)*settings.sadness_prob)
-        disgust_prob = settings.disgust_prob+(len(disgusted_neighbors_1_time_step)*settings.disgust_prob)
-        outside_effects_prob= settings.outside_effects_prob
-
-
-        num = random.random()
-
-
-        if(num<outside_effects_prob):
-            self.state['id'] = random.randint(1,4)
-
-            sentimentCorrelationNodeArray[self.id][self.env.now]=self.state['id'] #It is stored when it has been infected for the dynamic network
-            self.time_awareness[self.state['id']-1] = self.env.now
-            self.attrs['sentiment'] = self.state['id']
-
-
-
-        if(num<anger_prob):
-
-            self.state['id'] = 1
-            sentimentCorrelationNodeArray[self.id][self.env.now]=1
-            self.time_awareness[self.state['id']-1] = self.env.now
-        elif (num<joy_prob+anger_prob and num>anger_prob):
-
-            self.state['id'] = 2
-            sentimentCorrelationNodeArray[self.id][self.env.now]=2
-            self.time_awareness[self.state['id']-1] = self.env.now
-        elif (num<sadness_prob+anger_prob+joy_prob and num>joy_prob+anger_prob):
-
-
-            self.state['id'] = 3
-            sentimentCorrelationNodeArray[self.id][self.env.now]=3
-            self.time_awareness[self.state['id']-1] = self.env.now
-        elif (num<disgust_prob+sadness_prob+anger_prob+joy_prob and num>sadness_prob+anger_prob+joy_prob):
-
-
-            self.state['id'] = 4
-            sentimentCorrelationNodeArray[self.id][self.env.now]=4
-            self.time_awareness[self.state['id']-1] = self.env.now
-
-        self.attrs['sentiment'] = self.state['id']
-
-
-class BassModel(BaseBehaviour):
-    def __init__(self, environment=None, agent_id=0, state=()):
-        super().__init__(environment=environment, agent_id=agent_id, state=state)
-        self.innovation_prob = settings.innovation_prob
-        self.imitation_prob = settings.imitation_prob
-        sentimentCorrelationNodeArray[self.id][self.env.now]=0
-
-    def step(self, now):
-        self.behaviour()
-        super().step(now)
-
-    def behaviour(self):
-        #Outside effects
-        if random.random() < settings.innovation_prob:
-            if self.state['id'] == 0:
-                self.state['id'] = 1
-                sentimentCorrelationNodeArray[self.id][self.env.now]=1
-            else:
-                pass
-
-            self.attrs['status'] = self.state['id']
-            return
-
-        #Imitation effects
-        if self.state['id'] == 0:
-            aware_neighbors = self.get_neighboring_agents(state_id=1)
-            num_neighbors_aware = len(aware_neighbors)
-            if random.random() < (settings.imitation_prob*num_neighbors_aware):
-                self.state['id'] = 1
-                sentimentCorrelationNodeArray[self.id][self.env.now]=1
-
-            else:
-                pass
-            self.attrs['status'] = self.state['id']
-
-
-class IndependentCascadeModel(BaseBehaviour):
-    def __init__(self, environment=None, agent_id=0, state=()):
-        super().__init__(environment=environment, agent_id=agent_id, state=state)
-        self.innovation_prob = settings.innovation_prob
-        self.imitation_prob = settings.imitation_prob
-        self.time_awareness = 0
-        sentimentCorrelationNodeArray[self.id][self.env.now]=0
-
-    def step(self,now):
-        self.behaviour()
-        super().step(now)
-
-    def behaviour(self):
-        aware_neighbors_1_time_step=[]
-            #Outside effects
-        if random.random() < settings.innovation_prob:
-            if self.state['id'] == 0:
-                self.state['id'] = 1
-                sentimentCorrelationNodeArray[self.id][self.env.now]=1
-                self.time_awareness = self.env.now #To know when they have been infected
-            else:
-                pass
-
-            self.attrs['status'] = self.state['id']
-            return
-
-        #Imitation effects
-        if self.state['id'] == 0:
-            aware_neighbors = self.get_neighboring_agents(state_id=1)
-            for x in aware_neighbors:
-                if x.time_awareness == (self.env.now-1):
-                    aware_neighbors_1_time_step.append(x)
-            num_neighbors_aware = len(aware_neighbors_1_time_step)
-            if random.random() < (settings.imitation_prob*num_neighbors_aware):
-                self.state['id'] = 1
-                sentimentCorrelationNodeArray[self.id][self.env.now]=1
-            else:
-                pass
-
-            self.attrs['status'] = self.state['id']
-            return

setup.py

@@ -28,7 +28,16 @@ setup(
     download_url='https://github.com/gsi-upm/soil/archive/{}.tar.gz'.format(
         __version__),
     keywords=['agent', 'social', 'simulator'],
-    classifiers=[],
+    classifiers=[
+        'Development Status :: 5 - Production/Stable',
+        'Environment :: Console',
+        'Intended Audience :: End Users/Desktop',
+        'Intended Audience :: Developers',
+        'License :: OSI Approved :: Apache Software License',
+        'Operating System :: MacOS :: MacOS X',
+        'Operating System :: Microsoft :: Windows',
+        'Operating System :: POSIX',
+        'Programming Language :: Python :: 3'],
     install_requires=install_reqs,
     tests_require=test_reqs,
     setup_requires=['pytest-runner', ],

simulation.yml

@@ -2,12 +2,11 @@
 name: ControlModelM2_sim
 max_time: 50
 num_trials: 1
-timeout: 2
 network_params:
   generator: barabasi_albert_graph
   n: 100
   m: 2
-agent_distribution:
+network_agents:
   - agent_type: ControlModelM2
     weight: 0.1
     state:
@@ -29,14 +28,13 @@ environment_params:
 name: SISA_sm
 max_time: 50
 num_trials: 2
-timeout: 2
 network_params:
   generator: erdos_renyi_graph
-  n: 10000
+  n: 1000
   p: 0.05
 #other_agents:
 #  - agent_type: DrawingAgent
-agent_distribution:
+network_agents:
   - agent_type: SISaModel
     weight: 1
     state:

soil/__init__.py

@@ -1,19 +1,23 @@
 import importlib
 import sys
 import os
+import pdb
+import logging
 
-__version__ = "0.9.2"
+__version__ = "0.10"
 
 try:
     basestring
 except NameError:
     basestring = str
 
+logging.basicConfig()
+
 from . import agents
 from . import simulation
 from . import environment
 from . import utils
-from . import settings
+from . import analysis
 
 
 def main():
@@ -27,6 +31,12 @@ def main():
                         help='python module containing the simulation configuration.')
     parser.add_argument('--module', '-m', type=str,
                         help='file containing the code of any custom agents.')
+    parser.add_argument('--dry-run', '--dry', action='store_true',
+                        help='Do not store the results of the simulation.')
+    parser.add_argument('--pdb', action='store_true',
+                        help='Use a pdb console in case of exception.')
+    parser.add_argument('--output', '-o', type=str,
+                        help='folder to write results to. It defaults to the current directory.')
 
     args = parser.parse_args()
 
@@ -34,8 +44,15 @@ def main():
         sys.path.append(os.getcwd())
         importlib.import_module(args.module)
 
-    print('Loading config file: {}'.format(args.file))
-    simulation.run_from_config(args.file)
+    logging.info('Loading config file: {}'.format(args.file, args.output))
+
+    try:
+        simulation.run_from_config(args.file, dump=(not args.dry_run), results_dir=args.output)
+    except Exception as ex:
+        if args.pdb:
+            pdb.post_mortem()
+        else:
+            raise
 
 
 if __name__ == '__main__':

soil/__main__.py

@@ -0,0 +1,36 @@
+import importlib
+import sys
+import os
+import argparse
+import logging
+from . import simulation
+
+logger = logging.getLogger(__name__)
+
+
+def main():
+
+    parser = argparse.ArgumentParser(description='Run a SOIL simulation')
+    parser.add_argument('file', type=str,
+                        nargs="?",
+                        default='simulation.yml',
+                        help='python module containing the simulation configuration.')
+    parser.add_argument('--module', '-m', type=str,
+                        help='file containing the code of any custom agents.')
+    parser.add_argument('--dry-run', '--dry', action='store_true',
+                        help='Do not store the results of the simulation.')
+    parser.add_argument('--output', '-o', type=str,
+                        help='folder to write results to. It defaults to the current directory.')
+
+    args = parser.parse_args()
+
+    if args.module:
+        sys.path.append(os.getcwd())
+        importlib.import_module(args.module)
+
+    print('Loading config file: {}'.format(args.file, args.output))
+    simulation.run_from_config(args.file, dump=not args.dry_run, results_dir=args.output)
+
+
+if __name__ == '__main__':
+    main()

soil/agents/BaseBehaviour.py

@@ -1,123 +0,0 @@
-import nxsim
-from collections import OrderedDict
-from copy import deepcopy
-import json
-
-from functools import wraps
-
-
-class BaseAgent(nxsim.BaseAgent):
-    """
-    A special simpy BaseAgent that keeps track of its state history.
-    """
-
-    def __init__(self, *args, **kwargs):
-        self._history = OrderedDict()
-        self._neighbors = None
-        super().__init__(*args, **kwargs)
-        self._history[None] = deepcopy(self.state)
-
-    @property
-    def now(self):
-        try:
-            return self.env.now
-        except AttributeError:
-            # No environment
-            return None
-
-    def run(self):
-        while True:
-            res = self.step()
-            self._history[self.env.now] = deepcopy(self.state)
-            yield res or self.env.timeout(self.env.interval)
-
-    def step(self):
-        pass
-
-    def to_json(self):
-        return json.dumps(self._history)
-
-class NetworkAgent(BaseAgent, nxsim.BaseNetworkAgent):
-
-    def count_agents(self, state_id=None, limit_neighbors=False):
-        if limit_neighbors:
-            agents = self.global_topology.neighbors(self.id)
-        else:
-            agents = self.global_topology.nodes()
-        count = 0
-        for agent in agents:
-            if state_id and state_id != self.global_topology.node[agent]['agent'].state['id']:
-                continue
-            count += 1
-        return count
-
-    def count_neighboring_agents(self, state_id=None):
-        return self.count_agents(state_id, limit_neighbors=True)
-
-
-def state(func):
-
-    @wraps(func)
-    def func_wrapper(self):
-        when = None
-        next_state = func(self)
-        try:
-            next_state, when = next_state
-        except TypeError:
-            pass
-        if next_state:
-            try:
-                self.state['id'] = next_state.id
-            except AttributeError:
-                raise NotImplemented('State id %s is not valid.' % next_state)
-        return when
-
-    func_wrapper.id = func.__name__
-    func_wrapper.is_default = False
-    return func_wrapper
-
-
-def default_state(func):
-    func.is_default = True
-    return func
-
-
-class MetaFSM(type):
-    def __init__(cls, name, bases, nmspc):
-        super(MetaFSM, cls).__init__(name, bases, nmspc)
-        states = {}
-        # Re-use states from inherited classes
-        default_state = None
-        for i in bases:
-            if isinstance(i, MetaFSM):
-                for state_id, state in i.states.items():
-                    if state.is_default:
-                        default_state = state
-                    states[state_id] = state
-
-        # Add new states
-        for name, func in nmspc.items():
-            if hasattr(func, 'id'):
-                if func.is_default:
-                    default_state = func
-                states[func.id] = func
-        cls.default_state = default_state
-        cls.states = states
-
-
-class FSM(BaseAgent, metaclass=MetaFSM):
-    def __init__(self, *args, **kwargs):
-        super(FSM, self).__init__(*args, **kwargs)
-        if 'id' not in self.state:
-            self.state['id'] = self.default_state.id
-
-    def step(self):
-        if 'id' in self.state:
-            next_state = self.state['id']
-        elif self.default_state:
-            next_state = self.default_state.id
-        else:
-            raise Exception('{} has no valid state id or default state'.format(self))
-        if next_state not in self.states:
-            raise Exception('{} is not a valid id for {}'.format(next_state, self))
-        self.states[next_state](self)

soil/agents/BassModel.py

@@ -1,8 +1,8 @@
 import random
-from . import NetworkAgent
+from . import BaseAgent
 
 
-class BassModel(NetworkAgent):
+class BassModel(BaseAgent):
     """
     Settings:
         innovation_prob

soil/agents/BigMarketModel.py

@@ -1,8 +1,8 @@
 import random
-from . import NetworkAgent
+from . import BaseAgent
 
 
-class BigMarketModel(NetworkAgent):
+class BigMarketModel(BaseAgent):
     """
     Settings:
         Names:

soil/agents/CounterModel.py

@@ -1,7 +1,7 @@
-from . import NetworkAgent
+from . import BaseAgent
 
 
-class CounterModel(NetworkAgent):
+class CounterModel(BaseAgent):
     """
     Dummy behaviour. It counts the number of nodes in the network and neighbors
     in each step and adds it to its state.
@@ -9,14 +9,14 @@ class CounterModel(NetworkAgent):
 
     def step(self):
         # Outside effects
-        total = len(self.get_all_agents())
-        neighbors = len(self.get_neighboring_agents())
+        total = len(list(self.get_all_agents()))
+        neighbors = len(list(self.get_neighboring_agents()))
         self.state['times'] = self.state.get('times', 0) + 1
         self.state['neighbors'] = neighbors
         self.state['total'] = total
 
 
-class AggregatedCounter(NetworkAgent):
+class AggregatedCounter(BaseAgent):
     """
     Dummy behaviour. It counts the number of nodes in the network and neighbors
     in each step and adds it to its state.
@@ -24,8 +24,9 @@ class AggregatedCounter(NetworkAgent):
 
     def step(self):
         # Outside effects
-        total = len(self.get_all_agents())
-        neighbors = len(self.get_neighboring_agents())
+        total = len(list(self.get_all_agents()))
+        neighbors = len(list(self.get_neighboring_agents()))
         self.state['times'] = self.state.get('times', 0) + 1
         self.state['neighbors'] = self.state.get('neighbors', 0) + neighbors
-        self.state['total'] = self.state.get('total', 0) + total
+        self.state['total'] = total = self.state.get('total', 0) + total
+        self.debug('Running for step: {}. Total: {}'.format(self.now, total))

soil/agents/ModelM2.py

@@ -1,9 +1,9 @@
 import random
 import numpy as np
-from . import NetworkAgent
+from . import BaseAgent
 
 
-class SpreadModelM2(NetworkAgent):
+class SpreadModelM2(BaseAgent):
     """
     Settings:
         prob_neutral_making_denier
@@ -104,7 +104,7 @@ class SpreadModelM2(NetworkAgent):
                 neighbor.state['id'] = 2  # Cured
 
 
-class ControlModelM2(NetworkAgent):
+class ControlModelM2(BaseAgent):
     """
     Settings:
         prob_neutral_making_denier

soil/agents/SentimentCorrelationModel.py

@@ -1,8 +1,8 @@
 import random
-from . import NetworkAgent
+from . import BaseAgent
 
 
-class SentimentCorrelationModel(NetworkAgent):
+class SentimentCorrelationModel(BaseAgent):
     """
     Settings:
         outside_effects_prob

soil/agents/__init__.py

@@ -6,10 +6,13 @@
     
 
 import nxsim
+import logging
 from collections import OrderedDict
 from copy import deepcopy
+from functools import partial
 import json
 
+
 from functools import wraps
 
 
@@ -27,28 +30,38 @@ class BaseAgent(nxsim.BaseAgent, metaclass=MetaAgent):
     A special simpy BaseAgent that keeps track of its state history.
     """
 
-    def __init__(self, *args, **kwargs):
-        self._history = OrderedDict()
+    defaults = {}
+
+    def __init__(self, **kwargs):
         self._neighbors = None
-        super().__init__(*args, **kwargs)
+        self.alive = True
+        state = deepcopy(self.defaults)
+        state.update(kwargs.pop('state', {}))
+        kwargs['state'] = state
+        super().__init__(**kwargs)
+        if not hasattr(self, 'level'):
+            self.level = logging.DEBUG
+        self.logger = logging.getLogger('Agent-{}'.format(self.id))
+        self.logger.setLevel(self.level)
+
 
     def __getitem__(self, key):
         if isinstance(key, tuple):
             k, t_step = key
-            if k is not None:
-                if t_step is not None:
-                    return self._history[t_step][k]
-                else:
-                    return {tt: tv.get(k, None) for tt, tv in self._history.items()}
-            else:
-                return self._history[t_step]
-        return self.state[key]
+            return self.env[self.id, t_step, k]
+        return self.state.get(key, None)
+
+    def __delitem__(self, key):
+        del self.state[key]
+
+    def __contains__(self, key):
+        return key in self.state
 
     def __setitem__(self, key, value):
         self.state[key] = value
 
-    def save_state(self):
-        self._history[self.now] = deepcopy(self.state)
+    def get(self, key, default=None):
+        return self[key] if key in self else default
 
     @property
     def now(self):
@@ -59,18 +72,21 @@ class BaseAgent(nxsim.BaseAgent, metaclass=MetaAgent):
             return None
 
     def run(self):
-        while True:
+        interval = self.env.interval
+        while self.alive:
             res = self.step()
-            yield res or self.env.timeout(self.env.interval)
+            yield res or self.env.timeout(interval)
+
+    def die(self, remove=False):
+        self.alive = False
+        if remove:
+            super().die()
 
     def step(self):
         pass
 
     def to_json(self):
-        return json.dumps(self._history)
-
-
-class NetworkAgent(BaseAgent, nxsim.BaseNetworkAgent):
+        return json.dumps(self.state)
 
     def count_agents(self, state_id=None, limit_neighbors=False):
         if limit_neighbors:
@@ -85,7 +101,42 @@ class NetworkAgent(BaseAgent, nxsim.BaseNetworkAgent):
         return count
 
     def count_neighboring_agents(self, state_id=None):
-        return self.count_agents(state_id, limit_neighbors=True)
+        return len(super().get_agents(state_id, limit_neighbors=True))
+
+    def get_agents(self, state_id=None, limit_neighbors=False, iterator=False, **kwargs):
+        if limit_neighbors:
+            agents = super().get_agents(state_id, limit_neighbors)
+        else:
+            agents = filter(lambda x: state_id is None or x.state.get('id', None) == state_id,
+                            self.env.agents)
+
+        def matches_all(agent):
+            state = agent.state
+            for k, v in kwargs.items():
+                if state.get(k, None) != v:
+                    return False
+            return True
+
+        f = filter(matches_all, agents)
+        if iterator:
+            return f
+        return list(f)
+
+    def log(self, message, *args, level=logging.INFO, **kwargs):
+        message = message + " ".join(str(i) for i in args)
+        message = "\t@{:>5}:\t{}".format(self.now, message)
+        for k, v in kwargs:
+            message += " {k}={v} ".format(k, v)
+        extra = {}
+        extra['now'] = self.now
+        extra['id'] = self.id
+        return self.logger.log(level, message, extra=extra)
+
+    def debug(self, *args, **kwargs):
+        return self.log(*args, level=logging.DEBUG, **kwargs)
+
+    def info(self, *args, **kwargs):
+        return self.log(*args, level=logging.INFO, **kwargs)
 
 
 def state(func):
@@ -102,7 +153,7 @@ def state(func):
             try:
                 self.state['id'] = next_state.id
             except AttributeError:
-                raise NotImplemented('State id %s is not valid.' % next_state)
+                raise ValueError('State id %s is not valid.' % next_state)
         return when
 
     func_wrapper.id = func.__name__
@@ -155,6 +206,13 @@ class FSM(BaseAgent, metaclass=MetaFSM):
             raise Exception('{} is not a valid id for {}'.format(next_state, self))
         self.states[next_state](self)
 
+    def set_state(self, state):
+        if hasattr(state, 'id'):
+            state = state.id
+        if state not in self.states:
+            raise ValueError('{} is not a valid state'.format(state))
+        self.state['id'] = state
+
 
 from .BassModel import *
 from .BigMarketModel import *

soil/analysis.py

@@ -4,20 +4,175 @@ import glob
 import yaml
 from os.path import join
 
+from . import utils
 
-def get_data(pattern, process=True, attributes=None):
+
+def read_data(*args, group=False, **kwargs):
+    iterable = _read_data(*args, **kwargs)
+    if group:
+        return group_trials(iterable)
+    else:
+        return list(iterable)
+
+
+def _read_data(pattern, keys=None, convert_types=False,
+               process=None, from_csv=False, **kwargs):
     for folder in glob.glob(pattern):
         config_file = glob.glob(join(folder, '*.yml'))[0]
         config = yaml.load(open(config_file))
-        for trial_data in sorted(glob.glob(join(folder, '*.environment.csv'))):
-            df = pd.read_csv(trial_data)
-            if process:
-                if attributes is not None:
-                    df = df[df['attribute'].isin(attributes)]
-                df = df.pivot_table(values='attribute', index='tstep', columns=['value'], aggfunc='count').fillna(0)
-            yield config_file, df, config
+        df = None
+        if from_csv:
+            for trial_data in sorted(glob.glob(join(folder,
+                                                    '*.environment.csv'))):
+                df = read_csv(trial_data, convert_types=convert_types)
+                if process:
+                    df = process(df, **kwargs)
+                yield config_file, df, config
+        else:
+            for trial_data in sorted(glob.glob(join(folder, '*.db.sqlite'))):
+                df = read_sql(trial_data, convert_types=convert_types,
+                              keys=keys)
+                if process:
+                    df = process(df, **kwargs)
+                yield config_file, df, config
+
+
+def read_csv(filename, keys=None, convert_types=False, **kwargs):
+    '''
+    Read a CSV in canonical form: ::
+
+        <agent_id, t_step, key, value, value_type>
+
+    '''
+    df = pd.read_csv(filename)
+    if convert_types:
+        df = convert_types_slow(df)
+    if keys:
+        df = df[df['key'].isin(keys)]
+    return df
+
+
+def read_sql(filename, keys=None, convert_types=False, limit=-1):
+    condition = ''
+    if keys:
+        k = map(lambda x: "\'{}\'".format(x), keys)
+        condition = 'where key in ({})'.format(','.join(k))
+    query = 'select * from history {} limit {}'.format(condition, limit)
+    df = pd.read_sql_query(query, 'sqlite:///{}'.format(filename))
+    if convert_types:
+        df = convert_types_slow(df)
+    return df
+
+
+def convert_row(row):
+    row['value'] = utils.convert(row['value'], row['value_type'])
+    return row
+
+
+def convert_types_slow(df):
+    '''This is a slow operation.'''
+    dtypes = get_types(df)
+    for k, v in dtypes.items():
+        t = df[df['key']==k]
+        t['value'] = t['value'].astype(v)
+    df = df.apply(convert_row, axis=1)
+    return df
+
+def split_df(df):
+    '''
+    Split a dataframe in two dataframes: one with the history of agents,
+    and one with the environment history
+    '''
+    envmask = (df['agent_id'] == 'env')
+    n_env = envmask.sum()
+    if n_env == len(df):
+        return df, None
+    elif n_env == 0:
+        return None, df
+    agents, env = [x for _, x in df.groupby(envmask)]
+    return env, agents
+
+
+def process(df, **kwargs):
+    '''
+    Process a dataframe in canonical form ``(t_step, agent_id, key, value, value_type)`` into
+    two dataframes with a column per key: one with the history of the agents, and one for the
+    history of the environment.
+    '''
+    env, agents = split_df(df)
+    return process_one(env, **kwargs), process_one(agents, **kwargs)
+
+
+def get_types(df):
+    dtypes = df.groupby(by=['key'])['value_type'].unique()
+    return {k:v[0] for k,v in dtypes.iteritems()}
+
+
+def process_one(df, *keys, columns=['key'], values='value',
+                index=['t_step', 'agent_id'], aggfunc='first', **kwargs):
+    '''
+    Process a dataframe in canonical form ``(t_step, agent_id, key, value, value_type)`` into
+    a dataframe with a column per key
+    '''
+    if df is None:
+        return df
+    if keys:
+        df = df[df['key'].isin(keys)]
+
+    dtypes = get_types(df)
+
+    df = df.pivot_table(values=values, index=index, columns=columns,
+                        aggfunc=aggfunc, **kwargs)
+    df = df.fillna(0).astype(dtypes)
+    return df
+
+
+def get_count_processed(df, *keys):
+    if keys:
+        df = df[list(keys)]
+    # p = df.groupby(level=0).apply(pd.Series.value_counts)
+    p = df.unstack().apply(pd.Series.value_counts, axis=1)
+    return p
+
+
+def get_count(df, *keys):
+    if keys:
+        df = df[df['key'].isin(keys)]
+    p = df.groupby(by=['t_step', 'key', 'value']).size().unstack(level=[1,2]).fillna(0)
+    return p
+
+
+def get_value(df, *keys, aggfunc='sum'):
+    if keys:
+        df = df[df['key'].isin(keys)]
+    p = process_one(df, *keys)
+    p = p.groupby(level='t_step').agg(aggfunc)
+    return p
 
 
 def plot_all(*args, **kwargs):
-    for config_file, df, config in sorted(get_data(*args, **kwargs)):
+    '''
+    Read all the trial data and plot the result of applying a function on them.
+    '''
+    dfs = do_all(*args, **kwargs)
+    ps = []
+    for line in dfs:
+        f, df, config = line
         df.plot(title=config['name'])
+        ps.append(df)
+    return ps
+
+def do_all(pattern, func, *keys, include_env=False, **kwargs):
+    for config_file, df, config in read_data(pattern, keys=keys):
+        p = func(df, *keys, **kwargs)
+        p.plot(title=config['name'])
+        yield config_file, p, config
+
+
+def group_trials(trials, aggfunc=['mean', 'min', 'max', 'std']):
+    trials = list(trials)
+    trials = list(map(lambda x: x[1] if isinstance(x, tuple) else x, trials))
+    return pd.concat(trials).groupby(level=0).agg(aggfunc).reorder_levels([2, 0,1] ,axis=1)
+
+
+

soil/environment.py

@@ -1,12 +1,17 @@
 import os
-import csv
+import sqlite3
+import time
 import weakref
-from random import random
+import csv
+import random
+import simpy
 from copy import deepcopy
 
 import networkx as nx
 import nxsim
 
+from . import utils
+
 
 class SoilEnvironment(nxsim.NetworkEnvironment):
 
@@ -16,20 +21,43 @@ class SoilEnvironment(nxsim.NetworkEnvironment):
                  states=None,
                  default_state=None,
                  interval=1,
+                 seed=None,
+                 dump=False,
+                 simulation=None,
                  *args, **kwargs):
         self.name = name or 'UnnamedEnvironment'
-        self.states = deepcopy(states) or {}
+        if isinstance(states, list):
+            states = dict(enumerate(states))
+        self.states = deepcopy(states) if states else {}
         self.default_state = deepcopy(default_state) or {}
+        self.sim = weakref.ref(simulation)
+        if 'topology' not in kwargs and simulation:
+            kwargs['topology'] = self.sim().topology.copy()
         super().__init__(*args, **kwargs)
         self._env_agents = {}
-        self._history = {}
         self.interval = interval
-        self.logger = None
+        self.dump = dump
         # Add environment agents first, so their events get
         # executed before network agents
+        self['SEED'] = seed or time.time()
+        random.seed(self['SEED'])
+        self.process(self.save_state())
         self.environment_agents = environment_agents or []
         self.network_agents = network_agents or []
-        self.process(self.save_state())
+        if self.dump:
+            self._db_path = os.path.join(self.get_path(), '{}.db.sqlite'.format(self.name))
+        else:
+            self._db_path = ":memory:"
+        self.create_db(self._db_path)
+
+    def create_db(self, db_path=None):
+        db_path = db_path or self._db_path
+        if os.path.exists(db_path):
+            newname = db_path.replace('db.sqlite', 'backup{}.sqlite'.format(time.time()))
+            os.rename(db_path, newname)
+        self._db = sqlite3.connect(db_path)
+        with self._db:
+            self._db.execute('''CREATE TABLE IF NOT EXISTS history (agent_id text, t_step int, key text, value text, value_type text)''')
 
     @property
     def agents(self):
@@ -39,7 +67,7 @@ class SoilEnvironment(nxsim.NetworkEnvironment):
     @property
     def environment_agents(self):
         for ref in self._env_agents.values():
-            yield ref()
+            yield ref
 
     @environment_agents.setter
     def environment_agents(self, environment_agents):
@@ -50,9 +78,8 @@ class SoilEnvironment(nxsim.NetworkEnvironment):
             atype = kwargs.pop('agent_type')
             kwargs['agent_id'] = kwargs.get('agent_id', atype.__name__)
             kwargs['state'] = kwargs.get('state', {})
-            a = atype(**kwargs,
-                      environment=self)
-            self._env_agents[a.id] = weakref.ref(a)
+            a = atype(environment=self, **kwargs)
+            self._env_agents[a.id] = a
 
     @property
     def network_agents(self):
@@ -66,53 +93,114 @@ class SoilEnvironment(nxsim.NetworkEnvironment):
         for ix in self.G.nodes():
             i = ix
             node = self.G.node[i]
-            v = random()
-            found = False
-            for d in network_agents:
-                threshold = d['threshold']
-                if v >= threshold[0] and v < threshold[1]:
-                    agent = d['agent_type']
-                    state = None
-                    if 'state' in d:
-                        state = deepcopy(d['state'])
-                    else:
-                        try:
-                            state = self.states[i]
-                        except (IndexError, KeyError):
-                            state = deepcopy(self.default_state)
-                    node['agent'] = agent(environment=self,
-                                          agent_id=i,
-                                          state=state)
-                    found = True
-                    break
-            assert found
+            agent, state = utils.agent_from_distribution(network_agents)
+            self.set_agent(i, agent_type=agent, state=state)
+
+    def set_agent(self, agent_id, agent_type, state=None):
+        node = self.G.nodes[agent_id]
+        defstate = deepcopy(self.default_state)
+        defstate.update(self.states.get(agent_id, {}))
+        if state:
+            defstate.update(state)
+        state = defstate
+        state.update(node.get('state', {}))
+        a = agent_type(environment=self,
+                       agent_id=agent_id,
+                       state=state)
+        node['agent'] = a
+        return a
+
+    def add_node(self, agent_type, state=None):
+        agent_id = int(len(self.G.nodes()))
+        self.G.add_node(agent_id)
+        a = self.set_agent(agent_id, agent_type, state)
+        a['visible'] = True
+        return a
+
+    def add_edge(self, agent1, agent2, attrs=None):
+        return self.G.add_edge(agent1, agent2)
 
     def run(self, *args, **kwargs):
-        self._save_state()
         super().run(*args, **kwargs)
-        self._save_state()
 
-    def _save_state(self):
-        for agent in self.agents:
-            agent.save_state()
-        self._history[self.now] = deepcopy(self.environment_params)
+    def _save_state(self, now=None):
+        # for agent in self.agents:
+        #     agent.save_state()
+        utils.logger.debug('Saving state @{}'.format(self.now))
+        with self._db:
+            self._db.executemany("insert into history(agent_id, t_step, key, value, value_type) values (?, ?, ?, ?, ?)", self.state_to_tuples(now=now))
 
     def save_state(self):
-        while True:
+        self._save_state()
+        while self.peek() != simpy.core.Infinity:
+            delay = max(self.peek() - self.now, self.interval)
+            utils.logger.debug('Step: {}'.format(self.now))
             ev = self.event()
             ev._ok = True
             # Schedule the event with minimum priority so
-            # that it executes after all agents are done
-            self.schedule(ev, -1, self.interval)
+            # that it executes before all agents
+            self.schedule(ev, -999, delay)
             yield ev
             self._save_state()
 
     def __getitem__(self, key):
+        if isinstance(key, tuple):
+            values = {"agent_id": key[0],
+                      "t_step": key[1],
+                      "key": key[2],
+                      "value": None,
+                      "value_type": None
+            }
+
+            fields = list(k for k, v in values.items() if v is None)
+            conditions = " and ".join("{}='{}'".format(k, v) for k, v in values.items() if v is not None)
+
+            query = """SELECT {fields} from history""".format(fields=",".join(fields))
+            if conditions:
+                query = """{query} where {conditions}""".format(query=query,
+                                                                conditions=conditions)
+            with self._db:
+                rows = self._db.execute(query).fetchall()
+
+            utils.logger.debug(rows)
+            results = self.rows_to_dict(rows)
+            return results
+
         return self.environment_params[key]
 
+    def rows_to_dict(self, rows):
+        if len(rows) < 1:
+            return None
+
+        level = len(rows[0])-2
+
+        if level == 0:
+            if len(rows) != 1:
+                raise ValueError('Cannot convert {} to dictionaries'.format(rows))
+            value, value_type = rows[0]
+            return utils.convert(value, value_type)
+
+        results = {}
+        for row in rows:
+            item = results
+            for i in range(level-1):
+                key = row[i]
+                if key not in item:
+                    item[key] = {}
+                item = item[key]
+            key, value, value_type = row[level-1:]
+            item[key] = utils.convert(value, value_type)
+        return results
+
     def __setitem__(self, key, value):
         self.environment_params[key] = value
 
+    def __contains__(self, key):
+        return key in self.environment_params
+
+    def get(self, key, default=None):
+        return self[key] if key in self else default
+
     def get_path(self, dir_path=None):
         dir_path = dir_path or self.sim().dir_path
         if not os.path.exists(dir_path):
@@ -131,7 +219,7 @@ class SoilEnvironment(nxsim.NetworkEnvironment):
 
         with open(csv_name, 'w') as f:
             cr = csv.writer(f)
-            cr.writerow(('agent_id', 'tstep', 'attribute', 'value'))
+            cr.writerow(('agent_id', 't_step', 'key', 'value', 'value_type'))
             for i in self.history_to_tuples():
                 cr.writerow(i)
 
@@ -141,26 +229,36 @@ class SoilEnvironment(nxsim.NetworkEnvironment):
                                   self.name+".gexf")
         nx.write_gexf(G, graph_path, version="1.2draft")
 
-    def history_to_tuples(self):
-        for tstep, state in self._history.items():
-            for attribute, value in state.items():
-                yield ('env', tstep, attribute, value)
+    def state_to_tuples(self, now=None):
+        if now is None:
+            now = self.now
+        for k, v in self.environment_params.items():
+            v, v_t = utils.repr(v)
+            yield 'env', now, k, v, v_t
         for agent in self.agents:
-            for tstep, state in agent._history.items():
-                for attribute, value in state.items():
-                    yield (agent.id, tstep, attribute, value)
+            for k, v in agent.state.items():
+                v, v_t = utils.repr(v)
+                yield agent.id, now, k, v, v_t
+
+    def history_to_tuples(self):
+        with self._db:
+            res = self._db.execute("select agent_id, t_step, key, value, value_type from history ").fetchall()
+        yield from res
 
     def history_to_graph(self):
         G = nx.Graph(self.G)
 
-        for agent in self.agents:
+        for agent in self.network_agents:
 
             attributes = {'agent': str(agent.__class__)}
             lastattributes = {}
             spells = []
             lastvisible = False
             laststep = None
-            for t_step, state in reversed(agent._history.items()):
+            history = self[agent.id, None, None]
+            if not history:
+                continue
+            for t_step, state in reversed(sorted(list(history.items()))):
                 for attribute, value in state.items():
                     if attribute == 'visible':
                         nowvisible = state[attribute]
@@ -171,20 +269,25 @@ class SoilEnvironment(nxsim.NetworkEnvironment):
 
                         lastvisible = nowvisible
                     else:
-                        if attribute not in lastattributes or lastattributes[attribute][0] != value:
-                            laststep = lastattributes.get(attribute,
-                                                          (None, None))[1]
-                            value = (state[attribute], t_step, laststep)
-                            key = 'attr_' + attribute
+                        key = 'attr_' + attribute
+                        if key not in attributes:
+                            attributes[key] = list()
+                        if key not in lastattributes:
+                            lastattributes[key] = (state[attribute], t_step)
+                        elif lastattributes[key][0] != value:
+                            last_value, laststep = lastattributes[key]
+                            value = (last_value, t_step, laststep)
                             if key not in attributes:
                                 attributes[key] = list()
                             attributes[key].append(value)
-                            lastattributes[attribute] = (state[attribute], t_step)
+                            lastattributes[key] = (state[attribute], t_step)
+            for k, v in lastattributes.items():
+                attributes[k].append((v[0], 0, v[1]))
             if lastvisible:
                 spells.append((laststep, None))
             if spells:
-                G.add_node(agent.id, attributes, spells=spells)
+                G.add_node(agent.id, spells=spells, **attributes)
             else:
-                G.add_node(agent.id, attributes)
+                G.add_node(agent.id, **attributes)
 
         return G

soil/simulation.py

@@ -1,20 +1,19 @@
-import weakref
 import os
-import csv
 import time
+import imp
+import sys
 import yaml
 import networkx as nx
 from networkx.readwrite import json_graph
 
 from copy import deepcopy
-from random import random
-from matplotlib import pyplot as plt
 
 import pickle
 
 from nxsim import NetworkSimulation
 
 from . import agents, utils, environment, basestring
+from .utils import logger
 
 
 class SoilSimulation(NetworkSimulation):
@@ -47,9 +46,9 @@ class SoilSimulation(NetworkSimulation):
     """
     def __init__(self, name=None, topology=None, network_params=None,
                  network_agents=None, agent_type=None, states=None,
-                 default_state=None, interval=1,
-                 dir_path=None, num_trials=3, max_time=100,
-                 agent_module=None,
+                 default_state=None, interval=1, dump=False,
+                 dir_path=None, num_trials=1, max_time=100,
+                 agent_module=None, load_module=None, seed=None,
                  environment_agents=None, environment_params=None):
 
         if topology is None:
@@ -58,6 +57,8 @@ class SoilSimulation(NetworkSimulation):
         elif isinstance(topology, basestring) or isinstance(topology, dict):
             topology = json_graph.node_link_graph(topology)
 
+
+        self.load_module = load_module
         self.topology = nx.Graph(topology)
         self.network_params = network_params
         self.name = name or 'UnnamedSimulation'
@@ -66,8 +67,15 @@ class SoilSimulation(NetworkSimulation):
         self.default_state = default_state or {}
         self.dir_path = dir_path or os.getcwd()
         self.interval = interval
+        self.seed = str(seed) or str(time.time())
+        self.dump = dump
         self.environment_params = environment_params or {}
 
+        if load_module:
+            path = sys.path + [self.dir_path]
+            f, fp, desc = imp.find_module(load_module, path)
+            imp.load_module('soil.agents.custom', f, fp, desc)
+
         environment_agents = environment_agents or []
         self.environment_agents = self._convert_agent_types(environment_agents)
 
@@ -76,7 +84,7 @@ class SoilSimulation(NetworkSimulation):
         self.network_agents = self._convert_agent_types(distro)
 
         self.states = self.validate_states(states,
-                                           topology)
+                                           self.topology)
 
     def calculate_distribution(self,
                                network_agents=None,
@@ -132,12 +140,23 @@ class SoilSimulation(NetworkSimulation):
     def run(self):
         return list(self.run_simulation_gen())
 
-    def run_simulation_gen(self):
+    def run_simulation_gen(self, *args, **kwargs):
         with utils.timer('simulation'):
             for i in range(self.num_trials):
-                yield self.run_trial(i)
+                res = self.run_trial(i)
+                if self.dump:
+                    res.dump_gexf(self.dir_path)
+                    res.dump_csv(self.dir_path)
+                yield res
 
-    def run_trial(self, trial_id=0):
+            if self.dump:
+                logger.info('Dumping results to {}'.format(self.dir_path))
+                self.dump_pickle(self.dir_path)
+                self.dump_yaml(self.dir_path)
+            else:
+                logger.info('NOT dumping results')
+
+    def run_trial(self, trial_id=0, dump=False, dir_path=None):
         """Run a single trial of the simulation
 
         Parameters
@@ -145,21 +164,22 @@ class SoilSimulation(NetworkSimulation):
         trial_id : int
         """
         # Set-up trial environment and graph
-        print('Trial: {}'.format(trial_id))
+        logger.info('Trial: {}'.format(trial_id))
         env_name = '{}_trial_{}'.format(self.name, trial_id)
         env = environment.SoilEnvironment(name=env_name,
                                           topology=self.topology.copy(),
+                                          seed=self.seed+env_name,
                                           initial_time=0,
+                                          dump=self.dump,
                                           interval=self.interval,
                                           network_agents=self.network_agents,
                                           states=self.states,
                                           default_state=self.default_state,
                                           environment_agents=self.environment_agents,
+                                          simulation=self,
                                           **self.environment_params)
-
-        env.sim = weakref.ref(self)
         # Set up agents on nodes
-        print('\tRunning')
+        logger.info('\tRunning')
         with utils.timer('trial'):
             env.run(until=self.max_time)
         return env
@@ -221,7 +241,7 @@ def run_from_config(*configs, dump=True, results_dir=None, timestamp=False):
     for config_def in configs:
         for config, cpath in utils.load_config(config_def):
             name = config.get('name', 'unnamed')
-            print("Using config(s): {name}".format(name=name))
+            logger.info("Using config(s): {name}".format(name=name))
 
             sim = SoilSimulation(**config)
             if timestamp:
@@ -229,13 +249,7 @@ def run_from_config(*configs, dump=True, results_dir=None, timestamp=False):
                                             time.strftime("%Y-%m-%d_%H:%M:%S"))
             else:
                 sim_folder = sim.name
-            dir_path = os.path.join(results_dir,
-                                    sim_folder)
+            sim.dir_path = os.path.join(results_dir, sim_folder)
+            sim.dump = dump
+            logger.info('Dumping results to {} : {}'.format(sim.dir_path, dump))
             results = sim.run_simulation()
-
-            if dump:
-                sim.dump_pickle(dir_path)
-                sim.dump_yaml(dir_path)
-                for env in results:
-                    env.dump_gexf(dir_path)
-                    env.dump_csv(dir_path)

soil/utils.py

@@ -1,14 +1,23 @@
 import os
 import yaml
+import logging
 from time import time
 from glob import glob
+from random import random
+from copy import deepcopy
 
 import networkx as nx
 
 from contextlib import contextmanager
 
 
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
 def load_network(network_params, dir_path=None):
+    if network_params is None:
+        return nx.Graph()
     path = network_params.get('path', None)
     if path:
         if dir_path and not os.path.isabs(path):
@@ -51,7 +60,7 @@ def load_config(config):
 
 
 @contextmanager
-def timer(name='task', pre="", function=print, to_object=None):
+def timer(name='task', pre="", function=logger.info, to_object=None):
     start = time()
     yield start
     end = time()
@@ -59,3 +68,38 @@ def timer(name='task', pre="", function=print, to_object=None):
     if to_object:
         to_object.start = start
         to_object.end = end
+
+
+def agent_from_distribution(distribution, value=-1):
+    """Find the agent """
+    if value < 0:
+        value = random()
+    for d in distribution:
+        threshold = d['threshold']
+        if value >= threshold[0] and value < threshold[1]:
+            state = {}
+            if 'state' in d:
+                state = deepcopy(d['state'])
+            return d['agent_type'], state
+
+    raise Exception('Distribution for value {} not found in: {}'.format(value, distribution))
+
+
+def repr(v):
+    if isinstance(v, bool):
+        v = "true" if v else ""
+        return v, bool.__name__
+    return v, type(v).__name__
+
+def convert(value, type_):
+    import importlib
+    try:
+        # Check if it's a builtin type
+        module = importlib.import_module('builtins')
+        cls = getattr(module, type_)
+    except AttributeError:
+        # if not, separate module and class
+        module, type_ = type_.rsplit(".", 1)
+        module = importlib.import_module(module)
+        cls = getattr(module, type_)
+    return cls(value)

tests/test_main.py

@@ -5,7 +5,7 @@ import yaml
 from functools import partial
 
 from os.path import join
-from soil import simulation, agents, utils
+from soil import simulation, environment, agents, utils
 
 
 ROOT = os.path.abspath(os.path.dirname(__file__))
@@ -60,7 +60,7 @@ class TestMain(TestCase):
             'network_params': {
                 'path': join(ROOT, 'test.gexf')
             },
-            'agent_type': 'NetworkAgent',
+            'agent_type': 'BaseAgent',
             'environment_params': {
             }
         }
@@ -111,7 +111,7 @@ class TestMain(TestCase):
         env = s.run_simulation()[0]
         for agent in env.network_agents:
             last = 0
-            assert len(agent._history) == 11
+            assert len(agent[None, None]) == 11
             for step, total in agent['total', None].items():
                 if step > 0:
                     assert total == last + 2
@@ -119,7 +119,7 @@ class TestMain(TestCase):
 
     def test_custom_agent(self):
         """Allow for search of neighbors with a certain state_id"""
-        class CustomAgent(agents.NetworkAgent):
+        class CustomAgent(agents.BaseAgent):
             def step(self):
                 self.state['neighbors'] = self.count_agents(state_id=0,
                                                             limit_neighbors=True)
@@ -177,6 +177,7 @@ class TestMain(TestCase):
             recovered = yaml.load(serial)
         with utils.timer('deleting'):
             del recovered['topology']
+            del recovered['load_module']
         assert config == recovered
 
     def test_configuration_changes(self):
@@ -190,12 +191,33 @@ class TestMain(TestCase):
             s.run_simulation()
             nconfig = s.to_dict()
             del nconfig['topology']
+            del nconfig['load_module']
             assert config == nconfig
 
     def test_examples(self):
         """
         Make sure all examples in the examples folder are correct
         """
+        pass
+
+    def test_row_conversion(self):
+        sim = simulation.SoilSimulation()
+        env = environment.SoilEnvironment(simulation=sim)
+        env['test'] = 'test_value'
+        env._save_state(now=0)
+
+        res = list(env.history_to_tuples())
+        assert len(res) == len(env.environment_params)
+        assert ('env', 0, 'test', 'test_value', 'str') in res
+
+        env['test'] = 'second_value'
+        env._save_state(now=1)
+        res = list(env.history_to_tuples())
+
+        assert env['env', 0, 'test' ] == 'test_value'
+        assert env['env', 1, 'test' ] == 'second_value'
+
+
 
 
 def make_example_test(path, config):
@@ -205,9 +227,11 @@ def make_example_test(path, config):
         s = simulation.from_config(config)
         envs = s.run_simulation()
         for env in envs:
-            n = config['network_params'].get('n', None)
-            if n is not None:
+            try:
+                n = config['network_params']['n']
                 assert len(env.get_agents()) == n
+            except KeyError:
+                pass
         os.chdir(root)
     return wrapped