diff --git a/README.md b/README.md index 714d4df..7ea6f1d 100644 --- a/README.md +++ b/README.md @@ -39,6 +39,7 @@ As of this writing, This is a non-exhaustive list of tasks to achieve compatibility: * Environments.agents and mesa.Agent.agents are not the same. env is a property, and it only takes into account network and environment agents. Might rename environment_agents to other_agents or sth like that + - [ ] Integrate `soil.Simulation` with mesa's runners: - [ ] `soil.Simulation` could mimic/become a `mesa.batchrunner` - [ ] Integrate `soil.Environment` with `mesa.Model`: diff --git a/docs/configuration.rst b/docs/configuration.rst index c829a2f..3d5a88d 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -88,9 +88,18 @@ For example, the following configuration is equivalent to :code:`nx.complete_gra Environment ============ + The environment is the place where the shared state of the simulation is stored. -For instance, the probability of disease outbreak. -The configuration file may specify the initial value of the environment parameters: +That means both global parameters, such as the probability of disease outbreak. +But it also means other data, such as a map, or a network topology that connects multiple agents. +As a result, it is also typical to add custom functions in an environment that help agents interact with each other and with the state of the simulation. + +Last but not least, an environment controls when and how its agents will be executed. +By default, soil environments incorporate a ``soil.time.TimedActivation`` model for agent execution (more on this on the following section). + +Soil environments are very similar, and often interchangeable with, mesa models (``mesa.Model``). + +A configuration may specify the initial value of the environment parameters: .. code:: yaml @@ -98,23 +107,33 @@ The configuration file may specify the initial value of the environment paramete daily_probability_of_earthquake: 0.001 number_of_earthquakes: 0 -All agents have access to the environment parameters. +All agents have access to the environment (and its parameters). In some scenarios, it is useful to have a custom environment, to provide additional methods or to control the way agents update environment state. For example, if our agents play the lottery, the environment could provide a method to decide whether the agent wins, instead of leaving it to the agent. - Agents ====== + Agents are a way of modelling behavior. Agents can be characterized with two variables: agent type (``agent_type``) and state. -Only one agent is executed at a time (generally, every ``interval`` seconds), and it has access to its state and the environment parameters. +The agent type is a ``soil.Agent`` class, which contains the code that encapsulates the behavior of the agent. +The state is a set of variables, which may change during the simulation, and that the code may use to control the behavior. +All agents provide a ``step`` method either explicitly or implicitly (by inheriting it from a superclass), which controls how the agent will behave in each step of the simulation. + +When and how agent steps are executed in a simulation depends entirely on the ``environment``. +Most environments will internally use a scheduler (``mesa.time.BaseScheduler``), which controls the activation of agents. + +In soil, we generally used the ``soil.time.TimedActivation`` scheduler, which allows agents to specify when their next activation will happen, defaulting to a + +When an agent's step is executed (generally, every ``interval`` seconds), the agent has access to its state and the environment. Through the environment, it can access the network topology and the state of other agents. -There are three three types of agents according to how they are added to the simulation: network agents and environment agent. +There are two types of agents according to how they are added to the simulation: network agents and environment agent. Network Agents ############## + Network agents are attached to a node in the topology. The configuration file allows you to specify how agents will be mapped to topology nodes. @@ -125,7 +144,9 @@ Hence, every node in the network will be associated to an agent of that type. agent_type: SISaModel -It is also possible to add more than one type of agent to the simulation, and to control the ratio of each type (using the ``weight`` property). +It is also possible to add more than one type of agent to the simulation. + +To control the ratio of each type (using the ``weight`` property). For instance, with following configuration, it is five times more likely for a node to be assigned a CounterModel type than a SISaModel type. .. code:: yaml diff --git a/examples/complete.yml b/examples/complete.yml index b3d388a..a4b5b04 100644 --- a/examples/complete.yml +++ b/examples/complete.yml @@ -1,27 +1,38 @@ --- -name: simple -group: tests -dir_path: "/tmp/" -num_trials: 3 -max_time: 100 -interval: 1 -seed: "CompleteSeed!" -network_params: - generator: complete_graph - n: 10 -network_agents: - - agent_type: CounterModel - weight: 1 +general: + name: simple + group: tests + dir_path: "/tmp/" + num_trials: 3 + max_time: 100 + interval: 1 + seed: "CompleteSeed!" +network: + group: + network + params: + generator: complete_graph + n: 10 +environment: + environment_class: Environment + params: + am_i_complete: true +agents: + default: + agent_class: CounterModel state: - state_id: 0 - - agent_type: AggregatedCounter - weight: 0.2 -environment_agents: [] -environment_class: Environment -environment_params: - am_i_complete: true -default_state: - incidents: 0 -states: - - name: 'The first node' - - name: 'The second node' + times: 1 + environment: + fixed: + - agent_id: 'Environment Agent 1' + agent_class: CounterModel + state: + times: 10 + network: + distribution: + - agent_class: CounterModel + weight: 1 + state: + state_id: 0 + - agent_class: AggregatedCounter + weight: 0.2 diff --git a/examples/newsspread/NewsSpread.yml b/examples/newsspread/NewsSpread.yml index ffb1778..27f8411 100644 --- a/examples/newsspread/NewsSpread.yml +++ b/examples/newsspread/NewsSpread.yml @@ -1,6 +1,5 @@ --- default_state: {} -load_module: newsspread environment_agents: [] environment_params: prob_neighbor_spread: 0.0 @@ -9,11 +8,11 @@ interval: 1 max_time: 300 name: Sim_all_dumb network_agents: -- agent_type: DumbViewer +- agent_type: newsspread.DumbViewer state: has_tv: false weight: 1 -- agent_type: DumbViewer +- agent_type: newsspread.DumbViewer state: has_tv: true weight: 1 @@ -24,7 +23,6 @@ network_params: num_trials: 50 --- default_state: {} -load_module: newsspread environment_agents: [] environment_params: prob_neighbor_spread: 0.0 @@ -33,19 +31,19 @@ interval: 1 max_time: 300 name: Sim_half_herd network_agents: -- agent_type: DumbViewer +- agent_type: newsspread.DumbViewer state: has_tv: false weight: 1 -- agent_type: DumbViewer +- agent_type: newsspread.DumbViewer state: has_tv: true weight: 1 -- agent_type: HerdViewer +- agent_type: newsspread.HerdViewer state: has_tv: false weight: 1 -- agent_type: HerdViewer +- agent_type: newsspread.HerdViewer state: has_tv: true weight: 1 @@ -56,7 +54,6 @@ network_params: num_trials: 50 --- default_state: {} -load_module: newsspread environment_agents: [] environment_params: prob_neighbor_spread: 0.0 @@ -65,12 +62,12 @@ interval: 1 max_time: 300 name: Sim_all_herd network_agents: -- agent_type: HerdViewer +- agent_type: newsspread.HerdViewer state: has_tv: true state_id: neutral weight: 1 -- agent_type: HerdViewer +- agent_type: newsspread.HerdViewer state: has_tv: true state_id: neutral @@ -82,7 +79,6 @@ network_params: num_trials: 50 --- default_state: {} -load_module: newsspread environment_agents: [] environment_params: prob_neighbor_spread: 0.0 @@ -92,12 +88,12 @@ interval: 1 max_time: 300 name: Sim_wise_herd network_agents: -- agent_type: HerdViewer +- agent_type: newsspread.HerdViewer state: has_tv: true state_id: neutral weight: 1 -- agent_type: WiseViewer +- agent_type: newsspread.WiseViewer state: has_tv: true weight: 1 @@ -108,7 +104,6 @@ network_params: num_trials: 50 --- default_state: {} -load_module: newsspread environment_agents: [] environment_params: prob_neighbor_spread: 0.0 @@ -118,12 +113,12 @@ interval: 1 max_time: 300 name: Sim_all_wise network_agents: -- agent_type: WiseViewer +- agent_type: newsspread.WiseViewer state: has_tv: true state_id: neutral weight: 1 -- agent_type: WiseViewer +- agent_type: newsspread.WiseViewer state: has_tv: true weight: 1 diff --git a/examples/rabbits/rabbits.yml b/examples/rabbits/rabbits.yml index a78a5db..a4ec8e8 100644 --- a/examples/rabbits/rabbits.yml +++ b/examples/rabbits/rabbits.yml @@ -1,5 +1,4 @@ --- -load_module: rabbit_agents name: rabbits_example max_time: 100 interval: 1 diff --git a/examples/terrorism/TerroristNetworkModel.yml b/examples/terrorism/TerroristNetworkModel.yml index 401b77d..45dd1aa 100644 --- a/examples/terrorism/TerroristNetworkModel.yml +++ b/examples/terrorism/TerroristNetworkModel.yml @@ -1,5 +1,4 @@ name: TerroristNetworkModel_sim -load_module: TerroristNetworkModel max_time: 150 num_trials: 1 network_params: @@ -9,19 +8,19 @@ network_params: # theta: 20 n: 100 network_agents: - - agent_type: TerroristNetworkModel + - agent_type: TerroristNetworkModel.TerroristNetworkModel weight: 0.8 state: id: civilian # Civilians - - agent_type: TerroristNetworkModel + - agent_type: TerroristNetworkModel.TerroristNetworkModel weight: 0.1 state: id: leader # Leaders - - agent_type: TrainingAreaModel + - agent_type: TerroristNetworkModel.TrainingAreaModel weight: 0.05 state: id: terrorist # Terrorism - - agent_type: HavenModel + - agent_type: TerroristNetworkModel.HavenModel weight: 0.05 state: id: civilian # Civilian diff --git a/requirements.txt b/requirements.txt index 9d28021..2660dfd 100644 --- a/requirements.txt +++ b/requirements.txt @@ -7,3 +7,4 @@ SALib>=1.3 Jinja2 Mesa>=0.8.9 tsih>=0.1.6 +pydantic>=1.9 diff --git a/soil/config.py b/soil/config.py index 390be62..4f4ceed 100644 --- a/soil/config.py +++ b/soil/config.py @@ -1,251 +1,183 @@ +from __future__ import annotations +from pydantic import BaseModel, ValidationError, validator, root_validator + import yaml import os import sys -import networkx as nx -import collections.abc - -from . import serialization, utils, basestring, agents - -class Config(collections.abc.Mapping): - """ - - 1) agent type can be specified by name or by class. - 2) instead of just one type, a network agents distribution can be used. - The distribution specifies the weight (or probability) of each - agent type in the topology. This is an example distribution: :: - - [ - {'agent_type': 'agent_type_1', - 'weight': 0.2, - 'state': { - 'id': 0 - } - }, - {'agent_type': 'agent_type_2', - 'weight': 0.8, - 'state': { - 'id': 1 - } - } - ] - - In this example, 20% of the nodes will be marked as type - 'agent_type_1'. - 3) if no initial state is given, each node's state will be set - to `{'id': 0}`. - - Parameters - --------- - name : str, optional - name of the Simulation - group : str, optional - a group name can be used to link simulations - topology (optional): networkx.Graph instance or Node-Link topology as a dict or string (will be loaded with `json_graph.node_link_graph(topology`). - network_params : dict - parameters used to create a topology with networkx, if no topology is given - network_agents : dict - definition of agents to populate the topology with - agent_type : NetworkAgent subclass, optional - Default type of NetworkAgent to use for nodes not specified in network_agents - states : list, optional - List of initial states corresponding to the nodes in the topology. Basic form is a list of integers - whose value indicates the state - dir_path: str, optional - Directory path to load simulation assets (files, modules...) - seed : str, optional - Seed to use for the random generator - num_trials : int, optional - Number of independent simulation runs - max_time : int, optional - Maximum step/time for each simulation - environment_params : dict, optional - Dictionary of globally-shared environmental parameters - environment_agents: dict, optional - Similar to network_agents. Distribution of Agents that control the environment - environment_class: soil.environment.Environment subclass, optional - Class for the environment. It defailts to soil.environment.Environment - """ - __slots__ = 'name', 'agent_type', 'group', 'network_agents', 'environment_agents', 'states', 'default_state', 'interval', 'network_params', 'seed', 'num_trials', 'max_time', 'topology', 'schedule', 'initial_time', 'environment_params', 'environment_class', 'dir_path', '_added_to_path' - - def __init__(self, name=None, - group=None, - agent_type='BaseAgent', - network_agents=None, - environment_agents=None, - states=None, - default_state=None, - interval=1, - network_params=None, - seed=None, - num_trials=1, - max_time=None, - topology=None, - schedule=None, - initial_time=0, - environment_params={}, - environment_class='soil.Environment', - dir_path=None): - - self.network_params = network_params - self.name = name or 'Unnamed' - self.seed = str(seed or name) - self.group = group or '' - self.num_trials = num_trials - self.max_time = max_time - self.default_state = default_state or {} - self.dir_path = dir_path or os.getcwd() - self.interval = interval - - self._added_to_path = list(x for x in [os.getcwd(), self.dir_path] if x not in sys.path) - sys.path += self._added_to_path - - self.topology = topology - - self.schedule = schedule - self.initial_time = initial_time - - - self.environment_class = environment_class - self.environment_params = dict(environment_params) - - #TODO: Check agent distro vs fixed agents - self.environment_agents = environment_agents or [] - - self.agent_type = agent_type - - self.network_agents = network_agents or {} - - self.states = states or {} - - - def validate(self): - agents._validate_states(self.states, - self._topology) - - def restore_path(self): - for added in self._added_to_path: - sys.path.remove(added) - - def to_yaml(self): - return yaml.dump(self.to_dict()) - - def dump_yaml(self, f=None, outdir=None): - if not f and not outdir: - raise ValueError('specify a file or an output directory') - - if not f: - f = os.path.join(outdir, '{}.dumped.yml'.format(self.name)) - - with utils.open_or_reuse(f, 'w') as f: - f.write(self.to_yaml()) - - def to_yaml(self): - return yaml.dump(self.to_dict()) - - # TODO: See note on getstate - def to_dict(self): - return self.__getstate__() - - def dump_yaml(self, f=None, outdir=None): - if not f and not outdir: - raise ValueError('specify a file or an output directory') - - if not f: - f = os.path.join(outdir, '{}.dumped.yml'.format(self.name)) - - with utils.open_or_reuse(f, 'w') as f: - f.write(self.to_yaml()) - - def __getitem__(self, key): - return getattr(self, key) - - def __iter__(self): - return (k for k in self.__slots__ if k[0] != '_') - - def __len__(self): - return len(self.__slots__) - - def dump_pickle(self, f=None, outdir=None): - if not outdir and not f: - raise ValueError('specify a file or an output directory') - - if not f: - f = os.path.join(outdir, - '{}.simulation.pickle'.format(self.name)) - with utils.open_or_reuse(f, 'wb') as f: - pickle.dump(self, f) - - # TODO: remove this. A config should be sendable regardless. Non-pickable objects could be computed via properties and the like - # def __getstate__(self): - # state={} - # for k, v in self.__dict__.items(): - # if k[0] != '_': - # state[k] = v - # state['topology'] = json_graph.node_link_data(self.topology) - # state['network_agents'] = agents.serialize_definition(self.network_agents, - # known_modules = []) - # state['environment_agents'] = agents.serialize_definition(self.environment_agents, - # known_modules = []) - # state['environment_class'] = serialization.serialize(self.environment_class, - # known_modules=['soil.environment'])[1] # func, name - # if state['load_module'] is None: - # del state['load_module'] - # return state - - # # TODO: remove, same as __getstate__ - # def __setstate__(self, state): - # self.__dict__ = state - # self.load_module = getattr(self, 'load_module', None) - # if self.dir_path not in sys.path: - # sys.path += [self.dir_path, os.getcwd()] - # self.topology = json_graph.node_link_graph(state['topology']) - # self.network_agents = agents.calculate_distribution(agents._convert_agent_types(self.network_agents)) - # self.environment_agents = agents._convert_agent_types(self.environment_agents, - # known_modules=[self.load_module]) - # self.environment_class = serialization.deserialize(self.environment_class, - # known_modules=[self.load_module, - # 'soil.environment', ]) # func, name - -class CalculatedConfig(Config): - def __init__(self, config): - """ - Returns a configuration object that replaces some "plain" attributes (e.g., `environment_class` string) into - a Python object (`soil.environment.Environment` class). - """ - self._config = config - values = dict(config) - values['environment_class'] = self._environment_class() - values['environment_agents'] = self._environment_agents() - values['topology'] = self._topology() - values['network_agents'] = self._network_agents() - values['agent_type'] = serialization.deserialize(self.agent_type, known_modules=['soil.agents']) +from typing import Any, Callable, Dict, List, Optional, Union, Type +from pydantic import BaseModel, Extra + +class General(BaseModel): + id: str = 'Unnamed Simulation' + group: str = None + dir_path: str = None + num_trials: int = 1 + max_time: float = 100 + interval: float = 1 + seed: str = "" + + @staticmethod + def default(): + return General() + + +# Could use TypeAlias in python >= 3.10 +nodeId = int + +class Node(BaseModel): + id: nodeId + state: Dict[str, Any] + + +class Edge(BaseModel): + source: nodeId + target: nodeId + value: float = 1 + + +class Topology(BaseModel): + nodes: List[Node] + directed: bool + links: List[Edge] + + +class NetParams(BaseModel, extra=Extra.allow): + generator: Union[Callable, str] + n: int + + +class NetConfig(BaseModel): + group: str = 'network' + params: Optional[NetParams] + topology: Optional[Topology] + path: Optional[str] + + @staticmethod + def default(): + return NetConfig(topology=None, params=None) + + @root_validator + def validate_all(cls, values): + if 'params' not in values and 'topology' not in values: + raise ValueError('You must specify either a topology or the parameters to generate a graph') + return values + + +class EnvConfig(BaseModel): + environment_class: Union[Type, str] = 'soil.Environment' + params: Dict[str, Any] = {} + schedule: Union[Type, str] = 'soil.time.TimedActivation' + + @staticmethod + def default(): + return EnvConfig() + + +class SingleAgentConfig(BaseModel): + agent_class: Union[Type, str] = 'soil.Agent' + agent_id: Optional[Union[str, int]] = None + params: Dict[str, Any] = {} + state: Dict[str, Any] = {} + + +class AgentDistro(SingleAgentConfig): + weight: Optional[float] = None + n: Optional[int] = None + + @root_validator + def validate_all(cls, values): + if 'weight' in values and 'count' in values: + raise ValueError("You may either specify a weight in the distribution or an agent count") return values - def _topology(self): - topology = self._config.topology - if topology is None: - topology = serialization.load_network(self._config.network_params, - dir_path=self._config.dir_path) - elif isinstance(topology, basestring) or isinstance(topology, dict): - topology = json_graph.node_link_graph(topology) +class AgentConfig(SingleAgentConfig): + n: Optional[int] = None + distribution: Optional[List[AgentDistro]] = None + fixed: Optional[List[SingleAgentConfig]] = None + + @staticmethod + def default(): + return AgentConfig() + + +class Config(BaseModel, extra=Extra.forbid): + general: General = General.default() + network: Optional[NetConfig] = None + environment: EnvConfig = EnvConfig.default() + agents: Dict[str, AgentConfig] = {} + + +def convert_old(old): + ''' + Try to convert old style configs into the new format. + + This is still a work in progress and might not work in many cases. + ''' + new = {} + + + general = {} + for k in ['id', + 'group', + 'dir_path', + 'num_trials', + 'max_time', + 'interval', + 'seed']: + if k in old: + general[k] = old[k] + + network = {'group': 'network'} + + + if 'network_params' in old and old['network_params']: + for (k, v) in old['network_params'].items(): + if k == 'path': + network['path'] = v + else: + network.setdefault('params', {})[k] = v + + if 'topology' in old: + network['topology'] = old['topology'] + + agents = { + 'environment': { + 'fixed': [] + }, + 'network': {}, + 'default': {}, + } + + if 'agent_type' in old: + agents['default']['agent_class'] = old['agent_type'] + + if 'default_state' in old: + agents['default']['state'] = old['default_state'] + + + def updated_agent(agent): + newagent = dict(agent) + newagent['agent_class'] = newagent['agent_type'] + del newagent['agent_type'] + return newagent - return nx.Graph(topology) + for agent in old.get('environment_agents', []): + agents['environment']['fixed'].append(updated_agent(agent)) - def _environment_class(self): - return serialization.deserialize(self._config.environment_class, - known_modules=['soil.environment', ]) or Environment + for agent in old.get('network_agents', []): + agents['network'].setdefault('distribution', []).append(updated_agent(agent)) - def _environment_agents(self): - return agents._convert_agent_types(self._config.environment_agents) + environment = {'params': {}} + if 'environment_class' in old: + environment['environment_class'] = old['environment_class'] - def _network_agents(self): - distro = agents.calculate_distribution(self._config.network_agents, - self._config.agent_type) - return agents._convert_agent_types(distro) + for (k, v) in old.get('environment_params', {}).items(): + environment['params'][k] = v - def _environment_class(self): - return serialization.deserialize(self._config.environment_class, - known_modules=['soil.environment', ]) # func, name + return Config(general=general, + network=network, + environment=environment, + agents=agents) diff --git a/soil/config_old.py b/soil/config_old.py new file mode 100644 index 0000000..ca4eaa9 --- /dev/null +++ b/soil/config_old.py @@ -0,0 +1,264 @@ +from pydantic import BaseModel, ValidationError, validator + +import yaml +import os +import sys +import networkx as nx +import collections.abc + +from . import serialization, utils, basestring, agents + +class Config(collections.abc.Mapping): + """ + + 1) agent type can be specified by name or by class. + 2) instead of just one type, a network agents distribution can be used. + The distribution specifies the weight (or probability) of each + agent type in the topology. This is an example distribution: :: + + [ + {'agent_type': 'agent_type_1', + 'weight': 0.2, + 'state': { + 'id': 0 + } + }, + {'agent_type': 'agent_type_2', + 'weight': 0.8, + 'state': { + 'id': 1 + } + } + ] + + In this example, 20% of the nodes will be marked as type + 'agent_type_1'. + 3) if no initial state is given, each node's state will be set + to `{'id': 0}`. + + Parameters + --------- + name : str, optional + name of the Simulation + group : str, optional + a group name can be used to link simulations + topology (optional): networkx.Graph instance or Node-Link topology as a dict or string (will be loaded with `json_graph.node_link_graph(topology`). + network_params : dict + parameters used to create a topology with networkx, if no topology is given + network_agents : dict + definition of agents to populate the topology with + agent_type : NetworkAgent subclass, optional + Default type of NetworkAgent to use for nodes not specified in network_agents + states : list, optional + List of initial states corresponding to the nodes in the topology. Basic form is a list of integers + whose value indicates the state + dir_path: str, optional + Directory path to load simulation assets (files, modules...) + seed : str, optional + Seed to use for the random generator + num_trials : int, optional + Number of independent simulation runs + max_time : int, optional + Maximum step/time for each simulation + environment_params : dict, optional + Dictionary of globally-shared environmental parameters + environment_agents: dict, optional + Similar to network_agents. Distribution of Agents that control the environment + environment_class: soil.environment.Environment subclass, optional + Class for the environment. It defailts to soil.environment.Environment + """ + __slots__ = 'name', 'agent_type', 'group', 'description', 'network_agents', 'environment_agents', 'states', 'default_state', 'interval', 'network_params', 'seed', 'num_trials', 'max_time', 'topology', 'schedule', 'initial_time', 'environment_params', 'environment_class', 'dir_path', '_added_to_path', 'visualization_params' + + def __init__(self, name=None, + group=None, + agent_type='BaseAgent', + network_agents=None, + environment_agents=None, + states=None, + description=None, + default_state=None, + interval=1, + network_params=None, + seed=None, + num_trials=1, + max_time=None, + topology=None, + schedule=None, + initial_time=0, + environment_params={}, + environment_class='soil.Environment', + dir_path=None, + visualization_params=None, + ): + + self.network_params = network_params + self.name = name or 'Unnamed' + self.description = description or 'No simulation description available' + self.seed = str(seed or name) + self.group = group or '' + self.num_trials = num_trials + self.max_time = max_time + self.default_state = default_state or {} + self.dir_path = dir_path or os.getcwd() + self.interval = interval + self.visualization_params = visualization_params or {} + + self._added_to_path = list(x for x in [os.getcwd(), self.dir_path] if x not in sys.path) + sys.path += self._added_to_path + + self.topology = topology + + self.schedule = schedule + self.initial_time = initial_time + + + self.environment_class = environment_class + self.environment_params = dict(environment_params) + + #TODO: Check agent distro vs fixed agents + self.environment_agents = environment_agents or [] + + self.agent_type = agent_type + + self.network_agents = network_agents or {} + + self.states = states or {} + + + def validate(self): + agents._validate_states(self.states, + self._topology) + + def calculate(self): + return CalculatedConfig(self) + + def restore_path(self): + for added in self._added_to_path: + sys.path.remove(added) + + def to_yaml(self): + return yaml.dump(self.to_dict()) + + def dump_yaml(self, f=None, outdir=None): + if not f and not outdir: + raise ValueError('specify a file or an output directory') + + if not f: + f = os.path.join(outdir, '{}.dumped.yml'.format(self.name)) + + with utils.open_or_reuse(f, 'w') as f: + f.write(self.to_yaml()) + + def to_yaml(self): + return yaml.dump(self.to_dict()) + + # TODO: See note on getstate + def to_dict(self): + return dict(self) + + def __repr__(self): + return self.to_yaml() + + def dump_yaml(self, f=None, outdir=None): + if not f and not outdir: + raise ValueError('specify a file or an output directory') + + if not f: + f = os.path.join(outdir, '{}.dumped.yml'.format(self.name)) + + with utils.open_or_reuse(f, 'w') as f: + f.write(self.to_yaml()) + + def __getitem__(self, key): + return getattr(self, key) + + def __iter__(self): + return (k for k in self.__slots__ if k[0] != '_') + + def __len__(self): + return len(self.__slots__) + + def dump_pickle(self, f=None, outdir=None): + if not outdir and not f: + raise ValueError('specify a file or an output directory') + + if not f: + f = os.path.join(outdir, + '{}.simulation.pickle'.format(self.name)) + with utils.open_or_reuse(f, 'wb') as f: + pickle.dump(self, f) + + # TODO: remove this. A config should be sendable regardless. Non-pickable objects could be computed via properties and the like + # def __getstate__(self): + # state={} + # for k, v in self.__dict__.items(): + # if k[0] != '_': + # state[k] = v + # state['topology'] = json_graph.node_link_data(self.topology) + # state['network_agents'] = agents.serialize_definition(self.network_agents, + # known_modules = []) + # state['environment_agents'] = agents.serialize_definition(self.environment_agents, + # known_modules = []) + # state['environment_class'] = serialization.serialize(self.environment_class, + # known_modules=['soil.environment'])[1] # func, name + # if state['load_module'] is None: + # del state['load_module'] + # return state + + # # TODO: remove, same as __getstate__ + # def __setstate__(self, state): + # self.__dict__ = state + # self.load_module = getattr(self, 'load_module', None) + # if self.dir_path not in sys.path: + # sys.path += [self.dir_path, os.getcwd()] + # self.topology = json_graph.node_link_graph(state['topology']) + # self.network_agents = agents.calculate_distribution(agents._convert_agent_types(self.network_agents)) + # self.environment_agents = agents._convert_agent_types(self.environment_agents, + # known_modules=[self.load_module]) + # self.environment_class = serialization.deserialize(self.environment_class, + # known_modules=[self.load_module, + # 'soil.environment', ]) # func, name + +class CalculatedConfig(Config): + def __init__(self, config): + """ + Returns a configuration object that replaces some "plain" attributes (e.g., `environment_class` string) into + a Python object (`soil.environment.Environment` class). + """ + self._config = config + values = dict(config) + values['environment_class'] = self._environment_class() + values['environment_agents'] = self._environment_agents() + values['topology'] = self._topology() + values['network_agents'] = self._network_agents() + values['agent_type'] = serialization.deserialize(self.agent_type, known_modules=['soil.agents']) + + return values + + def _topology(self): + topology = self._config.topology + if topology is None: + topology = serialization.load_network(self._config.network_params, + dir_path=self._config.dir_path) + + elif isinstance(topology, basestring) or isinstance(topology, dict): + topology = json_graph.node_link_graph(topology) + + return nx.Graph(topology) + + def _environment_class(self): + return serialization.deserialize(self._config.environment_class, + known_modules=['soil.environment', ]) or Environment + + def _environment_agents(self): + return agents._convert_agent_types(self._config.environment_agents) + + def _network_agents(self): + distro = agents.calculate_distribution(self._config.network_agents, + self._config.agent_type) + return agents._convert_agent_types(distro) + + def _environment_class(self): + return serialization.deserialize(self._config.environment_class, + known_modules=['soil.environment', ]) # func, name + diff --git a/soil/environment.py b/soil/environment.py index 6635849..4e00cd9 100644 --- a/soil/environment.py +++ b/soil/environment.py @@ -16,13 +16,6 @@ from tsih import Record from . import serialization, agents, analysis, utils, time, config -# These properties will be copied when pickling/unpickling the environment -_CONFIG_PROPS = [ 'name', - 'states', - 'default_state', - 'interval', - ] - class Environment(Model): """ The environment is key in a simulation. It contains the network topology, @@ -34,76 +27,62 @@ class Environment(Model): :meth:`soil.environment.Environment.get` method. """ - def __init__(self, name=None, - network_agents=None, - environment_agents=None, - states=None, - default_state=None, - interval=1, - network_params=None, - seed=None, - topology=None, + def __init__(self, + env_id, + seed='default', schedule=None, - initial_time=0, - environment_params=None, + env_params=None, dir_path=None, **kwargs): - super().__init__() - self.schedule = schedule + + self.seed = '{}_{}'.format(seed, env_id) + self.id = env_id + + self.dir_path = dir_path or os.getcwd() + if schedule is None: - self.schedule = time.TimedActivation() + schedule = time.TimedActivation() + self.schedule = schedule - self.name = name or 'UnnamedEnvironment' seed = seed or current_time() + random.seed(seed) + if isinstance(states, list): states = dict(enumerate(states)) self.states = deepcopy(states) if states else {} self.default_state = deepcopy(default_state) or {} - if topology is None: - network_params = network_params or {} - topology = serialization.load_network(network_params, - dir_path=dir_path) - if not topology: - topology = nx.Graph() - self.G = nx.Graph(topology) - self.environment_params = environment_params or {} - self.environment_params.update(kwargs) + self.set_topology(topology=topology, + network_params=network_params) + + self.agents = agents or {} + + self.env_params = env_params or {} + self.env_params.update(kwargs) - self._env_agents = {} self.interval = interval self['SEED'] = seed - if network_agents: - distro = agents.calculate_distribution(network_agents) - self.network_agents = agents._convert_agent_types(distro) - else: - self.network_agents = [] - - environment_agents = environment_agents or [] - if environment_agents: - distro = agents.calculate_distribution(environment_agents) - environment_agents = agents._convert_agent_types(distro) - self.environment_agents = environment_agents self.logger = utils.logger.getChild(self.name) @staticmethod def from_config(conf: config.Config, trial_id, **kwargs) -> Environment: '''Create an environment for a trial of the simulation''' - - conf = config.Config(conf, **kwargs) - conf.seed = '{}_{}'.format(conf.seed, trial_id) - conf.name = '{}_trial_{}'.format(conf.name, trial_id).replace('.', '-') - opts = conf.environment_params.copy() + conf = conf + if kwargs: + conf = config.Config(**conf.dict(exclude_defaults=True), **kwargs) + seed = '{}_{}'.format(conf.general.seed, trial_id) + id = '{}_trial_{}'.format(conf.general.id, trial_id).replace('.', '-') + opts = conf.environment.params.copy() opts.update(conf) opts.update(kwargs) - env = serialization.deserialize(conf.environment_class)(**opts) + env = serialization.deserialize(conf.environment.environment_class)(env_id=id, seed=seed, **opts) return env @property @@ -112,21 +91,30 @@ class Environment(Model): return self.schedule.time raise Exception('The environment has not been scheduled, so it has no sense of time') - @property - def agents(self): - yield from self.environment_agents - yield from self.network_agents + + def set_topology(self, topology, network_params=None, dir_path=None): + if topology is None: + network_params = network_params or {} + topology = serialization.load_network(network_params, + dir_path=dir_path or self.dir_path) + if not topology: + topology = nx.Graph() + self.G = nx.Graph(topology) @property - def environment_agents(self): - for ref in self._env_agents.values(): - yield ref + def agents(self): + for agents in self.agents.values(): + yield from agents - @environment_agents.setter - def environment_agents(self, environment_agents): - self._environment_agents = environment_agents + @agents.setter + def agents(self, agents): + self.agents = {} - self._env_agents = agents._definition_to_dict(definition=environment_agents) + for (k, v) in agents.items(): + self.agents[k] = agents.from_config(v) + for agent in self.agents.get('network', []): + node = self.G.nodes[agent.unique_id] + node['agent'] = agent @property def network_agents(self): @@ -135,12 +123,6 @@ class Environment(Model): if 'agent' in node: yield node['agent'] - @network_agents.setter - def network_agents(self, network_agents): - self._network_agents = network_agents - for ix in self.G.nodes(): - self.init_agent(ix, agent_definitions=network_agents) - def init_agent(self, agent_id, agent_definitions): node = self.G.nodes[agent_id] init = False @@ -251,20 +233,20 @@ class Environment(Model): value=value) def __contains__(self, key): - return key in self.environment_params + return key in self.env_params def get(self, key, default=None): ''' Get the value of an environment attribute. Return `default` if the value is not set. ''' - return self.environment_params.get(key, default) + return self.env_params.get(key, default) def __getitem__(self, key): - return self.environment_params.get(key) + return self.env_params.get(key) def __setitem__(self, key, value): - return self.environment_params.__setitem__(key, value) + return self.env_params.__setitem__(key, value) def get_agent(self, agent_id): return self.G.nodes[agent_id]['agent'] @@ -292,7 +274,7 @@ class Environment(Model): yield from self._agent_to_tuples(agent, now) return - for k, v in self.environment_params.items(): + for k, v in self.env_params.items(): yield Record(dict_id='env', t_step=now, key=k, @@ -300,23 +282,5 @@ class Environment(Model): for agent in self.agents: yield from self._agent_to_tuples(agent, now) - def __getstate__(self): - state = {} - for prop in _CONFIG_PROPS: - state[prop] = self.__dict__[prop] - state['G'] = json_graph.node_link_data(self.G) - state['environment_agents'] = self._env_agents - state['schedule'] = self.schedule - return state - - def __setstate__(self, state): - for prop in _CONFIG_PROPS: - self.__dict__[prop] = state[prop] - self._env_agents = state['environment_agents'] - self.G = json_graph.node_link_graph(state['G']) - # self._env = None - self.schedule = state['schedule'] - self._queue = [] - SoilEnvironment = Environment diff --git a/soil/exporters.py b/soil/exporters.py index f2bbe78..0653517 100644 --- a/soil/exporters.py +++ b/soil/exporters.py @@ -2,6 +2,8 @@ import os import csv as csvlib from time import time as current_time from io import BytesIO +from sqlalchemy import create_engine + import matplotlib.pyplot as plt import networkx as nx @@ -48,8 +50,8 @@ class Exporter: self.simulation = simulation outdir = outdir or os.path.join(os.getcwd(), 'soil_output') self.outdir = os.path.join(outdir, - simulation.config.group or '', - simulation.config.name) + simulation.config.general.group or '', + simulation.config.general.id) self.dry_run = dry_run self.copy_to = copy_to @@ -84,24 +86,33 @@ class Exporter: class default(Exporter): '''Default exporter. Writes sqlite results, as well as the simulation YAML''' - def sim_start(self): - if not self.dry_run: - logger.info('Dumping results to %s', self.outdir) - self.simulation.dump_yaml(outdir=self.outdir) - else: - logger.info('NOT dumping results') - - def trial_start(self, env, stats): - if not self.dry_run: - with timer('Dumping simulation {} trial {}'.format(self.simulation.name, - env.name)): - with self.output('{}.sqlite'.format(env.name), mode='wb') as f: - env.dump_sqlite(f) - - def sim_end(self, stats): - with timer('Dumping simulation {}\'s stats'.format(self.simulation.name)): - with self.output('{}.sqlite'.format(self.simulation.name), mode='wb') as f: - self.simulation.dump_sqlite(f) + # def sim_start(self): + # if not self.dry_run: + # logger.info('Dumping results to %s', self.outdir) + # self.simulation.dump_yaml(outdir=self.outdir) + # else: + # logger.info('NOT dumping results') + + # def trial_start(self, env, stats): + # if not self.dry_run: + # with timer('Dumping simulation {} trial {}'.format(self.simulation.name, + # env.name)): + # engine = create_engine('sqlite:///{}.sqlite'.format(env.name), echo=False) + + # dc = env.datacollector + # tables = {'env': dc.get_model_vars_dataframe(), + # 'agents': dc.get_agent_vars_dataframe(), + # 'agents': dc.get_agent_vars_dataframe()} + # for table in dc.tables: + # tables[table] = dc.get_table_dataframe(table) + # for (t, df) in tables.items(): + # df.to_sql(t, con=engine) + + # def sim_end(self, stats): + # with timer('Dumping simulation {}\'s stats'.format(self.simulation.name)): + # engine = create_engine('sqlite:///{}.sqlite'.format(self.simulation.name), echo=False) + # with self.output('{}.sqlite'.format(self.simulation.name), mode='wb') as f: + # self.simulation.dump_sqlite(f) diff --git a/soil/serialization.py b/soil/serialization.py index dd94108..3a10de1 100644 --- a/soil/serialization.py +++ b/soil/serialization.py @@ -51,8 +51,6 @@ def load_network(network_params, dir_path=None): return G - - def load_file(infile): folder = os.path.dirname(infile) if folder not in sys.path: @@ -138,7 +136,9 @@ def load_config(config): builtins = importlib.import_module('builtins') -def name(value, known_modules=[]): +KNOWN_MODULES = ['soil', ] + +def name(value, known_modules=KNOWN_MODULES): '''Return a name that can be imported, to serialize/deserialize an object''' if value is None: return 'None' @@ -167,7 +167,7 @@ def serializer(type_): return lambda x: x -def serialize(v, known_modules=[]): +def serialize(v, known_modules=KNOWN_MODULES): '''Get a text representation of an object.''' tname = name(v, known_modules=known_modules) func = serializer(tname) @@ -176,7 +176,7 @@ def serialize(v, known_modules=[]): IS_CLASS = re.compile(r"") -def deserializer(type_, known_modules=[]): +def deserializer(type_, known_modules=KNOWN_MODULES): if type(type_) != str: # Already deserialized return type_ if type_ == 'str': @@ -194,10 +194,9 @@ def deserializer(type_, known_modules=[]): return getattr(cls, 'deserialize', cls) # Otherwise, see if we can find the module and the class - modules = known_modules or [] options = [] - for mod in modules: + for mod in known_modules: if mod: options.append((mod, type_)) @@ -226,7 +225,7 @@ def deserialize(type_, value=None, **kwargs): return des(value) -def deserialize_all(names, *args, known_modules=['soil'], **kwargs): +def deserialize_all(names, *args, known_modules=KNOWN_MODULES, **kwargs): '''Return the list of deserialized objects''' objects = [] for name in names: diff --git a/soil/simulation.py b/soil/simulation.py index 9985d3e..2f346ca 100644 --- a/soil/simulation.py +++ b/soil/simulation.py @@ -18,7 +18,7 @@ from .utils import logger from .exporters import default from .stats import defaultStats -from .config import Config +from .config import Config, convert_old #TODO: change documentation for simulation @@ -34,18 +34,21 @@ class Simulation: def __init__(self, config=None, **kwargs): - - if bool(config) == bool(kwargs): - raise ValueError("Specify either a configuration or the parameters to initialize a configuration") - if kwargs: - config = Config(**kwargs) + cfg = {} + if config: + cfg.update(config.dict(include_defaults=False)) + cfg.update(kwargs) + config = Config(**cfg) + if not config: + raise ValueError("You need to specify a simulation configuration") self.config = config + @property def name(self) -> str: - return self.config.name + return self.config.general.id def run_simulation(self, *args, **kwargs): return self.run(*args, **kwargs) @@ -58,13 +61,13 @@ class Simulation: if parallel and not os.environ.get('SENPY_DEBUG', None): p = Pool() func = partial(self.run_trial_exceptions, **kwargs) - for i in p.imap_unordered(func, range(self.config.num_trials)): + for i in p.imap_unordered(func, range(self.config.general.num_trials)): if isinstance(i, Exception): logger.error('Trial failed:\n\t%s', i.message) continue yield i else: - for i in range(self.config.num_trials): + for i in range(self.config.general.num_trials): yield self.run_trial(trial_id=i, **kwargs) @@ -88,7 +91,7 @@ class Simulation: known_modules=['soil.stats',], **stats_params) - with utils.timer('simulation {}'.format(self.config.name)): + with utils.timer('simulation {}'.format(self.config.general.id)): for stat in stats: stat.sim_start() @@ -157,11 +160,11 @@ class Simulation: if log_level: logger.setLevel(log_level) # Set-up trial environment and graph - until = until or self.config.max_time + until = until or self.config.general.max_time env = Environment.from_config(self.config, trial_id=trial_id) # Set up agents on nodes - with utils.timer('Simulation {} trial {}'.format(self.config.name, trial_id)): + with utils.timer('Simulation {} trial {}'.format(self.config.general.id, trial_id)): env.run(until) return env @@ -194,15 +197,22 @@ def from_config(conf_or_path): sim = Simulation(**config) return sim +def from_old_config(conf_or_path): + config = list(serialization.load_config(conf_or_path)) + if len(config) > 1: + raise AttributeError('Provide only one configuration') + config = convert_old(config[0][0]) + return Simulation(config) + def run_from_config(*configs, **kwargs): for config_def in configs: # logger.info("Found {} config(s)".format(len(ls))) for config, path in serialization.load_config(config_def): - name = config.get('name', 'unnamed') + name = config.general.id logger.info("Using config(s): {name}".format(name=name)) - dir_path = config.pop('dir_path', os.path.dirname(path)) + dir_path = config.general.dir_path or os.path.dirname(path) sim = Simulation(dir_path=dir_path, **config) sim.run_simulation(**kwargs) diff --git a/tests/old_complete.yml b/tests/old_complete.yml new file mode 100644 index 0000000..4382935 --- /dev/null +++ b/tests/old_complete.yml @@ -0,0 +1,32 @@ +--- +name: simple +group: tests +dir_path: "/tmp/" +num_trials: 3 +max_time: 100 +interval: 1 +seed: "CompleteSeed!" +network_params: + generator: complete_graph + n: 10 +network_agents: + - agent_type: CounterModel + weight: 1 + state: + state_id: 0 + - agent_type: AggregatedCounter + weight: 0.2 +environment_agents: + - agent_id: 'Environment Agent 1' + agent_type: CounterModel + state: + times: 10 +environment_class: Environment +environment_params: + am_i_complete: true +agent_type: CounterModel +default_state: + times: 1 +states: + - name: 'The first node' + - name: 'The second node' diff --git a/tests/test_config.py b/tests/test_config.py new file mode 100644 index 0000000..359e8ad --- /dev/null +++ b/tests/test_config.py @@ -0,0 +1,62 @@ +from unittest import TestCase +import os +from os.path import join + +from soil import serialization, config + +ROOT = os.path.abspath(os.path.dirname(__file__)) +EXAMPLES = join(ROOT, '..', 'examples') + +FORCE_TESTS = os.environ.get('FORCE_TESTS', '') + + +class TestConfig(TestCase): + + def test_conversion(self): + new = serialization.load_file(join(EXAMPLES, "complete.yml"))[0] + old = serialization.load_file(join(ROOT, "old_complete.yml"))[0] + converted = config.convert_old(old).dict(skip_defaults=True) + for (k, v) in new.items(): + assert v == converted[k] + + +def make_example_test(path, cfg): + def wrapped(self): + root = os.getcwd() + s = config.Config(**cfg) + import pdb;pdb.set_trace() + # for s in simulation.all_from_config(path): + # iterations = s.config.max_time * s.config.num_trials + # if iterations > 1000: + # s.config.max_time = 100 + # s.config.num_trials = 1 + # if config.get('skip_test', False) and not FORCE_TESTS: + # self.skipTest('Example ignored.') + # envs = s.run_simulation(dry_run=True) + # assert envs + # for env in envs: + # assert env + # try: + # n = config['network_params']['n'] + # assert len(list(env.network_agents)) == n + # assert env.now > 0 # It has run + # assert env.now <= config['max_time'] # But not further than allowed + # except KeyError: + # pass + return wrapped + + +def add_example_tests(): + for config, path in serialization.load_files( + join(EXAMPLES, '*', '*.yml'), + join(EXAMPLES, '*.yml'), + ): + p = make_example_test(path=path, cfg=config) + fname = os.path.basename(path) + p.__name__ = 'test_example_file_%s' % fname + p.__doc__ = '%s should be a valid configuration' % fname + setattr(TestConfig, p.__name__, p) + del p + + +add_example_tests() diff --git a/tests/test_examples.py b/tests/test_examples.py index 6a00367..a516c27 100644 --- a/tests/test_examples.py +++ b/tests/test_examples.py @@ -18,10 +18,10 @@ def make_example_test(path, config): def wrapped(self): root = os.getcwd() for s in simulation.all_from_config(path): - iterations = s.max_time * s.num_trials + iterations = s.config.max_time * s.config.num_trials if iterations > 1000: - s.max_time = 100 - s.num_trials = 1 + s.config.max_time = 100 + s.config.num_trials = 1 if config.get('skip_test', False) and not FORCE_TESTS: self.skipTest('Example ignored.') envs = s.run_simulation(dry_run=True) diff --git a/tests/test_history.py b/tests/test_history.py new file mode 100644 index 0000000..4a76d5c --- /dev/null +++ b/tests/test_history.py @@ -0,0 +1,128 @@ +from unittest import TestCase + +import os +import io +import yaml +import copy +import pickle +import networkx as nx +from functools import partial + +from os.path import join +from soil import (simulation, Environment, agents, serialization, + utils) +from soil.time import Delta +from tsih import NoHistory, History + + +ROOT = os.path.abspath(os.path.dirname(__file__)) +EXAMPLES = join(ROOT, '..', 'examples') + + +class CustomAgent(agents.FSM): + @agents.default_state + @agents.state + def normal(self): + self.neighbors = self.count_agents(state_id='normal', + limit_neighbors=True) + @agents.state + def unreachable(self): + return + +class TestHistory(TestCase): + + def test_counter_agent_history(self): + """ + The evolution of the state should be recorded in the logging agent + """ + config = { + 'name': 'CounterAgent', + 'network_params': { + 'path': join(ROOT, 'test.gexf') + }, + 'network_agents': [{ + 'agent_type': 'AggregatedCounter', + 'weight': 1, + 'state': {'state_id': 0} + + }], + 'max_time': 10, + 'environment_params': { + } + } + s = simulation.from_config(config) + env = s.run_simulation(dry_run=True)[0] + for agent in env.network_agents: + last = 0 + assert len(agent[None, None]) == 11 + for step, total in sorted(agent['total', None]): + assert total == last + 2 + last = total + + def test_row_conversion(self): + env = Environment(history=True) + env['test'] = 'test_value' + + res = list(env.history_to_tuples()) + assert len(res) == len(env.environment_params) + + env.schedule.time = 1 + env['test'] = 'second_value' + res = list(env.history_to_tuples()) + + assert env['env', 0, 'test' ] == 'test_value' + assert env['env', 1, 'test' ] == 'second_value' + + def test_nohistory(self): + ''' + Make sure that no history(/sqlite) is used by default + ''' + env = Environment(topology=nx.Graph(), network_agents=[]) + assert isinstance(env._history, NoHistory) + + def test_save_graph_history(self): + ''' + The history_to_graph method should return a valid networkx graph. + + The state of the agent should be encoded as intervals in the nx graph. + ''' + G = nx.cycle_graph(5) + distribution = agents.calculate_distribution(None, agents.BaseAgent) + env = Environment(topology=G, network_agents=distribution, history=True) + env[0, 0, 'testvalue'] = 'start' + env[0, 10, 'testvalue'] = 'finish' + nG = env.history_to_graph() + values = nG.nodes[0]['attr_testvalue'] + assert ('start', 0, 10) in values + assert ('finish', 10, None) in values + + def test_save_graph_nohistory(self): + ''' + The history_to_graph method should return a valid networkx graph. + + When NoHistory is used, only the last known value is known + ''' + G = nx.cycle_graph(5) + distribution = agents.calculate_distribution(None, agents.BaseAgent) + env = Environment(topology=G, network_agents=distribution, history=False) + env.get_agent(0)['testvalue'] = 'start' + env.schedule.time = 10 + env.get_agent(0)['testvalue'] = 'finish' + nG = env.history_to_graph() + values = nG.nodes[0]['attr_testvalue'] + assert ('start', 0, None) not in values + assert ('finish', 10, None) in values + + def test_pickle_agent_environment(self): + env = Environment(name='Test', history=True) + a = agents.BaseAgent(model=env, unique_id=25) + + a['key'] = 'test' + + pickled = pickle.dumps(a) + recovered = pickle.loads(pickled) + + assert recovered.env.name == 'Test' + assert list(recovered.env._history.to_tuples()) + assert recovered['key', 0] == 'test' + assert recovered['key'] == 'test' diff --git a/tests/test_main.py b/tests/test_main.py index 15ad035..2e89a2d 100644 --- a/tests/test_main.py +++ b/tests/test_main.py @@ -3,6 +3,7 @@ from unittest import TestCase import os import io import yaml +import copy import pickle import networkx as nx from functools import partial @@ -11,8 +12,6 @@ from os.path import join from soil import (simulation, Environment, agents, serialization, utils) from soil.time import Delta -from tsih import NoHistory, History - ROOT = os.path.abspath(os.path.dirname(__file__)) EXAMPLES = join(ROOT, '..', 'examples') @@ -79,59 +78,53 @@ class TestMain(TestCase): 'environment_params': { } } - s = simulation.from_config(config) + s = simulation.from_old_config(config) s.run_simulation(dry_run=True) - def test_counter_agent(self): + + def test_network_agent(self): """ The initial states should be applied to the agent and the agent should be able to update its state.""" config = { 'name': 'CounterAgent', 'network_params': { - 'path': join(ROOT, 'test.gexf') + 'generator': nx.complete_graph, + 'n': 2, }, 'agent_type': 'CounterModel', - 'states': [{'times': 10}, {'times': 20}], + 'states': { + 0: {'times': 10}, + 1: {'times': 20}, + }, 'max_time': 2, 'num_trials': 1, 'environment_params': { } } - s = simulation.from_config(config) - env = s.run_simulation(dry_run=True)[0] - assert env.get_agent(0)['times', 0] == 11 - assert env.get_agent(0)['times', 1] == 12 - assert env.get_agent(1)['times', 0] == 21 - assert env.get_agent(1)['times', 1] == 22 - - def test_counter_agent_history(self): - """ - The evolution of the state should be recorded in the logging agent + s = simulation.from_old_config(config) + def test_counter_agent(self): """ + The initial states should be applied to the agent and the + agent should be able to update its state.""" config = { 'name': 'CounterAgent', 'network_params': { 'path': join(ROOT, 'test.gexf') }, - 'network_agents': [{ - 'agent_type': 'AggregatedCounter', - 'weight': 1, - 'state': {'state_id': 0} - - }], - 'max_time': 10, + 'agent_type': 'CounterModel', + 'states': [{'times': 10}, {'times': 20}], + 'max_time': 2, + 'num_trials': 1, 'environment_params': { } } - s = simulation.from_config(config) + s = simulation.from_old_config(config) env = s.run_simulation(dry_run=True)[0] - for agent in env.network_agents: - last = 0 - assert len(agent[None, None]) == 10 - for step, total in sorted(agent['total', None]): - assert total == last + 2 - last = total + assert env.get_agent(0)['times', 0] == 11 + assert env.get_agent(0)['times', 1] == 12 + assert env.get_agent(1)['times', 0] == 21 + assert env.get_agent(1)['times', 1] == 22 def test_custom_agent(self): """Allow for search of neighbors with a certain state_id""" @@ -148,7 +141,7 @@ class TestMain(TestCase): 'environment_params': { } } - s = simulation.from_config(config) + s = simulation.from_old_config(config) env = s.run_simulation(dry_run=True)[0] assert env.get_agent(1).count_agents(state_id='normal') == 2 assert env.get_agent(1).count_agents(state_id='normal', limit_neighbors=True) == 1 @@ -159,7 +152,7 @@ class TestMain(TestCase): config = serialization.load_file(join(EXAMPLES, 'torvalds.yml'))[0] config['network_params']['path'] = join(EXAMPLES, config['network_params']['path']) - s = simulation.from_config(config) + s = simulation.from_old_config(config) env = s.run_simulation(dry_run=True)[0] for a in env.network_agents: skill_level = a.state['skill_level'] @@ -178,19 +171,23 @@ class TestMain(TestCase): def test_yaml(self): """ - The YAML version of a newly created simulation - should be equivalent to the configuration file used + The YAML version of a newly created configuration should be equivalent + to the configuration file used. + Values not present in the original config file should have reasonable + defaults. """ with utils.timer('loading'): config = serialization.load_file(join(EXAMPLES, 'complete.yml'))[0] - s = simulation.from_config(config) + s = simulation.from_old_config(config) with utils.timer('serializing'): - serial = s.to_yaml() + serial = s.config.to_yaml() with utils.timer('recovering'): recovered = yaml.load(serial, Loader=yaml.SafeLoader) with utils.timer('deleting'): del recovered['topology'] - assert config == recovered + for (k, v) in config.items(): + assert recovered[k] == v + # assert config == recovered def test_configuration_changes(self): """ @@ -198,26 +195,13 @@ class TestMain(TestCase): the simulation. """ config = serialization.load_file(join(EXAMPLES, 'complete.yml'))[0] - s = simulation.from_config(config) + s = simulation.from_old_config(config) + init_config = copy.copy(s.config) s.run_simulation(dry_run=True) - nconfig = s.to_dict() - del nconfig['topology'] - assert config == nconfig - - def test_row_conversion(self): - env = Environment(history=True) - env['test'] = 'test_value' - - res = list(env.history_to_tuples()) - assert len(res) == len(env.environment_params) - - env.schedule.time = 1 - env['test'] = 'second_value' - res = list(env.history_to_tuples()) - - assert env['env', 0, 'test' ] == 'test_value' - assert env['env', 1, 'test' ] == 'second_value' + nconfig = s.config + # del nconfig['to + assert init_config == nconfig def test_save_geometric(self): """ @@ -229,51 +213,15 @@ class TestMain(TestCase): f = io.BytesIO() env.dump_gexf(f) - def test_nohistory(self): - ''' - Make sure that no history(/sqlite) is used by default - ''' - env = Environment(topology=nx.Graph(), network_agents=[]) - assert isinstance(env._history, NoHistory) - - def test_save_graph_history(self): - ''' - The history_to_graph method should return a valid networkx graph. - - The state of the agent should be encoded as intervals in the nx graph. - ''' - G = nx.cycle_graph(5) - distribution = agents.calculate_distribution(None, agents.BaseAgent) - env = Environment(topology=G, network_agents=distribution, history=True) - env[0, 0, 'testvalue'] = 'start' - env[0, 10, 'testvalue'] = 'finish' - nG = env.history_to_graph() - values = nG.nodes[0]['attr_testvalue'] - assert ('start', 0, 10) in values - assert ('finish', 10, None) in values - - def test_save_graph_nohistory(self): - ''' - The history_to_graph method should return a valid networkx graph. - - When NoHistory is used, only the last known value is known - ''' - G = nx.cycle_graph(5) - distribution = agents.calculate_distribution(None, agents.BaseAgent) - env = Environment(topology=G, network_agents=distribution, history=False) - env.get_agent(0)['testvalue'] = 'start' - env.schedule.time = 10 - env.get_agent(0)['testvalue'] = 'finish' - nG = env.history_to_graph() - values = nG.nodes[0]['attr_testvalue'] - assert ('start', 0, None) not in values - assert ('finish', 10, None) in values - def test_serialize_class(self): - ser, name = serialization.serialize(agents.BaseAgent) + ser, name = serialization.serialize(agents.BaseAgent, known_modules=[]) assert name == 'soil.agents.BaseAgent' assert ser == agents.BaseAgent + ser, name = serialization.serialize(agents.BaseAgent, known_modules=['soil', ]) + assert name == 'BaseAgent' + assert ser == agents.BaseAgent + ser, name = serialization.serialize(CustomAgent) assert name == 'test_main.CustomAgent' assert ser == CustomAgent @@ -327,20 +275,6 @@ class TestMain(TestCase): assert converted[1]['agent_type'] == 'test_main.CustomAgent' pickle.dumps(converted) - def test_pickle_agent_environment(self): - env = Environment(name='Test', history=True) - a = agents.BaseAgent(model=env, unique_id=25) - - a['key'] = 'test' - - pickled = pickle.dumps(a) - recovered = pickle.loads(pickled) - - assert recovered.env.name == 'Test' - assert list(recovered.env._history.to_tuples()) - assert recovered['key', 0] == 'test' - assert recovered['key'] == 'test' - def test_subgraph(self): '''An agent should be able to subgraph the global topology''' G = nx.Graph() @@ -371,7 +305,7 @@ class TestMain(TestCase): 'num_trials': 50, 'environment_params': {} } - s = simulation.from_config(config) + s = simulation.from_old_config(config) runs = list(s.run_simulation(dry_run=True)) over = list(x.now for x in runs if x.now>2) assert len(runs) == config['num_trials']