mirror of https://github.com/gsi-upm/senpy
Macro commit
* Fixed Options for extra_params in UI * Enhanced meta-programming for models * Plugins can be imported from a python file if they're named `senpy_<whatever>.py>` (no need for `.senpy` anymore!) * Add docstings and tests to most plugins * Read plugin description from the docstring * Refactor code to get rid of unnecessary `.senpy`s * Load models, plugins and utils into the main namespace (see __init__.py) * Enhanced plugin development/experience with utils (easy_test, easy_serve) * Fix bug in check_template that wouldn't check objects * Make model defaults a private variable * Add option to list loaded plugins in CLI * Update docspre-1.0
parent
abd401f863
commit
21a5a3f201
@ -0,0 +1,113 @@
|
|||||||
|
Advanced plugin definition
|
||||||
|
--------------------------
|
||||||
|
In addition to finding plugins defined in source code files, senpy can also load a special type of definition file (`.senpy` files).
|
||||||
|
This used to be the only mechanism for loading in earlier versions of senpy.
|
||||||
|
|
||||||
|
The definition file contains basic information
|
||||||
|
|
||||||
|
Lastly, it is also possible to add new plugins programmatically.
|
||||||
|
|
||||||
|
.. contents:: :local:
|
||||||
|
|
||||||
|
What is a plugin?
|
||||||
|
=================
|
||||||
|
|
||||||
|
A plugin is a program that, given a text, will add annotations to it.
|
||||||
|
In practice, a plugin consists of at least two files:
|
||||||
|
|
||||||
|
- Definition file: a `.senpy` file that describes the plugin (e.g. what input parameters it accepts, what emotion model it uses).
|
||||||
|
- Python module: the actual code that will add annotations to each input.
|
||||||
|
|
||||||
|
This separation allows us to deploy plugins that use the same code but employ different parameters.
|
||||||
|
For instance, one could use the same classifier and processing in several plugins, but train with different datasets.
|
||||||
|
This scenario is particularly useful for evaluation purposes.
|
||||||
|
|
||||||
|
The only limitation is that the name of each plugin needs to be unique.
|
||||||
|
|
||||||
|
Definition files
|
||||||
|
================
|
||||||
|
|
||||||
|
The definition file complements and overrides the attributes provided by the plugin.
|
||||||
|
It can be written in YAML or JSON.
|
||||||
|
The most important attributes are:
|
||||||
|
|
||||||
|
* **name**: unique name that senpy will use internally to identify the plugin.
|
||||||
|
* **module**: indicates the module that contains the plugin code, which will be automatically loaded by senpy.
|
||||||
|
* **version**
|
||||||
|
* extra_params: to add parameters to the senpy API when this plugin is requested. Those parameters may be required, and have aliased names. For instance:
|
||||||
|
|
||||||
|
.. code:: yaml
|
||||||
|
|
||||||
|
extra_params:
|
||||||
|
hello_param:
|
||||||
|
aliases: # required
|
||||||
|
- hello_param
|
||||||
|
- hello
|
||||||
|
required: true
|
||||||
|
default: Hi you
|
||||||
|
values:
|
||||||
|
- Hi you
|
||||||
|
- Hello y'all
|
||||||
|
- Howdy
|
||||||
|
|
||||||
|
A complete example:
|
||||||
|
|
||||||
|
.. code:: yaml
|
||||||
|
|
||||||
|
name: <Name of the plugin>
|
||||||
|
module: <Python file>
|
||||||
|
version: 0.1
|
||||||
|
|
||||||
|
And the json equivalent:
|
||||||
|
|
||||||
|
.. code:: json
|
||||||
|
|
||||||
|
{
|
||||||
|
"name": "<Name of the plugin>",
|
||||||
|
"module": "<Python file>",
|
||||||
|
"version": "0.1"
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
Example plugin with a definition file
|
||||||
|
=====================================
|
||||||
|
|
||||||
|
In this section, we will implement a basic sentiment analysis plugin.
|
||||||
|
To determine the polarity of each entry, the plugin will compare the length of the string to a threshold.
|
||||||
|
This threshold will be included in the definition file.
|
||||||
|
|
||||||
|
The definition file would look like this:
|
||||||
|
|
||||||
|
.. code:: yaml
|
||||||
|
|
||||||
|
name: helloworld
|
||||||
|
module: helloworld
|
||||||
|
version: 0.0
|
||||||
|
threshold: 10
|
||||||
|
description: Hello World
|
||||||
|
|
||||||
|
Now, in a file named ``helloworld.py``:
|
||||||
|
|
||||||
|
.. code:: python
|
||||||
|
|
||||||
|
#!/bin/env python
|
||||||
|
#helloworld.py
|
||||||
|
|
||||||
|
from senpy import AnalysisPlugin
|
||||||
|
from senpy import Sentiment
|
||||||
|
|
||||||
|
|
||||||
|
class HelloWorld(AnalysisPlugin):
|
||||||
|
|
||||||
|
def analyse_entry(entry, params):
|
||||||
|
'''Basically do nothing with each entry'''
|
||||||
|
|
||||||
|
sentiment = Sentiment()
|
||||||
|
if len(entry.text) < self.threshold:
|
||||||
|
sentiment['marl:hasPolarity'] = 'marl:Positive'
|
||||||
|
else:
|
||||||
|
sentiment['marl:hasPolarity'] = 'marl:Negative'
|
||||||
|
entry.sentiments.append(sentiment)
|
||||||
|
yield entry
|
||||||
|
|
||||||
|
The complete code of the example plugin is available `here <https://lab.cluster.gsi.dit.upm.es/senpy/plugin-prueba>`__.
|
@ -0,0 +1,13 @@
|
|||||||
|
This is a collection of plugins that exemplify certain aspects of plugin development with senpy.
|
||||||
|
In ascending order of complexity, there are:
|
||||||
|
|
||||||
|
* Basic: a very basic analysis that does sentiment analysis based on emojis.
|
||||||
|
* Configurable: a version of `basic` with a configurable map of emojis for each sentiment.
|
||||||
|
* Parameterized: like `basic_info`, but users set the map in each query (via `extra_parameters`).
|
||||||
|
* mynoop: shows how to add a definition file with external requirements for a plugin. Doing this with a python-only module would require moving all imports of the requirements to their functions, which is considered bad practice.
|
||||||
|
* Async: a barebones example of training a plugin and analyzing data in parallel.
|
||||||
|
|
||||||
|
All of the plugins in this folder include a set of test cases and they are periodically tested with the latest version of senpy.
|
||||||
|
|
||||||
|
Additioanlly, for an example of stand-alone plugin that can be tested and deployed with docker, take a look at: lab.cluster.gsi.dit.upm.es/senpy/plugin-example
|
||||||
|
bbm
|
@ -1,8 +0,0 @@
|
|||||||
---
|
|
||||||
name: Async
|
|
||||||
module: asyncplugin
|
|
||||||
description: I am async
|
|
||||||
author: "@balkian"
|
|
||||||
version: '0.1'
|
|
||||||
async: true
|
|
||||||
extra_params: {}
|
|
@ -0,0 +1,23 @@
|
|||||||
|
#!/usr/local/bin/python
|
||||||
|
# coding: utf-8
|
||||||
|
|
||||||
|
emoticons = {
|
||||||
|
'marl:Positive': [':)', ':]', '=)', ':D'],
|
||||||
|
'marl:Negative': [':(', ':[', '=(']
|
||||||
|
}
|
||||||
|
|
||||||
|
emojis = {
|
||||||
|
'marl:Positive': ['😁', '😂', '😃', '😄', '😆', '😅', '😄' '😍'],
|
||||||
|
'marl:Negative': ['😢', '😡', '😠', '😞', '😖', '😔', '😓', '😒']
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def get_polarity(text, dictionaries=[emoticons, emojis]):
|
||||||
|
polarity = 'marl:Neutral'
|
||||||
|
for dictionary in dictionaries:
|
||||||
|
for label, values in dictionary.items():
|
||||||
|
for emoticon in values:
|
||||||
|
if emoticon and emoticon in text:
|
||||||
|
polarity = label
|
||||||
|
break
|
||||||
|
return polarity
|
@ -0,0 +1,40 @@
|
|||||||
|
#!/usr/local/bin/python
|
||||||
|
# coding: utf-8
|
||||||
|
|
||||||
|
from senpy import easy_test, models, plugins
|
||||||
|
|
||||||
|
import basic
|
||||||
|
|
||||||
|
|
||||||
|
class Basic(plugins.SentimentPlugin):
|
||||||
|
'''Provides sentiment annotation using a lexicon'''
|
||||||
|
|
||||||
|
author = '@balkian'
|
||||||
|
version = '0.1'
|
||||||
|
|
||||||
|
def analyse_entry(self, entry, params):
|
||||||
|
|
||||||
|
polarity = basic.get_polarity(entry.text)
|
||||||
|
|
||||||
|
s = models.Sentiment(marl__hasPolarity=polarity)
|
||||||
|
s.prov(self)
|
||||||
|
entry.sentiments.append(s)
|
||||||
|
yield entry
|
||||||
|
|
||||||
|
test_cases = [{
|
||||||
|
'input': 'Hello :)',
|
||||||
|
'polarity': 'marl:Positive'
|
||||||
|
}, {
|
||||||
|
'input': 'So sad :(',
|
||||||
|
'polarity': 'marl:Negative'
|
||||||
|
}, {
|
||||||
|
'input': 'Yay! Emojis 😁',
|
||||||
|
'polarity': 'marl:Positive'
|
||||||
|
}, {
|
||||||
|
'input': 'But no emoticons 😢',
|
||||||
|
'polarity': 'marl:Negative'
|
||||||
|
}]
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
easy_test()
|
@ -0,0 +1,102 @@
|
|||||||
|
#!/usr/local/bin/python
|
||||||
|
# coding: utf-8
|
||||||
|
|
||||||
|
from senpy import easy_test, models, plugins
|
||||||
|
|
||||||
|
import basic
|
||||||
|
|
||||||
|
|
||||||
|
class Dictionary(plugins.SentimentPlugin):
|
||||||
|
'''Sentiment annotation using a configurable lexicon'''
|
||||||
|
|
||||||
|
author = '@balkian'
|
||||||
|
version = '0.2'
|
||||||
|
|
||||||
|
dictionaries = [basic.emojis, basic.emoticons]
|
||||||
|
|
||||||
|
def analyse_entry(self, entry, params):
|
||||||
|
polarity = basic.get_polarity(entry.text, self.dictionaries)
|
||||||
|
|
||||||
|
s = models.Sentiment(marl__hasPolarity=polarity)
|
||||||
|
s.prov(self)
|
||||||
|
entry.sentiments.append(s)
|
||||||
|
yield entry
|
||||||
|
|
||||||
|
test_cases = [{
|
||||||
|
'input': 'Hello :)',
|
||||||
|
'polarity': 'marl:Positive'
|
||||||
|
}, {
|
||||||
|
'input': 'So sad :(',
|
||||||
|
'polarity': 'marl:Negative'
|
||||||
|
}, {
|
||||||
|
'input': 'Yay! Emojis 😁',
|
||||||
|
'polarity': 'marl:Positive'
|
||||||
|
}, {
|
||||||
|
'input': 'But no emoticons 😢',
|
||||||
|
'polarity': 'marl:Negative'
|
||||||
|
}]
|
||||||
|
|
||||||
|
|
||||||
|
class EmojiOnly(Dictionary):
|
||||||
|
'''Sentiment annotation with a basic lexicon of emojis'''
|
||||||
|
description = 'A plugin'
|
||||||
|
dictionaries = [basic.emojis]
|
||||||
|
|
||||||
|
test_cases = [{
|
||||||
|
'input': 'Hello :)',
|
||||||
|
'polarity': 'marl:Neutral'
|
||||||
|
}, {
|
||||||
|
'input': 'So sad :(',
|
||||||
|
'polarity': 'marl:Neutral'
|
||||||
|
}, {
|
||||||
|
'input': 'Yay! Emojis 😁',
|
||||||
|
'polarity': 'marl:Positive'
|
||||||
|
}, {
|
||||||
|
'input': 'But no emoticons 😢',
|
||||||
|
'polarity': 'marl:Negative'
|
||||||
|
}]
|
||||||
|
|
||||||
|
|
||||||
|
class EmoticonsOnly(Dictionary):
|
||||||
|
'''Sentiment annotation with a basic lexicon of emoticons'''
|
||||||
|
dictionaries = [basic.emoticons]
|
||||||
|
|
||||||
|
test_cases = [{
|
||||||
|
'input': 'Hello :)',
|
||||||
|
'polarity': 'marl:Positive'
|
||||||
|
}, {
|
||||||
|
'input': 'So sad :(',
|
||||||
|
'polarity': 'marl:Negative'
|
||||||
|
}, {
|
||||||
|
'input': 'Yay! Emojis 😁',
|
||||||
|
'polarity': 'marl:Neutral'
|
||||||
|
}, {
|
||||||
|
'input': 'But no emoticons 😢',
|
||||||
|
'polarity': 'marl:Neutral'
|
||||||
|
}]
|
||||||
|
|
||||||
|
|
||||||
|
class Salutes(Dictionary):
|
||||||
|
'''Sentiment annotation with a custom lexicon, for illustration purposes'''
|
||||||
|
dictionaries = [{
|
||||||
|
'marl:Positive': ['Hello', '!'],
|
||||||
|
'marl:Negative': ['sad', ]
|
||||||
|
}]
|
||||||
|
|
||||||
|
test_cases = [{
|
||||||
|
'input': 'Hello :)',
|
||||||
|
'polarity': 'marl:Positive'
|
||||||
|
}, {
|
||||||
|
'input': 'So sad :(',
|
||||||
|
'polarity': 'marl:Negative'
|
||||||
|
}, {
|
||||||
|
'input': 'Yay! Emojis 😁',
|
||||||
|
'polarity': 'marl:Positive'
|
||||||
|
}, {
|
||||||
|
'input': 'But no emoticons 😢',
|
||||||
|
'polarity': 'marl:Neutral'
|
||||||
|
}]
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
easy_test()
|
@ -0,0 +1,25 @@
|
|||||||
|
from senpy import AnalysisPlugin, easy
|
||||||
|
|
||||||
|
|
||||||
|
class Dummy(AnalysisPlugin):
|
||||||
|
'''This is a dummy self-contained plugin'''
|
||||||
|
author = '@balkian'
|
||||||
|
version = '0.1'
|
||||||
|
|
||||||
|
def analyse_entry(self, entry, params):
|
||||||
|
entry['nif:isString'] = entry['nif:isString'][::-1]
|
||||||
|
entry.reversed = entry.get('reversed', 0) + 1
|
||||||
|
yield entry
|
||||||
|
|
||||||
|
test_cases = [{
|
||||||
|
'entry': {
|
||||||
|
'nif:isString': 'Hello',
|
||||||
|
},
|
||||||
|
'expected': {
|
||||||
|
'nif:isString': 'olleH'
|
||||||
|
}
|
||||||
|
}]
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
easy()
|
@ -1,11 +0,0 @@
|
|||||||
from senpy.plugins import SentimentPlugin
|
|
||||||
|
|
||||||
|
|
||||||
class DummyPlugin(SentimentPlugin):
|
|
||||||
def analyse_entry(self, entry, params):
|
|
||||||
entry['nif:isString'] = entry['nif:isString'][::-1]
|
|
||||||
entry.reversed = entry.get('reversed', 0) + 1
|
|
||||||
yield entry
|
|
||||||
|
|
||||||
def test(self):
|
|
||||||
pass
|
|
@ -1,15 +0,0 @@
|
|||||||
{
|
|
||||||
"name": "Dummy",
|
|
||||||
"module": "dummy",
|
|
||||||
"description": "I am dummy",
|
|
||||||
"author": "@balkian",
|
|
||||||
"version": "0.1",
|
|
||||||
"extra_params": {
|
|
||||||
"example": {
|
|
||||||
"@id": "example_parameter",
|
|
||||||
"aliases": ["example", "ex"],
|
|
||||||
"required": false,
|
|
||||||
"default": 0
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
@ -1,27 +0,0 @@
|
|||||||
from senpy.plugins import SentimentPlugin
|
|
||||||
|
|
||||||
|
|
||||||
class DummyNoInfo(SentimentPlugin):
|
|
||||||
|
|
||||||
description = 'This is a dummy self-contained plugin'
|
|
||||||
author = '@balkian'
|
|
||||||
version = '0.1'
|
|
||||||
|
|
||||||
def analyse_entry(self, entry, params):
|
|
||||||
entry['nif:isString'] = entry['nif:isString'][::-1]
|
|
||||||
entry.reversed = entry.get('reversed', 0) + 1
|
|
||||||
yield entry
|
|
||||||
|
|
||||||
test_cases = [{
|
|
||||||
"entry": {
|
|
||||||
"nif:isString": "Hello world!"
|
|
||||||
},
|
|
||||||
"expected": [{
|
|
||||||
"nif:isString": "!dlrow olleH"
|
|
||||||
}]
|
|
||||||
}]
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
d = DummyNoInfo()
|
|
||||||
d.test()
|
|
@ -1,2 +0,0 @@
|
|||||||
name: DummyNoInfo
|
|
||||||
module: dummy_noinfo
|
|
@ -1,14 +0,0 @@
|
|||||||
{
|
|
||||||
"name": "DummyRequired",
|
|
||||||
"module": "dummy",
|
|
||||||
"description": "I am dummy",
|
|
||||||
"author": "@balkian",
|
|
||||||
"version": "0.1",
|
|
||||||
"extra_params": {
|
|
||||||
"example": {
|
|
||||||
"@id": "example_parameter",
|
|
||||||
"aliases": ["example", "ex"],
|
|
||||||
"required": true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
@ -0,0 +1,40 @@
|
|||||||
|
from senpy import AnalysisPlugin, easy
|
||||||
|
|
||||||
|
|
||||||
|
class DummyRequired(AnalysisPlugin):
|
||||||
|
'''This is a dummy self-contained plugin'''
|
||||||
|
author = '@balkian'
|
||||||
|
version = '0.1'
|
||||||
|
extra_params = {
|
||||||
|
'example': {
|
||||||
|
'description': 'An example parameter',
|
||||||
|
'required': True,
|
||||||
|
'options': ['a', 'b']
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
def analyse_entry(self, entry, params):
|
||||||
|
entry['nif:isString'] = entry['nif:isString'][::-1]
|
||||||
|
entry.reversed = entry.get('reversed', 0) + 1
|
||||||
|
yield entry
|
||||||
|
|
||||||
|
test_cases = [{
|
||||||
|
'entry': {
|
||||||
|
'nif:isString': 'Hello',
|
||||||
|
},
|
||||||
|
'expected': None
|
||||||
|
}, {
|
||||||
|
'entry': {
|
||||||
|
'nif:isString': 'Hello',
|
||||||
|
},
|
||||||
|
'params': {
|
||||||
|
'example': 'a'
|
||||||
|
},
|
||||||
|
'expected': {
|
||||||
|
'nif:isString': 'olleH'
|
||||||
|
}
|
||||||
|
}]
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
easy()
|
@ -0,0 +1,24 @@
|
|||||||
|
import noop
|
||||||
|
from senpy.plugins import SentimentPlugin
|
||||||
|
|
||||||
|
|
||||||
|
class NoOp(SentimentPlugin):
|
||||||
|
'''This plugin does nothing. Literally nothing.'''
|
||||||
|
|
||||||
|
version = 0
|
||||||
|
|
||||||
|
def analyse_entry(self, entry, *args, **kwargs):
|
||||||
|
yield entry
|
||||||
|
|
||||||
|
def test(self):
|
||||||
|
print(dir(noop))
|
||||||
|
super(NoOp, self).test()
|
||||||
|
|
||||||
|
test_cases = [{
|
||||||
|
'entry': {
|
||||||
|
'nif:isString': 'hello'
|
||||||
|
},
|
||||||
|
'expected': {
|
||||||
|
'nif:isString': 'hello'
|
||||||
|
}
|
||||||
|
}]
|
@ -0,0 +1,3 @@
|
|||||||
|
module: mynoop
|
||||||
|
requirements:
|
||||||
|
- noop
|
@ -1,5 +0,0 @@
|
|||||||
from senpy.plugins import SentimentPlugin
|
|
||||||
|
|
||||||
|
|
||||||
class NoOp(SentimentPlugin):
|
|
||||||
import noop
|
|
@ -0,0 +1,63 @@
|
|||||||
|
#!/usr/local/bin/python
|
||||||
|
# coding: utf-8
|
||||||
|
|
||||||
|
from senpy import easy_test, models, plugins
|
||||||
|
|
||||||
|
import basic
|
||||||
|
|
||||||
|
|
||||||
|
class ParameterizedDictionary(plugins.SentimentPlugin):
|
||||||
|
|
||||||
|
description = 'This is a basic self-contained plugin'
|
||||||
|
author = '@balkian'
|
||||||
|
version = '0.2'
|
||||||
|
|
||||||
|
extra_params = {
|
||||||
|
'positive-words': {
|
||||||
|
'description': 'Comma-separated list of words that are considered positive',
|
||||||
|
'aliases': ['positive'],
|
||||||
|
'required': True
|
||||||
|
},
|
||||||
|
'negative-words': {
|
||||||
|
'description': 'Comma-separated list of words that are considered negative',
|
||||||
|
'aliases': ['negative'],
|
||||||
|
'required': False
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
def analyse_entry(self, entry, params):
|
||||||
|
positive_words = params['positive-words'].split(',')
|
||||||
|
negative_words = params['negative-words'].split(',')
|
||||||
|
dictionary = {
|
||||||
|
'marl:Positive': positive_words,
|
||||||
|
'marl:Negative': negative_words,
|
||||||
|
}
|
||||||
|
polarity = basic.get_polarity(entry.text, [dictionary])
|
||||||
|
|
||||||
|
s = models.Sentiment(marl__hasPolarity=polarity)
|
||||||
|
s.prov(self)
|
||||||
|
entry.sentiments.append(s)
|
||||||
|
yield entry
|
||||||
|
|
||||||
|
test_cases = [
|
||||||
|
{
|
||||||
|
'input': 'Hello :)',
|
||||||
|
'polarity': 'marl:Positive',
|
||||||
|
'parameters': {
|
||||||
|
'positive': "Hello,:)",
|
||||||
|
'negative': "sad,:()"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
'input': 'Hello :)',
|
||||||
|
'polarity': 'marl:Negative',
|
||||||
|
'parameters': {
|
||||||
|
'positive': "",
|
||||||
|
'negative': "Hello"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
easy_test()
|
@ -0,0 +1,27 @@
|
|||||||
|
from senpy.plugins import AnalysisPlugin
|
||||||
|
from time import sleep
|
||||||
|
|
||||||
|
|
||||||
|
class Sleep(AnalysisPlugin):
|
||||||
|
'''Dummy plugin to test async'''
|
||||||
|
author = "@balkian"
|
||||||
|
version = "0.2"
|
||||||
|
timeout = 0.05
|
||||||
|
extra_params = {
|
||||||
|
"timeout": {
|
||||||
|
"@id": "timeout_sleep",
|
||||||
|
"aliases": ["timeout", "to"],
|
||||||
|
"required": False,
|
||||||
|
"default": 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
def activate(self, *args, **kwargs):
|
||||||
|
sleep(self.timeout)
|
||||||
|
|
||||||
|
def analyse_entry(self, entry, params):
|
||||||
|
sleep(float(params.get("timeout", self.timeout)))
|
||||||
|
yield entry
|
||||||
|
|
||||||
|
def test(self):
|
||||||
|
pass
|
@ -1,14 +0,0 @@
|
|||||||
from senpy.plugins import AnalysisPlugin
|
|
||||||
from time import sleep
|
|
||||||
|
|
||||||
|
|
||||||
class SleepPlugin(AnalysisPlugin):
|
|
||||||
def activate(self, *args, **kwargs):
|
|
||||||
sleep(self.timeout)
|
|
||||||
|
|
||||||
def analyse_entry(self, entry, params):
|
|
||||||
sleep(float(params.get("timeout", self.timeout)))
|
|
||||||
yield entry
|
|
||||||
|
|
||||||
def test(self):
|
|
||||||
pass
|
|
@ -1,16 +0,0 @@
|
|||||||
{
|
|
||||||
"name": "Sleep",
|
|
||||||
"module": "sleep",
|
|
||||||
"description": "I am dummy",
|
|
||||||
"author": "@balkian",
|
|
||||||
"version": "0.1",
|
|
||||||
"timeout": 0.05,
|
|
||||||
"extra_params": {
|
|
||||||
"timeout": {
|
|
||||||
"@id": "timeout_sleep",
|
|
||||||
"aliases": ["timeout", "to"],
|
|
||||||
"required": false,
|
|
||||||
"default": 0
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
@ -0,0 +1,124 @@
|
|||||||
|
'''
|
||||||
|
Meta-programming for the models.
|
||||||
|
'''
|
||||||
|
import os
|
||||||
|
import json
|
||||||
|
import jsonschema
|
||||||
|
import inspect
|
||||||
|
import copy
|
||||||
|
|
||||||
|
from abc import ABCMeta
|
||||||
|
|
||||||
|
|
||||||
|
class BaseMeta(ABCMeta):
|
||||||
|
'''
|
||||||
|
Metaclass for models. It extracts the default values for the fields in
|
||||||
|
the model.
|
||||||
|
|
||||||
|
For instance, instances of the following class wouldn't need to mark
|
||||||
|
their version or description on initialization:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
class MyPlugin(Plugin):
|
||||||
|
version=0.3
|
||||||
|
description='A dull plugin'
|
||||||
|
|
||||||
|
|
||||||
|
Note that these operations could be included in the __init__ of the
|
||||||
|
class, but it would be very inefficient.
|
||||||
|
'''
|
||||||
|
_subtypes = {}
|
||||||
|
|
||||||
|
def __new__(mcs, name, bases, attrs, **kwargs):
|
||||||
|
defaults = {}
|
||||||
|
register_afterwards = False
|
||||||
|
|
||||||
|
attrs = mcs.expand_with_schema(name, attrs)
|
||||||
|
if 'schema' in attrs:
|
||||||
|
register_afterwards = True
|
||||||
|
defaults = mcs.get_defaults(attrs['schema'])
|
||||||
|
for b in bases:
|
||||||
|
if hasattr(b, '_defaults'):
|
||||||
|
defaults.update(b._defaults)
|
||||||
|
info, attrs = mcs.split_attrs(attrs)
|
||||||
|
defaults.update(info)
|
||||||
|
attrs['_defaults'] = defaults
|
||||||
|
|
||||||
|
cls = super(BaseMeta, mcs).__new__(mcs, name, tuple(bases), attrs)
|
||||||
|
|
||||||
|
if register_afterwards:
|
||||||
|
mcs.register(cls, cls._defaults['@type'])
|
||||||
|
return cls
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def register(mcs, rsubclass, rtype=None):
|
||||||
|
mcs._subtypes[rtype or rsubclass.__name__] = rsubclass
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def expand_with_schema(name, attrs):
|
||||||
|
if 'schema' in attrs: # Schema specified by name
|
||||||
|
schema_file = '{}.json'.format(attrs['schema'])
|
||||||
|
elif 'schema_file' in attrs:
|
||||||
|
schema_file = attrs['schema_file']
|
||||||
|
del attrs['schema_file']
|
||||||
|
else:
|
||||||
|
return attrs
|
||||||
|
|
||||||
|
if '/' not in 'schema_file':
|
||||||
|
thisdir = os.path.dirname(os.path.realpath(__file__))
|
||||||
|
schema_file = os.path.join(thisdir,
|
||||||
|
'schemas',
|
||||||
|
schema_file)
|
||||||
|
|
||||||
|
schema_path = 'file://' + schema_file
|
||||||
|
|
||||||
|
with open(schema_file) as f:
|
||||||
|
schema = json.load(f)
|
||||||
|
|
||||||
|
resolver = jsonschema.RefResolver(schema_path, schema)
|
||||||
|
attrs['@type'] = "".join((name[0].lower(), name[1:]))
|
||||||
|
attrs['_schema_file'] = schema_file
|
||||||
|
attrs['schema'] = schema
|
||||||
|
attrs['_validator'] = jsonschema.Draft4Validator(schema, resolver=resolver)
|
||||||
|
return attrs
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def is_attr(k, v):
|
||||||
|
return (not(inspect.isroutine(v) or
|
||||||
|
inspect.ismethod(v) or
|
||||||
|
inspect.ismodule(v) or
|
||||||
|
isinstance(v, property)) and
|
||||||
|
k[0] != '_' and
|
||||||
|
k != 'schema' and
|
||||||
|
k != 'data')
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def split_attrs(attrs):
|
||||||
|
'''
|
||||||
|
Extract the attributes of the class.
|
||||||
|
|
||||||
|
This allows adding default values in the class definition.
|
||||||
|
e.g.:
|
||||||
|
'''
|
||||||
|
isattr = {}
|
||||||
|
notattr = {}
|
||||||
|
for key, value in attrs.items():
|
||||||
|
if BaseMeta.is_attr(key, value):
|
||||||
|
if key[0] != '_':
|
||||||
|
key = key.replace("__", ":", 1)
|
||||||
|
isattr[key] = copy.deepcopy(value)
|
||||||
|
else:
|
||||||
|
notattr[key] = value
|
||||||
|
return isattr, notattr
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def get_defaults(schema):
|
||||||
|
temp = {}
|
||||||
|
for obj in [
|
||||||
|
schema,
|
||||||
|
] + schema.get('allOf', []):
|
||||||
|
for k, v in obj.get('properties', {}).items():
|
||||||
|
if 'default' in v and k not in temp:
|
||||||
|
temp[k] = copy.deepcopy(v['default'])
|
||||||
|
return temp
|
@ -1,9 +0,0 @@
|
|||||||
---
|
|
||||||
name: emoRand
|
|
||||||
module: emoRand
|
|
||||||
description: A sample plugin that returns a random emotion annotation
|
|
||||||
author: "@balkian"
|
|
||||||
version: '0.1'
|
|
||||||
url: "https://github.com/gsi-upm/senpy-plugins-community"
|
|
||||||
requirements: {}
|
|
||||||
onyx:usesEmotionModel: "emoml:big6"
|
|
@ -0,0 +1,32 @@
|
|||||||
|
import random
|
||||||
|
|
||||||
|
from senpy.plugins import EmotionPlugin
|
||||||
|
from senpy.models import EmotionSet, Emotion, Entry
|
||||||
|
|
||||||
|
|
||||||
|
class EmoRand(EmotionPlugin):
|
||||||
|
'''A sample plugin that returns a random emotion annotation'''
|
||||||
|
author = '@balkian'
|
||||||
|
version = '0.1'
|
||||||
|
url = "https://github.com/gsi-upm/senpy-plugins-community"
|
||||||
|
onyx__usesEmotionModel = "emoml:big6"
|
||||||
|
|
||||||
|
def analyse_entry(self, entry, params):
|
||||||
|
category = "emoml:big6happiness"
|
||||||
|
number = max(-1, min(1, random.gauss(0, 0.5)))
|
||||||
|
if number > 0:
|
||||||
|
category = "emoml:big6anger"
|
||||||
|
emotionSet = EmotionSet()
|
||||||
|
emotion = Emotion({"onyx:hasEmotionCategory": category})
|
||||||
|
emotionSet.onyx__hasEmotion.append(emotion)
|
||||||
|
emotionSet.prov__wasGeneratedBy = self.id
|
||||||
|
entry.emotions.append(emotionSet)
|
||||||
|
yield entry
|
||||||
|
|
||||||
|
def test(self):
|
||||||
|
params = dict()
|
||||||
|
results = list()
|
||||||
|
for i in range(100):
|
||||||
|
res = next(self.analyse_entry(Entry(nif__isString="Hello"), params))
|
||||||
|
res.validate()
|
||||||
|
results.append(res.emotions[0]['onyx:hasEmotion'][0]['onyx:hasEmotionCategory'])
|
@ -1,10 +0,0 @@
|
|||||||
---
|
|
||||||
name: rand
|
|
||||||
module: rand
|
|
||||||
description: A sample plugin that returns a random sentiment annotation
|
|
||||||
author: "@balkian"
|
|
||||||
version: '0.1'
|
|
||||||
url: "https://github.com/gsi-upm/senpy-plugins-community"
|
|
||||||
requirements: {}
|
|
||||||
marl:maxPolarityValue: '1'
|
|
||||||
marl:minPolarityValue: "-1"
|
|
@ -1,33 +1,35 @@
|
|||||||
import random
|
import random
|
||||||
|
from senpy import SentimentPlugin, Sentiment, Entry
|
||||||
|
|
||||||
from senpy.plugins import SentimentPlugin
|
|
||||||
from senpy.models import Sentiment, Entry
|
|
||||||
|
|
||||||
|
class Rand(SentimentPlugin):
|
||||||
|
'''A sample plugin that returns a random sentiment annotation'''
|
||||||
|
author = "@balkian"
|
||||||
|
version = '0.1'
|
||||||
|
url = "https://github.com/gsi-upm/senpy-plugins-community"
|
||||||
|
marl__maxPolarityValue = '1'
|
||||||
|
marl__minPolarityValue = "-1"
|
||||||
|
|
||||||
class RandPlugin(SentimentPlugin):
|
|
||||||
def analyse_entry(self, entry, params):
|
def analyse_entry(self, entry, params):
|
||||||
lang = params.get("language", "auto")
|
|
||||||
|
|
||||||
polarity_value = max(-1, min(1, random.gauss(0.2, 0.2)))
|
polarity_value = max(-1, min(1, random.gauss(0.2, 0.2)))
|
||||||
polarity = "marl:Neutral"
|
polarity = "marl:Neutral"
|
||||||
if polarity_value > 0:
|
if polarity_value > 0:
|
||||||
polarity = "marl:Positive"
|
polarity = "marl:Positive"
|
||||||
elif polarity_value < 0:
|
elif polarity_value < 0:
|
||||||
polarity = "marl:Negative"
|
polarity = "marl:Negative"
|
||||||
sentiment = Sentiment({
|
sentiment = Sentiment(marl__hasPolarity=polarity,
|
||||||
"marl:hasPolarity": polarity,
|
marl__polarityValue=polarity_value)
|
||||||
"marl:polarityValue": polarity_value
|
sentiment.prov(self)
|
||||||
})
|
|
||||||
sentiment["prov:wasGeneratedBy"] = self.id
|
|
||||||
entry.sentiments.append(sentiment)
|
entry.sentiments.append(sentiment)
|
||||||
entry.language = lang
|
|
||||||
yield entry
|
yield entry
|
||||||
|
|
||||||
def test(self):
|
def test(self):
|
||||||
|
'''Run several random analyses.'''
|
||||||
params = dict()
|
params = dict()
|
||||||
results = list()
|
results = list()
|
||||||
for i in range(100):
|
for i in range(20):
|
||||||
res = next(self.analyse_entry(Entry(nif__isString="Hello"), params))
|
res = next(self.analyse_entry(Entry(nif__isString="Hello"),
|
||||||
|
params))
|
||||||
res.validate()
|
res.validate()
|
||||||
results.append(res.sentiments[0]['marl:hasPolarity'])
|
results.append(res.sentiments[0]['marl:hasPolarity'])
|
||||||
assert 'marl:Positive' in results
|
assert 'marl:Positive' in results
|
Loading…
Reference in New Issue