1
0
mirror of https://github.com/gsi-upm/senpy synced 2025-10-19 17:58:28 +00:00

Compare commits

...

90 Commits

Author SHA1 Message Date
militarpancho
83e2d415a1 Change name to split, according to issue #37 2017-06-13 19:44:40 +02:00
militarpancho
f8ca595bc9 Added chunker plugin to tokenize texts 2017-06-13 14:00:40 +02:00
J. Fernando Sánchez
312e7f7f12 Avoid python temporary files in pip tests 2017-06-12 21:50:51 +02:00
J. Fernando Sánchez
c555b9547e Non-interactive pip test 2017-06-12 21:27:02 +02:00
J. Fernando Sánchez
991ade8f4d Make sdist non-interactive non-tty 2017-06-12 21:20:07 +02:00
J. Fernando Sánchez
1104e816cb Push pip for tags without a preceding v 2017-06-12 21:06:34 +02:00
J. Fernando Sánchez
c19d03b41d Added SSH access to github fetch 2017-06-12 20:47:46 +02:00
J. Fernando Sánchez
42c9068991 Add pull policy to k8s deployment
* Add git fetch to (try to) fix github push from gitlab
2017-06-12 20:43:39 +02:00
J. Fernando Sánchez
96843827bd Removed __main__ from test coverage reports 2017-06-12 20:29:29 +02:00
J. Fernando Sánchez
d76e4618fe Removed python 3.4 from travis versions 2017-06-12 20:18:56 +02:00
J. Fernando Sánchez
c9bc485535 Merge branch '36-estimate-vad' 2017-06-12 20:10:21 +02:00
J. Fernando Sánchez
6d7575bbcd Merge branch '35-timeout-and-blocking-requests' 2017-06-12 19:57:28 +02:00
J. Fernando Sánchez
852bcc72ba Better centroid conversion
Also added **simple** tests for backward and forward conversion.
In future versions we should add thorough tests.

Should close gsi-upm/senpy#31
2017-06-12 19:52:00 +02:00
J. Fernando Sánchez
bf5ed1bd7d Merge remote-tracking branch 'drevicko/patch-6' 2017-06-12 18:14:15 +02:00
J. Fernando Sánchez
00da75153a Change conversion to Euclidean distance
* Added neutral point (if present)

Closes !gsi-upm/senpy#37 (Ian's)
2017-06-12 18:09:58 +02:00
J. Fernando Sánchez
fa082e11e7 Use flask's server by default
Using this server in production is discouraged, but to implement a
proper asynchronous server with tornado/gevent every blocking call would
have to be converted to a non-blocking call.

Failing to do so causes deadlocks like senpy/senpy#35

For now, it is easier to just use the default server.
2017-06-12 17:29:01 +02:00
J. Fernando Sánchez
6331d31b18 Merge branch '34-document-plugin-repo-creation' into 24-improve-docs
Closes #34
Closes #24
2017-06-12 12:53:24 +02:00
J. Fernando Sánchez
8ee324f566 Clearer docs 2017-06-12 09:31:42 +02:00
J. Fernando Sánchez
188c33332a Removed nbsphinx
It requires pandoc, which cannot be installed with pip.

We can either link to the nbfile or convert the file
manually/automatically:

```
nbconvert SenpyClientUse.ipynb --to rst
```
2017-06-12 09:31:42 +02:00
militarpancho
955e17eb2a Added travis, readthedocs and pypi badges 2017-06-12 09:31:42 +02:00
militarpancho
3e0f55dcff Improve docs. (Badges missing) 2017-06-12 09:31:38 +02:00
J. Fernando Sánchez
2ea01aef42 Fixed deployment IMAGENAME 2017-06-02 20:10:06 +02:00
J. Fernando Sánchez
147fd4a333 Fixed IMAGENAME 2017-06-02 20:02:27 +02:00
J. Fernando Sánchez
e31bca7016 Push to dockerhub instead of private registry 2017-06-02 19:42:22 +02:00
J. Fernando Sánchez
7956d54c35 K8s deployment with limits 2017-06-02 19:17:27 +02:00
militarpancho
5bab9a6a02 #34. Fixed some errors from plugins examples 2017-06-02 17:43:18 +02:00
militarpancho
69ac95bb08 Added example plugin in docs. #34 2017-06-02 17:39:27 +02:00
drevicko
6b843a4384 fixes typo in code 2017-05-29 12:15:35 +01:00
drevicko
65d6e47513 Implements Fernando's suggestion in #31
I've added a neutral point definition (in the converters senpy file) as used in pull request #29
2017-05-29 12:13:21 +01:00
drevicko
8d56a0b630 fixes #31
I've used euclidean metric instead of taxicab as I feel it makes more sense (taxicab has bizzare unintuitive effects for points far from the centroids).
2017-05-29 12:06:44 +01:00
drevicko
e7ac6e66b0 update _forward_conversion docstring + minor edits 2017-05-29 11:50:14 +01:00
J. Fernando Sánchez
0f8d1dff69 Fixed image repository 2017-05-19 19:59:45 +02:00
J. Fernando Sánchez
236183593c Hidden variables 2017-05-19 19:36:25 +02:00
J. Fernando Sánchez
7637498517 Added push to github 2017-05-19 19:33:00 +02:00
J. Fernando Sánchez
8c70433312 Added push to github 2017-05-19 18:54:57 +02:00
J. Fernando Sánchez
ce83fb3981 Added k8s deployment 2017-05-19 16:52:10 +02:00
J. Fernando Sánchez
28f29d159a Merge branch 'gh-34-broken-shelf' into 0.8.x 2017-05-17 17:39:14 +02:00
J. Fernando Sánchez
c803f60fd4 Merge branch 'drevicko/provide-analyse-traceback-in-log' into 0.8.x 2017-05-17 17:38:41 +02:00
J. Fernando Sánchez
12eae16e37 Merge branch 'drevicko-patch-5' into 0.8.x 2017-05-17 17:35:50 +02:00
J. Fernando Sánchez
f3372c27b6 Merge branch '32-update-module-dev-environment' into 0.8.x 2017-05-17 17:34:21 +02:00
Ian Wood
b6de72a143 chaged exception logging to 'exception()' in analysie() 2017-05-17 17:10:05 +02:00
J. Fernando Sánchez
0f89b92457 Fixed pickling error in py2.7 2017-05-17 16:51:01 +02:00
J. Fernando Sánchez
ea91e3e4a4 Add an option to force the load of shelf plugins
Closes gsi-upm/senpy#34
2017-05-17 16:30:01 +02:00
Ian Wood
f76b777b9f don't fail if shelf pickle file broken 2017-05-16 15:09:46 +01:00
drevicko
dcc965ea63 removed superfluous 'neutral' centroid
Neutral is included as an 'origin' field. This is partly because emoml has no vocab for "Neutral" in dimensional models.
2017-05-08 14:34:28 +01:00
drevicko
400f647b7b removed unneccessary defaultdict import 2017-05-08 14:32:53 +01:00
Ian Wood
ec1a2ff5f9 added 'origin' to VAD representation, incorporated into weighed sum for Cat->VAD conversion 2017-05-08 14:28:51 +01:00
J. Fernando Sánchez
e112dd55ce Pip install editable
Closes senpy/senpy#32
2017-05-05 17:27:12 +02:00
J. Fernando Sánchez
60ef304108 Analysis set as a python list
Closes senpy/senpy#31
2017-05-05 17:05:17 +02:00
Ian Wood
1a9dd07f7e Merge branch 'master' 0.8.7 into patch-6 2017-05-05 15:02:15 +01:00
Ian Wood
b80b0c7947 used more specific exception specifier (KeyError) 2017-04-11 11:25:50 +01:00
Ian Wood
1ca6ec52fd fixed weighted average, no explicit treatment of 'neutral' 2017-04-11 11:12:02 +01:00
J. Fernando Sánchez
7927cf1587 add option to show senpy version number
Merge branch 'patch-5' of https://github.com/drevicko/senpy into drevicko-patch-5
2017-04-10 21:03:17 +02:00
J. Fernando Sánchez
13cefbedfb Clean dev containers in makefile 2017-04-10 20:38:12 +02:00
J. Fernando Sánchez
4ba9535d56 Merge branches into 0.8.x
'25-validation-errors'
'27-add-method-to-get-list-of-plugins'
'28-fix-multiprocessing-issues'
2017-04-10 20:31:26 +02:00
J. Fernando Sánchez
e582ef07d4 Fix multiprocessing tests in python2.7
Closes #28 for python 2.

Apparently, process pools are not contexts in python 2.7.
On the other hand, in py2 you cannot pickle instance methods, so
you have to implement Pool tasks as independent functions.
2017-04-10 20:17:38 +02:00
J. Fernando Sánchez
ef40bdb545 Replace gevent with tornado
Closes #28

Added:

* Async test (still missing one that includes the IOLoop)
* Async plugin under tests. To manually try async functionalities:
```
senpy -f tests/
```
2017-04-10 18:16:45 +02:00
J. Fernando Sánchez
e0b4c76238 Add plugin method to client
Closes #28
2017-04-10 18:07:34 +02:00
J. Fernando Sánchez
14c86ec38c Set plugin list as a @set and fixed test case
It turns out setting "plugins" as a @list in the context causes the
"plugins" property to expand to its full name.
Removing the type causes a regression of #17, which I initially missed
because the test in #17 was wrong.

Closes #26
2017-04-10 17:24:39 +02:00
J. Fernando Sánchez
d3d05b3218 Fixed expansion of "plugins"
Closes #26

There was no need to add @list, and it was causing JSON-LD to expand the
URI of 'plugins'
2017-04-07 16:24:28 +02:00
J. Fernando Sánchez
eababcadb0 Analysis as strings or objects in results
Closes #25
2017-04-07 16:13:57 +02:00
drevicko
7efece0224 add option to show senpy version number 2017-04-07 10:04:51 +01:00
drevicko
53138e6942 Estimate VAD by weighted average
Does a weighted average of centroids.

If intensity sums to zero for a category, a 'neutral' category is used or 0 if it's not present. I'm not 100% sure this is the best approach, and the name of the "neutral" category perhaps should use some convention?

Note that if there are no categories present, then no VAD (or other dimensional) estimate is returned. It may be better to use the neutral centroid if it's present in this case also.
2017-04-04 15:37:07 +01:00
J. Fernando Sánchez
1302b0b93c Fixed pip tests (added version) 2017-04-04 11:42:18 +02:00
J. Fernando Sánchez
ad1092690b Merge branch '20-improve-docs' into 0.8.x 2017-04-04 11:26:33 +02:00
J. Fernando Sánchez
e35e810ede Rephrase info on demo plugins
Closes #20
2017-04-04 11:26:05 +02:00
militarpancho
d5ddcb8d3f Change repository url 2017-04-04 11:21:08 +02:00
militarpancho
54c0c9c437 demo doc changed 2017-04-04 11:14:51 +02:00
J. Fernando Sánchez
6e970d01f2 Merge branch '21-ascii-cant-encode' into 0.8.x 2017-04-04 11:12:39 +02:00
J. Fernando Sánchez
1d0a54ecd2 Merge branch '22-pip-screws-with-logging-config' into 0.8.x 2017-04-04 11:12:23 +02:00
J. Fernando Sánchez
800d4a9c2c Fixed typos in Ian's patch 2017-04-04 11:11:51 +02:00
drevicko
035ef98b7e removed broken "/api" link
In index.html, there is a suggestion to try out the api with a link to "/api". Clicking that link results in a json error report - not ideal. 
Instead, I added text suggesting that a use can find example api url's after clickgin "Analyse!".
2017-04-04 11:07:32 +02:00
J. Fernando Sánchez
d7e115d7c2 Encode HEADERS
Closes #21
2017-04-03 19:23:18 +02:00
J. Fernando Sánchez
548cb4c9ba Doc changes
* Alabaster theme
* Restructured
* Simplified introduction
* Reference to entries/models
* Fixed examples
2017-04-03 18:20:09 +02:00
J. Fernando Sánchez
7e5b55ff9c Run pip with Popen
Closes #22
2017-03-30 17:38:17 +02:00
militarpancho
8b2c3e8d40 Update readthedocs. Mainly Api and What is senpy section 2017-03-28 12:34:39 +02:00
J. Fernando Sánchez
0c8f98d466 Pre-0.8.6
* Improved debugging (back to using Flask's built-in mechanisms)
* Recursive model loading from json
* Added DEVPORT to Makefile
* Accept json-ld input. Closes #16
* Improved Exception handling in client
* Modified default plugin selection to only include analysis plugins
* More tests
2017-03-14 19:59:06 +01:00
J. Fernando Sánchez
cc298742ec Merge branch '17-...' into 0.8.x 2017-03-14 13:20:20 +01:00
J. Fernando Sánchez
250052fb99 Options as a set in the JSON-LD context
Closes #18
2017-03-14 13:17:47 +01:00
J. Fernando Sánchez
603e086606 Fix list of plugins
Closes #17
2017-03-14 13:05:52 +01:00
J. Fernando Sánchez
a8614bab0c Accept plugin pipelines
Closes #15
2017-03-13 21:08:21 +01:00
J. Fernando Sánchez
70ca74b03c Added instructions for developers 2017-03-09 00:04:02 +01:00
J. Fernando Sánchez
c9e6d78183 Fixed alises, added PAD and FSRE
Closes #13
2017-03-08 23:23:40 +01:00
J. Fernando Sánchez
1a582c0843 Filter conversion plugins
Closes #12

* Shows only analysis plugins by default on /api/plugins
* Adds a plugin_type parameter to get other types of plugins
* default_plugin chosen from analysis plugins
2017-03-08 22:54:57 +01:00
drevicko
0394bcd69c add make version to readme for pip install
pip install needs the VERSION file - `make version` will create that file

I also added the -U flag to pip install to force install (this is important if the user is playing with the code or trying out different older versions, as pip will not install if it thinks the git repo represents a version already installed or older than the one installed)
2017-03-02 11:08:02 +00:00
J. Fernando Sánchez
cbeb3adbdb Added fallback version '0.0'
Installing depends on the VERSION file, so it raies an error if it is
installed in some other way.

ReadTheDocs installs the package so it can generate code docs.
This commit adds a default version 0.0
2017-03-01 18:53:54 +01:00
J. Fernando Sánchez
efb305173e Removed future from __init__
Since __init__ is imported by setup.py, future may not be installed yet.

Other options would be:

* Read VERSION -> and that code has to be duplicated in setup.py and
  senpy (to avoid the import, once again)
* Eval version.py
* Do without versioning :)
2017-03-01 18:28:20 +01:00
J. Fernando Sánchez
2288b04c92 Remove iteritems for py2/3 compatibility 2017-03-01 18:14:44 +01:00
J. Fernando Sánchez
7899cb4d33 Fixed docker upload
Doing docker push without a tag makes the client upload **ALL** the
images it has for that repo.
2017-03-01 17:59:35 +01:00
J. Fernando Sánchez
62ddca79ac Fixed conversion docs 2017-03-01 17:56:17 +01:00
64 changed files with 2096 additions and 681 deletions

View File

@@ -1,18 +1,19 @@
image: gsiupm/dockermake:latest # Uncomment if you want to use docker-in-docker
# image: gsiupm/dockermake:latest
# services:
# - docker:dind
# When using dind, it's wise to use the overlayfs driver for # When using dind, it's wise to use the overlayfs driver for
# improved performance. # improved performance.
variables:
DOCKER_DRIVER: overlay
DOCKERFILE: Dockerfile
IMAGENAME: $CI_REGISTRY_IMAGE
stages: stages:
- test - test
- push - push
- deploy
- clean - clean
before_script:
- docker login -u $HUB_USER -p $HUB_PASSWORD
.test: &test_definition .test: &test_definition
stage: test stage: test
script: script:
@@ -28,11 +29,8 @@ test-2.7:
variables: variables:
PYTHON_VERSION: "2.7" PYTHON_VERSION: "2.7"
.image: &image_definition .image: &image_definition
stage: push stage: push
before_script:
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
script: script:
- make -e push-$PYTHON_VERSION - make -e push-$PYTHON_VERSION
only: only:
@@ -57,9 +55,52 @@ push-latest:
- master - master
- triggers - triggers
push-github:
stage: deploy
script:
- make -e push-github
only:
- master
- triggers
deploy_pypi:
stage: deploy
script: # Configure the PyPI credentials, then push the package, and cleanup the creds.
- echo "[server-login]" >> ~/.pypirc
- echo "username=" ${PYPI_USER} >> ~/.pypirc
- echo "password=" ${PYPI_PASSWORD} >> ~/.pypirc
- make pip_upload
- echo "" > ~/.pypirc && rm ~/.pypirc # If the above fails, this won't run.
only:
- /^v?\d+\.\d+\.\d+([abc]\d*)?$/ # PEP-440 compliant version (tags)
except:
- branches
deploy:
stage: deploy
environment: test
script:
- make -e deploy
only:
- master
push-github:
stage: deploy
script:
- make -e push-github
only:
- master
- triggers
clean : clean :
stage: clean stage: clean
script: script:
- make -e clean - make -e clean
only: when: manual
- master
cleanup_py:
stage: clean
when: always # this is important; run even if preceding stages failed.
script:
- rm -vf ~/.pypirc # we don't want to leave these around, but GitLab may clean up anyway.
- docker logout

View File

@@ -7,7 +7,6 @@ language: python
env: env:
- PYV=2.7 - PYV=2.7
- PYV=3.4
- PYV=3.5 - PYV=3.5
# run nosetests - Tests # run nosetests - Tests
script: make test-$PYV script: make test-$PYV

View File

@@ -17,6 +17,6 @@ WORKDIR /usr/src/app
COPY test-requirements.txt requirements.txt /usr/src/app/ COPY test-requirements.txt requirements.txt /usr/src/app/
RUN pip install --use-wheel -r test-requirements.txt -r requirements.txt RUN pip install --use-wheel -r test-requirements.txt -r requirements.txt
COPY . /usr/src/app/ COPY . /usr/src/app/
RUN pip install --no-deps --no-index . RUN pip install --no-index --no-deps --editable .
ENTRYPOINT ["python", "-m", "senpy", "-f", "/senpy-plugins/", "--host", "0.0.0.0"] ENTRYPOINT ["python", "-m", "senpy", "-f", "/senpy-plugins/", "--host", "0.0.0.0"]

View File

@@ -1,12 +1,30 @@
NAME=senpy
VERSION=$(shell git describe --tags --dirty 2>/dev/null)
GITHUB_REPO=git@github.com:gsi-upm/senpy.git
IMAGENAME=gsiupm/senpy
IMAGEWTAG=$(IMAGENAME):$(VERSION)
PYVERSIONS=3.5 2.7 PYVERSIONS=3.5 2.7
PYMAIN=$(firstword $(PYVERSIONS)) PYMAIN=$(firstword $(PYVERSIONS))
NAME=senpy
REPO=gsiupm DEVPORT=5000
VERSION=$(shell git describe --tags --dirty 2>/dev/null)
TARNAME=$(NAME)-$(VERSION).tar.gz TARNAME=$(NAME)-$(VERSION).tar.gz
IMAGENAME=$(REPO)/$(NAME)
IMAGEWTAG=$(IMAGENAME):$(VERSION)
action="test-${PYMAIN}" action="test-${PYMAIN}"
GITHUB_REPO=git@github.com:gsi-upm/senpy.git
KUBE_CA_PEM_FILE=""
KUBE_URL=""
KUBE_TOKEN=""
KUBE_NAMESPACE=$(NAME)
KUBECTL=docker run --rm -v $(KUBE_CA_PEM_FILE):/tmp/ca.pem -v $$PWD:/tmp/cwd/ -i lachlanevenson/k8s-kubectl --server="$(KUBE_URL)" --token="$(KUBE_TOKEN)" --certificate-authority="/tmp/ca.pem" -n $(KUBE_NAMESPACE)
CI_REGISTRY=docker.io
CI_REGISTRY_USER=gitlab
CI_BUILD_TOKEN=""
CI_COMMIT_REF_NAME=master
all: build run all: build run
@@ -17,7 +35,7 @@ version: .FORCE
@echo $(VERSION) @echo $(VERSION)
yapf: yapf:
yapf -i -r senpy yapf -i -r $(NAME)
yapf -i -r tests yapf -i -r tests
init: init:
@@ -36,14 +54,14 @@ quick_build: $(addprefix build-, $(PYMAIN))
build: $(addprefix build-, $(PYVERSIONS)) build: $(addprefix build-, $(PYVERSIONS))
build-%: version Dockerfile-% build-%: version Dockerfile-%
docker build -t '$(IMAGEWTAG)-python$*' -f Dockerfile-$* .; docker build -t '$(IMAGEWTAG)-python$*' --cache-from $(IMAGENAME):python$* -f Dockerfile-$* .;
quick_test: $(addprefix test-,$(PYMAIN)) quick_test: $(addprefix test-,$(PYMAIN))
dev-%: dev-%:
@docker start $(NAME)-dev$* || (\ @docker start $(NAME)-dev$* || (\
$(MAKE) build-$*; \ $(MAKE) build-$*; \
docker run -d -w /usr/src/app/ -v $$PWD:/usr/src/app --entrypoint=/bin/bash -ti --name $(NAME)-dev$* '$(IMAGEWTAG)-python$*'; \ docker run -d -w /usr/src/app/ -p $(DEVPORT):5000 -v $$PWD:/usr/src/app --entrypoint=/bin/bash -ti --name $(NAME)-dev$* '$(IMAGEWTAG)-python$*'; \
)\ )\
docker exec -ti $(NAME)-dev$* bash docker exec -ti $(NAME)-dev$* bash
@@ -52,26 +70,29 @@ dev: dev-$(PYMAIN)
test-all: $(addprefix test-,$(PYVERSIONS)) test-all: $(addprefix test-,$(PYVERSIONS))
test-%: build-% test-%:
docker run --rm --entrypoint /usr/local/bin/python -w /usr/src/app $(IMAGEWTAG)-python$* setup.py test docker run --rm --entrypoint /usr/local/bin/python -w /usr/src/app $(IMAGENAME):python$* setup.py test
test: test-$(PYMAIN) test: test-$(PYMAIN)
dist/$(TARNAME): dist/$(TARNAME): version
docker run --rm -ti -v $$PWD:/usr/src/app/ -w /usr/src/app/ python:$(PYMAIN) python setup.py sdist; python setup.py sdist;
sdist: dist/$(TARNAME) sdist: dist/$(TARNAME)
pip_test-%: sdist pip_test-%: sdist
docker run --rm -v $$PWD/dist:/dist/ -ti python:$* pip install /dist/$(TARNAME); docker run --rm -v $$PWD/dist:/dist/ python:$* pip install /dist/$(TARNAME);
pip_test: $(addprefix pip_test-,$(PYVERSIONS)) pip_test: $(addprefix pip_test-,$(PYVERSIONS))
clean: pip_upload: pip_test
@docker ps -a | awk '/$(REPO)\/$(NAME)/{ split($$2, vers, "-"); if(vers[0] != "${VERSION}"){ print $$1;}}' | xargs docker rm -v 2>/dev/null|| true python setup.py sdist upload ;
@docker images | awk '/$(REPO)\/$(NAME)/{ split($$2, vers, "-"); if(vers[0] != "${VERSION}"){ print $$1":"$$2;}}' | xargs docker rmi 2>/dev/null|| true
@docker rmi $(NAME)-dev 2>/dev/null || true
clean:
@docker ps -a | grep $(IMAGENAME) | awk '{ split($$2, vers, "-"); if(vers[0] != "${VERSION}"){ print $$1;}}' | xargs docker rm -v 2>/dev/null|| true
@docker images | grep $(IMAGENAME) | awk '{ split($$2, vers, "-"); if(vers[0] != "${VERSION}"){ print $$1":"$$2;}}' | xargs docker rmi 2>/dev/null|| true
@docker stop $(addprefix $(NAME)-dev,$(PYVERSIONS)) 2>/dev/null || true
@docker rm $(addprefix $(NAME)-dev,$(PYVERSIONS)) 2>/dev/null || true
git_commit: git_commit:
git commit -a git commit -a
@@ -82,26 +103,45 @@ git_tag:
git_push: git_push:
git push --tags origin master git push --tags origin master
pip_upload:
python setup.py sdist upload ;
pip_test: $(addprefix pip_test-,$(PYVERSIONS))
run-%: build-% run-%: build-%
docker run --rm -p 5000:5000 -ti '$(IMAGEWTAG)-python$(PYMAIN)' --default-plugins docker run --rm -p $(DEVPORT):5000 -ti '$(IMAGEWTAG)-python$(PYMAIN)' --default-plugins
run: run-$(PYMAIN) run: run-$(PYMAIN)
push-latest: build-$(PYMAIN) push-latest: $(addprefix push-latest-,$(PYVERSIONS))
docker tag '$(IMAGEWTAG)-python$(PYMAIN)' '$(IMAGEWTAG)' docker tag '$(IMAGEWTAG)-python$(PYMAIN)' '$(IMAGEWTAG)'
docker tag '$(IMAGEWTAG)-python$(PYMAIN)' '$(IMAGENAME)' docker tag '$(IMAGEWTAG)-python$(PYMAIN)' '$(IMAGENAME)'
docker push '$(IMAGENAME)' docker push '$(IMAGENAME):latest'
docker push '$(IMAGEWTAG)' docker push '$(IMAGEWTAG)'
push-latest-%: build-%
docker tag $(IMAGENAME):$(VERSION)-python$* $(IMAGENAME):python$*
docker push $(IMAGENAME):$(VERSION)-python$*
docker push $(IMAGENAME):python$*
push-%: build-% push-%: build-%
docker push $(IMAGENAME):$(VERSION)-python$* docker push $(IMAGENAME):$(VERSION)-python$*
push: $(addprefix push-,$(PYVERSIONS))
docker tag '$(IMAGEWTAG)-python$(PYMAIN)' '$(IMAGEWTAG)'
docker push $(IMAGENAME):$(VERSION)
push-github:
$(eval KEY_FILE := $(shell mktemp))
@echo "$$GITHUB_DEPLOY_KEY" > $(KEY_FILE)
@git remote rm github-deploy || true
git remote add github-deploy $(GITHUB_REPO)
@GIT_SSH_COMMAND="ssh -i $(KEY_FILE)" git fetch github-deploy $(CI_COMMIT_REF_NAME) || true
@GIT_SSH_COMMAND="ssh -i $(KEY_FILE)" git push github-deploy $(CI_COMMIT_REF_NAME)
rm $(KEY_FILE)
ci: ci:
gitlab-runner exec docker --docker-volumes /var/run/docker.sock:/var/run/docker.sock --env CI_PROJECT_NAME=$(NAME) ${action} gitlab-runner exec docker --docker-volumes /var/run/docker.sock:/var/run/docker.sock --env CI_PROJECT_NAME=$(NAME) ${action}
.PHONY: test test-% test-all build-% build test pip_test run yapf push-main push-% dev ci version .FORCE deploy:
@$(KUBECTL) delete secret $(CI_REGISTRY) || true
@$(KUBECTL) create secret docker-registry $(CI_REGISTRY) --docker-server=$(CI_REGISTRY) --docker-username=$(CI_REGISTRY_USER) --docker-email=$(CI_REGISTRY_USER) --docker-password=$(CI_BUILD_TOKEN)
@$(KUBECTL) apply -f /tmp/cwd/k8s/
.PHONY: test test-% test-all build-% build test pip_test run yapf push-main push-% dev ci version .FORCE deploy

View File

@@ -23,7 +23,7 @@ Through PIP
.. code:: bash .. code:: bash
pip install --user senpy pip install -U --user senpy
Alternatively, you can use the development version: Alternatively, you can use the development version:
@@ -42,6 +42,53 @@ Build the image or use the pre-built one: ``docker run -ti -p 5000:5000 gsiupm/s
To add custom plugins, add a volume and tell senpy where to find the plugins: ``docker run -ti -p 5000:5000 -v <PATH OF PLUGINS>:/plugins gsiupm/senpy --default-plugins -f /plugins`` To add custom plugins, add a volume and tell senpy where to find the plugins: ``docker run -ti -p 5000:5000 -v <PATH OF PLUGINS>:/plugins gsiupm/senpy --default-plugins -f /plugins``
Developing
----------
Developing/debugging
********************
This command will run the senpy container using the latest image available, mounting your current folder so you get your latest code:
.. code:: bash
# Python 3.5
make dev
# Python 2.7
make dev-2.7
Building a docker image
***********************
.. code:: bash
# Python 3.5
make build-3.5
# Python 2.7
make build-2.7
Testing
*******
.. code:: bash
make test
Running
*******
This command will run the senpy server listening on localhost:5000
.. code:: bash
# Python 3.5
make run-3.5
# Python 2.7
make run-2.7
Usage Usage
----- -----
@@ -49,12 +96,14 @@ However, the easiest and recommended way is to just use the command-line tool to
.. code:: bash .. code:: bash
senpy senpy
or, alternatively: or, alternatively:
.. code:: bash .. code:: bash
python -m senpy python -m senpy

317
docs/SenpyClientUse.ipynb Normal file
View File

@@ -0,0 +1,317 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"ExecuteTime": {
"end_time": "2017-04-10T17:05:31.465571Z",
"start_time": "2017-04-10T19:05:31.458282+02:00"
},
"deletable": true,
"editable": true
},
"source": [
"# Client"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true,
"deletable": true,
"editable": true
},
"source": [
"The built-in senpy client allows you to query any Senpy endpoint. We will illustrate how to use it with the public demo endpoint, and then show you how to spin up your own endpoint using docker."
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"Demo Endpoint\n",
"-------------"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"To start using senpy, simply create a new Client and point it to your endpoint. In this case, the latest version of Senpy at GSI."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"ExecuteTime": {
"end_time": "2017-04-10T17:29:12.827640Z",
"start_time": "2017-04-10T19:29:12.818617+02:00"
},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"from senpy.client import Client\n",
"\n",
"c = Client('http://latest.senpy.cluster.gsi.dit.upm.es/api')\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"Now, let's use that client analyse some queries:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"ExecuteTime": {
"end_time": "2017-04-10T17:29:14.011657Z",
"start_time": "2017-04-10T19:29:13.701808+02:00"
},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"r = c.analyse('I like sugar!!', algorithm='sentiment140')\n",
"r"
]
},
{
"cell_type": "markdown",
"metadata": {
"ExecuteTime": {
"end_time": "2017-04-10T17:08:19.616754Z",
"start_time": "2017-04-10T19:08:19.610767+02:00"
},
"deletable": true,
"editable": true
},
"source": [
"As you can see, that gave us the full JSON result. A more concise way to print it would be:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"ExecuteTime": {
"end_time": "2017-04-10T17:29:14.854213Z",
"start_time": "2017-04-10T19:29:14.842068+02:00"
},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"for entry in r.entries:\n",
" print('{} -> {}'.format(entry['text'], entry['sentiments'][0]['marl:hasPolarity']))"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"We can also obtain a list of available plugins with the client:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"ExecuteTime": {
"end_time": "2017-04-10T17:29:16.245198Z",
"start_time": "2017-04-10T19:29:16.056545+02:00"
},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"c.plugins()"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"Or, more concisely:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"ExecuteTime": {
"end_time": "2017-04-10T17:29:17.663275Z",
"start_time": "2017-04-10T19:29:17.484623+02:00"
},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"c.plugins().keys()"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"Local Endpoint\n",
"--------------"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"To run your own instance of senpy, just create a docker container with the latest Senpy image. Using `--default-plugins` you will get some extra plugins to start playing with the API."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"ExecuteTime": {
"end_time": "2017-04-10T17:29:20.637539Z",
"start_time": "2017-04-10T19:29:19.938322+02:00"
},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"!docker run -ti --name 'SenpyEndpoint' -d -p 6000:5000 gsiupm/senpy:0.8.6 --host 0.0.0.0 --default-plugins"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"To use this endpoint:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"ExecuteTime": {
"end_time": "2017-04-10T17:29:21.263976Z",
"start_time": "2017-04-10T19:29:21.260595+02:00"
},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"c_local = Client('http://127.0.0.1:6000/api')"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"That's all! After you are done with your analysis, stop the docker container:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"ExecuteTime": {
"end_time": "2017-04-10T17:29:33.226686Z",
"start_time": "2017-04-10T19:29:22.392121+02:00"
},
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"!docker stop SenpyEndpoint\n",
"!docker rm SenpyEndpoint"
]
}
],
"metadata": {
"anaconda-cloud": {},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.0"
},
"toc": {
"colors": {
"hover_highlight": "#DAA520",
"running_highlight": "#FF0000",
"selected_highlight": "#FFD700"
},
"moveMenuLeft": true,
"nav_menu": {
"height": "68px",
"width": "252px"
},
"navigate_menu": true,
"number_sections": true,
"sideBar": true,
"threshold": 4,
"toc_cell": false,
"toc_section_display": "block",
"toc_window_display": false
}
},
"nbformat": 4,
"nbformat_minor": 1
}

BIN
docs/_static/header.png vendored Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 208 KiB

11
docs/about.rst Normal file
View File

@@ -0,0 +1,11 @@
About
--------
If you use Senpy in your research, please cite `Senpy: A Pragmatic Linked Sentiment Analysis Framework <http://gsi.dit.upm.es/index.php/es/investigacion/publicaciones?view=publication&task=show&id=417>`__ (`BibTex <http://gsi.dit.upm.es/index.php/es/investigacion/publicaciones?controller=publications&task=export&format=bibtex&id=417>`__):
.. code-block:: text
Sánchez-Rada, J. F., Iglesias, C. A., Corcuera, I., & Araque, Ó. (2016, October).
Senpy: A Pragmatic Linked Sentiment Analysis Framework.
In Data Science and Advanced Analytics (DSAA),
2016 IEEE International Conference on (pp. 735-742). IEEE.

View File

@@ -1,5 +1,5 @@
NIF API NIF API
======= -------
.. http:get:: /api .. http:get:: /api
Basic endpoint for sentiment/emotion analysis. Basic endpoint for sentiment/emotion analysis.
@@ -22,38 +22,32 @@ NIF API
Content-Type: text/javascript Content-Type: text/javascript
{ {
"@context": [ "@context":"http://127.0.0.1/api/contexts/Results.jsonld",
"http://127.0.0.1/static/context.jsonld", "@id":"_:Results_11241245.22",
], "@type":"results"
"analysis": [ "analysis": [
{ "plugins/sentiment-140_0.1"
"@id": "SentimentAnalysisExample", ],
"@type": "marl:SentimentAnalysis", "entries": [
"dc:language": "en", {
"marl:maxPolarityValue": 10.0, "@id": "_:Entry_11241245.22"
"marl:minPolarityValue": 0.0 "@type":"entry",
} "emotions": [],
], "entities": [],
"domain": "wndomains:electronics", "sentiments": [
"entries": [ {
{ "@id": "Sentiment0",
"opinions": [ "@type": "sentiment",
{ "marl:hasPolarity": "marl:Negative",
"prov:generatedBy": "SentimentAnalysisExample", "marl:polarityValue": 0,
"marl:polarityValue": 7.8, "prefix": ""
"marl:hasPolarity": "marl:Positive", }
"marl:describesObject": "http://www.gsi.dit.upm.es", ],
} "suggestions": [],
], "text": "This text makes me sad.\nwhilst this text makes me happy and surprised at the same time.\nI cannot believe it!",
"nif:isString": "I love GSI", "topics": []
"strings": [ }
{ ]
"nif:anchorOf": "GSI",
"nif:taIdentRef": "http://www.gsi.dit.upm.es"
}
]
}
]
} }
:query i input: No default. Depends on informat and intype :query i input: No default. Depends on informat and intype
@@ -92,58 +86,59 @@ NIF API
.. sourcecode:: http .. sourcecode:: http
{ {
"@context": { "@id": "plugins/sentiment-140_0.1",
... "@type": "sentimentPlugin",
}, "author": "@balkian",
"@type": "plugins", "description": "Sentiment classifier using rule-based classification for English and Spanish. This plugin uses sentiment140 data to perform classification. For more information: http://help.sentiment140.com/for-students/",
"plugins": [ "extra_params": {
{ "language": {
"name": "sentiment140", "@id": "lang_sentiment140",
"is_activated": true, "aliases": [
"version": "0.1", "language",
"extra_params": { "l"
"@id": "extra_params_sentiment140_0.1", ],
"language": { "options": [
"required": false, "es",
"@id": "lang_sentiment140", "en",
"options": [ "auto"
"es", ],
"en", "required": false
"auto" }
], },
"aliases": [ "is_activated": true,
"language", "maxPolarityValue": 1.0,
"l" "minPolarityValue": 0.0,
] "module": "sentiment-140",
} "name": "sentiment-140",
}, "requirements": {},
"@id": "sentiment140_0.1" "version": "0.1"
}, { },
"name": "rand", {
"is_activated": true, "@id": "plugins/ExamplePlugin_0.1",
"version": "0.1", "@type": "sentimentPlugin",
"extra_params": { "author": "@balkian",
"@id": "extra_params_rand_0.1", "custom_attribute": "42",
"language": { "description": "I am just an example",
"required": false, "extra_params": {
"@id": "lang_rand", "parameter": {
"options": [ "@id": "parameter",
"es", "aliases": [
"en", "parameter",
"auto" "param"
], ],
"aliases": [ "default": 42,
"language", "required": true
"l" }
] },
} "is_activated": true,
}, "maxPolarityValue": 1.0,
"@id": "rand_0.1" "minPolarityValue": 0.0,
} "module": "example",
] "name": "ExamplePlugin",
} "requirements": "noop",
"version": "0.1"
}
.. http:get:: /api/plugins/<pluginname> .. http:get:: /api/plugins/<pluginname>
@@ -162,30 +157,60 @@ NIF API
.. sourcecode:: http .. sourcecode:: http
{ {
"@id": "rand_0.1", "@context": "http://127.0.0.1/api/contexts/ExamplePlugin.jsonld",
"@type": "sentimentPlugin", "@id": "plugins/ExamplePlugin_0.1",
"extra_params": { "@type": "sentimentPlugin",
"@id": "extra_params_rand_0.1", "author": "@balkian",
"language": { "custom_attribute": "42",
"@id": "lang_rand", "description": "I am just an example",
"aliases": [ "extra_params": {
"language", "parameter": {
"l" "@id": "parameter",
], "aliases": [
"options": [ "parameter",
"es", "param"
"en", ],
"auto" "default": 42,
], "required": true
"required": false }
} },
}, "is_activated": true,
"is_activated": true, "maxPolarityValue": 1.0,
"name": "rand", "minPolarityValue": 0.0,
"version": "0.1" "module": "example",
"name": "ExamplePlugin",
"requirements": "noop",
"version": "0.1"
} }
.. http:get:: /api/plugins/default .. http:get:: /api/plugins/default
Return the information about the default plugin. Return the information about the default plugin.

7
docs/apischema.rst Normal file
View File

@@ -0,0 +1,7 @@
API and Examples
################
.. toctree::
vocabularies.rst
api.rst
examples.rst

View File

@@ -0,0 +1,78 @@
{
"@context": "http://mixedemotions-project.eu/ns/context.jsonld",
"@id": "me:Result1",
"@type": "results",
"analysis": [
"me:SAnalysis1",
"me:SgAnalysis1",
"me:EmotionAnalysis1",
"me:NER1",
{
"@type": "analysis",
"@id": "wrong"
}
],
"entries": [
{
"@id": "http://micro.blog/status1",
"@type": [
"nif:RFC5147String",
"nif:Context"
],
"nif:isString": "Dear Microsoft, put your Windows Phone on your newest #open technology program. You'll be awesome. #opensource",
"entities": [
{
"@id": "http://micro.blog/status1#char=5,13",
"nif:beginIndex": 5,
"nif:endIndex": 13,
"nif:anchorOf": "Microsoft",
"me:references": "http://dbpedia.org/page/Microsoft",
"prov:wasGeneratedBy": "me:NER1"
},
{
"@id": "http://micro.blog/status1#char=25,37",
"nif:beginIndex": 25,
"nif:endIndex": 37,
"nif:anchorOf": "Windows Phone",
"me:references": "http://dbpedia.org/page/Windows_Phone",
"prov:wasGeneratedBy": "me:NER1"
}
],
"suggestions": [
{
"@id": "http://micro.blog/status1#char=16,77",
"nif:beginIndex": 16,
"nif:endIndex": 77,
"nif:anchorOf": "put your Windows Phone on your newest #open technology program",
"prov:wasGeneratedBy": "me:SgAnalysis1"
}
],
"sentiments": [
{
"@id": "http://micro.blog/status1#char=80,97",
"nif:beginIndex": 80,
"nif:endIndex": 97,
"nif:anchorOf": "You'll be awesome.",
"marl:hasPolarity": "marl:Positive",
"marl:polarityValue": 0.9,
"prov:wasGeneratedBy": "me:SAnalysis1"
}
],
"emotions": [
{
"@id": "http://micro.blog/status1#char=0,109",
"nif:anchorOf": "Dear Microsoft, put your Windows Phone on your newest #open technology program. You'll be awesome. #opensource",
"prov:wasGeneratedBy": "me:EAnalysis1",
"onyx:hasEmotion": [
{
"onyx:hasEmotionCategory": "wna:liking"
},
{
"onyx:hasEmotionCategory": "wna:excitement"
}
]
}
]
}
]
}

9
docs/commandline.rst Normal file
View File

@@ -0,0 +1,9 @@
Command line
============
This video shows how to analyse text directly on the command line using the senpy tool.
.. image:: https://asciinema.org/a/9uwef1ghkjk062cw2t4mhzpyk.png
:width: 100%
:target: https://asciinema.org/a/9uwef1ghkjk062cw2t4mhzpyk
:alt: CLI demo

View File

@@ -37,6 +37,7 @@ extensions = [
'sphinx.ext.todo', 'sphinx.ext.todo',
'sphinxcontrib.httpdomain', 'sphinxcontrib.httpdomain',
'sphinx.ext.coverage', 'sphinx.ext.coverage',
'sphinx.ext.autosectionlabel',
] ]
# Add any paths that contain templates here, relative to this directory. # Add any paths that contain templates here, relative to this directory.
@@ -54,20 +55,21 @@ master_doc = 'index'
# General information about the project. # General information about the project.
project = u'Senpy' project = u'Senpy'
copyright = u'2016, J. Fernando Sánchez' copyright = u'2016, J. Fernando Sánchez'
description = u'A framework for sentiment and emotion analysis services'
# The version info for the project you're documenting, acts as replacement for # The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the # |version| and |release|, also used in various other places throughout the
# built documents. # built documents.
# #
# The short X.Y version. # The short X.Y version.
with open('../senpy/VERSION') as f: # with open('../senpy/VERSION') as f:
version = f.read().strip() # version = f.read().strip()
# The full version, including alpha/beta/rc tags. # The full version, including alpha/beta/rc tags.
release = version # release = version
# The language for content autogenerated by Sphinx. Refer to documentation # The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages. # for a list of supported languages.
#language = None language = None
# There are two options for replacing |today|: either, you set today to some # There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used: # non-false value, then it is used:
@@ -104,14 +106,14 @@ pygments_style = 'sphinx'
#keep_warnings = False #keep_warnings = False
html_theme = 'alabaster'
# -- Options for HTML output ---------------------------------------------- # -- Options for HTML output ----------------------------------------------
if not on_rtd: # only import and set the theme if we're building docs locally # if not on_rtd: # only import and set the theme if we're building docs locally
import sphinx_rtd_theme # import sphinx_rtd_theme
html_theme = 'sphinx_rtd_theme' # html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
else: # else:
html_theme = 'default' # html_theme = 'default'
# The theme to use for HTML and HTML Help pages. See the documentation for # The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes. # a list of builtin themes.
@@ -119,7 +121,13 @@ else:
# Theme options are theme-specific and customize the look and feel of a theme # Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the # further. For a list of options available for each theme, see the
# documentation. # documentation.
#html_theme_options = {} html_theme_options = {
'logo': 'header.png',
'github_user': 'gsi-upm',
'github_repo': 'senpy',
'github_banner': True,
}
# Add any paths that contain custom themes here, relative to this directory. # Add any paths that contain custom themes here, relative to this directory.
#html_theme_path = [] #html_theme_path = []
@@ -159,7 +167,13 @@ html_static_path = ['_static']
#html_use_smartypants = True #html_use_smartypants = True
# Custom sidebar templates, maps document names to template names. # Custom sidebar templates, maps document names to template names.
#html_sidebars = {} html_sidebars = {
'**': [
'about.html',
'navigation.html',
'searchbox.html',
]
}
# Additional templates that should be rendered to pages, maps page names to # Additional templates that should be rendered to pages, maps page names to
# template names. # template names.

View File

@@ -1,13 +1,13 @@
Introduction Conversion
------------ ----------
Senpy includes experimental support for emotion/sentiment conversion plugins. Senpy includes experimental support for emotion/sentiment conversion plugins.
Use Use
--- ===
Consider the original query: `http://127.0.0.1:5000/api/?i=hello&algo=emoRand`_ Consider the original query: http://127.0.0.1:5000/api/?i=hello&algo=emoRand
The requested plugin (emoRand) returns emotions using Ekman's model (or big6 in EmotionML): The requested plugin (emoRand) returns emotions using Ekman's model (or big6 in EmotionML):
@@ -28,7 +28,7 @@ The requested plugin (emoRand) returns emotions using Ekman's model (or big6 in
To get these emotions in VAD space (FSRE dimensions in EmotionML), we'd do this: To get these emotions in VAD space (FSRE dimensions in EmotionML), we'd do this:
`http://127.0.0.1:5000/api/?i=hello&algo=emoRand&emotionModel=emoml:fsre-dimensions`_ http://127.0.0.1:5000/api/?i=hello&algo=emoRand&emotionModel=emoml:fsre-dimensions
This call, provided there is a valid conversion plugin from Ekman's to VAD, would return something like this: This call, provided there is a valid conversion plugin from Ekman's to VAD, would return something like this:
@@ -87,7 +87,7 @@ It is also possible to get the original emotion nested within the new converted
Lastly, `conversion=filtered` would only return the converted emotions. Lastly, `conversion=filtered` would only return the converted emotions.
Developing a conversion plugin Developing a conversion plugin
------------------------------ ================================
Conversion plugins are discovered by the server just like any other plugin. Conversion plugins are discovered by the server just like any other plugin.
The difference is the slightly different API, and the need to specify the `source` and `target` of the conversion. The difference is the slightly different API, and the need to specify the `source` and `target` of the conversion.

View File

@@ -1,7 +1,8 @@
Demo Demo
---- ----
There is a demo available on http://senpy.demos.gsi.dit.upm.es/, where you can a serie of different plugins. You can use them in the playground or make a directly requests to the service. There is a demo available on http://senpy.cluster.gsi.dit.upm.es/, where you can test a serie of different plugins.
You can use the playground (a web interface) or make HTTP requests to the service API.
.. image:: senpy-playground.png .. image:: senpy-playground.png
:height: 400px :height: 400px
@@ -12,64 +13,4 @@ There is a demo available on http://senpy.demos.gsi.dit.upm.es/, where you can a
Plugins Demo Plugins Demo
============ ============
The next plugins are available at the demo: The source code and description of the plugins used in the demo is available here: https://lab.cluster.gsi.dit.upm.es/senpy/senpy-plugins-community/.
* emoTextAnew extracts the VAD (valence-arousal-dominance) of a sentence by matching words from the ANEW dictionary.
* emoTextWordnetAffect based on the hierarchy of WordnetAffect to calculate the emotion of the sentence.
* vaderSentiment utilizes the software from vaderSentiment to calculate the sentiment of a sentence.
* sentiText is a software developed during the TASS 2015 competition, it has been adapted for English and Spanish.
emoTextANEW plugin
******************
This plugin is going to used the ANEW lexicon dictionary to calculate de VAD (valence-arousal-dominance) of the sentence and the determinate which emotion is closer to this value.
Each emotion has a centroid, which it has been approximated using the formula described in this article:
http://www.aclweb.org/anthology/W10-0208
The plugin is going to look for the words in the sentence that appear in the ANEW dictionary and calculate the average VAD score for the sentence. Once this score is calculated, it is going to seek the emotion that is closest to this value.
emoTextWAF plugin
*****************
This plugin uses WordNet-Affect (http://wndomains.fbk.eu/wnaffect.html) to calculate the percentage of each emotion. The emotions that are going to be used are: anger, fear, disgust, joy and sadness. It is has been used a emotion mapping enlarge the emotions:
* anger : general-dislike
* fear : negative-fear
* disgust : shame
* joy : gratitude, affective, enthusiasm, love, joy, liking
* sadness : ingrattitude, daze, humlity, compassion, despair, anxiety, sadness
sentiText plugin
****************
This plugin is based in the classifier developed for the TASS 2015 competition. It has been developed for Spanish and English. The different phases that has this plugin when it is activated:
* Train both classifiers (English and Spanish).
* Initialize resources (dictionaries,stopwords,etc.).
* Extract bag of words,lemmas and chars.
Once the plugin is activated, the features that are going to be extracted for the classifiers are:
* Matches with the bag of words extracted from the train corpus.
* Sentiment score of the sentences extracted from the dictionaries (lexicons and emoticons).
* Identify negations and intensifiers in the sentences.
* Complementary features such as exclamation and interrogation marks, eloganted and caps words, hashtags, etc.
The plugin has a preprocessor, which is focues on Twitter corpora, that is going to be used for cleaning the text to simplify the feature extraction.
There is more information avaliable in the next article.
Aspect based Sentiment Analysis of Spanish Tweets, Oscar Araque and Ignacio Corcuera-Platas and Constantino Román-Gómez and Carlos A. Iglesias and J. Fernando Sánchez-Rada. http://gsi.dit.upm.es/es/investigacion/publicaciones?view=publication&task=show&id=37
vaderSentiment plugin
*********************
For developing this plugin, it has been used the module vaderSentiment, which is described in the paper: VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text C.J. Hutto and Eric Gilbert Eighth International Conference on Weblogs and Social Media (ICWSM-14). Ann Arbor, MI, June 2014.
If you use this plugin in your research, please cite the above paper
For more information about the functionality, check the official repository
https://github.com/cjhutto/vaderSentiment

78
docs/examples.rst Normal file
View File

@@ -0,0 +1,78 @@
Examples
------
All the examples in this page use the :download:`the main schema <_static/schemas/definitions.json>`.
Simple NIF annotation
.....................
Description
,,,,,,,,,,,
This example covers the basic example in the NIF documentation: `<http://persistence.uni-leipzig.org/nlp2rdf/ontologies/nif-core/nif-core.html>`_.
Representation
,,,,,,,,,,,,,,
.. literalinclude:: examples/results/example-basic.json
:language: json-ld
Sentiment Analysis
.....................
Description
,,,,,,,,,,,
This annotation corresponds to the sentiment analysis of an input. The example shows the sentiment represented according to Marl format.
The sentiments detected are contained in the Sentiments array with their related part of the text.
Representation
,,,,,,,,,,,,,,
.. literalinclude:: examples/results/example-sentiment.json
:emphasize-lines: 5-10,25-33
:language: json-ld
Suggestion Mining
.................
Description
,,,,,,,,,,,
The suggestions schema represented below shows the suggestions detected in the text. Within it, we can find the NIF fields highlighted that corresponds to the text of the detected suggestion.
Representation
,,,,,,,,,,,,,,
.. literalinclude:: examples/results/example-suggestion.json
:emphasize-lines: 5-8,22-27
:language: json-ld
Emotion Analysis
................
Description
,,,,,,,,,,,
This annotation represents the emotion analysis of an input to Senpy. The emotions are contained in the emotions section with the text that refers to following Onyx format and the emotion model defined beforehand.
Representation
,,,,,,,,,,,,,,
.. literalinclude:: examples/results/example-emotion.json
:language: json-ld
:emphasize-lines: 5-8,25-37
Named Entity Recognition
........................
Description
,,,,,,,,,,,
The Named Entity Recognition is represented as follows. In this particular case, it can be seen within the entities array the entities recognised. For the example input, Microsoft and Windows Phone are the ones detected.
Representation
,,,,,,,,,,,,,,
.. literalinclude:: examples/results/example-ner.json
:emphasize-lines: 5-8,19-34
:language: json-ld
Complete example
................
Description
,,,,,,,,,,,
This example covers all of the above cases, integrating all the annotations in the same document.
Representation
,,,,,,,,,,,,,,
.. literalinclude:: examples/results/example-complete.json
:language: json-ld

View File

@@ -0,0 +1,74 @@
{
"@context": "http://mixedemotions-project.eu/ns/context.jsonld",
"@id": "me:Result1",
"@type": "results",
"analysis": [
"me:SAnalysis1",
"me:SgAnalysis1",
"me:EmotionAnalysis1",
"me:NER1"
],
"entries": [
{
"@id": "http://micro.blog/status1",
"@type": [
"nif:RFC5147String",
"nif:Context"
],
"nif:isString": "Dear Microsoft, put your Windows Phone on your newest #open technology program. You'll be awesome. #opensource",
"entities": [
{
"@id": "http://micro.blog/status1#char=5,13",
"nif:beginIndex": 5,
"nif:endIndex": 13,
"nif:anchorOf": "Microsoft",
"me:references": "http://dbpedia.org/page/Microsoft",
"prov:wasGeneratedBy": "me:NER1"
},
{
"@id": "http://micro.blog/status1#char=25,37",
"nif:beginIndex": 25,
"nif:endIndex": 37,
"nif:anchorOf": "Windows Phone",
"me:references": "http://dbpedia.org/page/Windows_Phone",
"prov:wasGeneratedBy": "me:NER1"
}
],
"suggestions": [
{
"@id": "http://micro.blog/status1#char=16,77",
"nif:beginIndex": 16,
"nif:endIndex": 77,
"nif:anchorOf": "put your Windows Phone on your newest #open technology program",
"prov:wasGeneratedBy": "me:SgAnalysis1"
}
],
"sentiments": [
{
"@id": "http://micro.blog/status1#char=80,97",
"nif:beginIndex": 80,
"nif:endIndex": 97,
"nif:anchorOf": "You'll be awesome.",
"marl:hasPolarity": "marl:Positive",
"marl:polarityValue": 0.9,
"prov:wasGeneratedBy": "me:SAnalysis1"
}
],
"emotions": [
{
"@id": "http://micro.blog/status1#char=0,109",
"nif:anchorOf": "Dear Microsoft, put your Windows Phone on your newest #open technology program. You'll be awesome. #opensource",
"prov:wasGeneratedBy": "me:EAnalysis1",
"onyx:hasEmotion": [
{
"onyx:hasEmotionCategory": "wna:liking"
},
{
"onyx:hasEmotionCategory": "wna:excitement"
}
]
}
]
}
]
}

View File

@@ -1,19 +1,35 @@
.. Senpy documentation master file, created by
sphinx-quickstart on Tue Feb 24 08:57:32 2015.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to Senpy's documentation! Welcome to Senpy's documentation!
================================= =================================
.. image:: https://readthedocs.org/projects/senpy/badge/?version=latest
:target: http://senpy.readthedocs.io/en/latest/
.. image:: https://badge.fury.io/py/senpy.svg
:target: https://badge.fury.io/py/senpy
.. image:: https://lab.cluster.gsi.dit.upm.es/senpy/senpy/badges/master/build.svg
:target: https://lab.cluster.gsi.dit.upm.es/senpy/senpy/commits/master
.. image:: https://lab.cluster.gsi.dit.upm.es/senpy/senpy/badges/master/coverage.svg
:target: https://lab.cluster.gsi.dit.upm.es/senpy/senpy/commits/master
.. image:: https://img.shields.io/pypi/l/requests.svg
:target: https://lab.cluster.gsi.dit.upm.es/senpy/senpy/
Contents:
Senpy is a framework for sentiment and emotion analysis services.
Services built with senpy are interchangeable and easy to use because they share a common :doc:`apischema`.
It also simplifies service development.
.. image:: senpy-architecture.png
:width: 100%
:align: center
.. toctree:: .. toctree::
:caption: Learn more about senpy:
:maxdepth: 2
senpy senpy
installation installation
usage
api
schema
plugins
demo demo
:maxdepth: 2 usage
apischema
plugins
conversion
about

View File

@@ -1,6 +1,16 @@
Installation Installation
------------ ------------
The stable version can be installed in three ways. The stable version can be used in two ways: as a system/user library through pip, or as a docker image.
The docker image is the recommended way because it is self-contained and isolated from the system, which means:
* Downloading and using it is just one command
* All dependencies are included
* It is OS-independent (MacOS, Windows, GNU/Linux)
* Several versions may coexist in the same machine without additional virtual environments
Additionally, you may create your own docker image with your custom plugins, ready to be used by others.
Through PIP Through PIP
*********** ***********
@@ -22,6 +32,41 @@ If you want to install senpy globally, use sudo instead of the ``--user`` flag.
Docker Image Docker Image
************ ************
Build the image or use the pre-built one: ``docker run -ti -p 5000:5000 gsiupm/senpy --host 0.0.0.0 --default-plugins``. Build the image or use the pre-built one:
To add custom plugins, add a volume and tell senpy where to find the plugins: ``docker run -ti -p 5000:5000 -v <PATH OF PLUGINS>:/plugins gsiupm/senpy --host 0.0.0.0 --default-plugins -f /plugins`` .. code:: bash
docker run -ti -p 5000:5000 gsiupm/senpy --host 0.0.0.0 --default-plugins
To add custom plugins, use a docker volume:
.. code:: bash
docker run -ti -p 5000:5000 -v <PATH OF PLUGINS>:/plugins gsiupm/senpy --host 0.0.0.0 --default-plugins -f /plugins
Python 2
........
There is a Senpy version for python2 too:
.. code:: bash
docker run -ti -p 5000:5000 gsiupm/senpy:python2.7 --host 0.0.0.0 --default-plugins
Alias
.....
If you are using the docker approach regularly, it is advisable to use a script or an alias to simplify your executions:
.. code:: bash
alias senpy='docker run --rm -ti -p 5000:5000 -v $PWD:/senpy-plugins gsiupm/senpy --default-plugins'
Now, you may run senpy from any folder in your computer like so:
.. code:: bash
senpy --version

View File

@@ -2,27 +2,36 @@ Developing new plugins
---------------------- ----------------------
This document describes how to develop a new analysis plugin. For an example of conversion plugins, see :doc:`conversion`. This document describes how to develop a new analysis plugin. For an example of conversion plugins, see :doc:`conversion`.
Each plugin represents a different analysis process.There are two types of files that are needed by senpy for loading a plugin: A more step-by-step tutorial with slides is available `here <https://lab.cluster.gsi.dit.upm.es/senpy/senpy-tutorial>`__
- Definition file, has the ".senpy" extension. .. contents:: :local:
- Code file, is a python file.
This separation will allow us to deploy plugins that use the same code but employ different parameters. What is a plugin?
=================
A plugin is a program that, given a text, will add annotations to it.
In practice, a plugin consists of at least two files:
- Definition file: a `.senpy` file that describes the plugin (e.g. what input parameters it accepts, what emotion model it uses).
- Python module: the actual code that will add annotations to each input.
This separation allows us to deploy plugins that use the same code but employ different parameters.
For instance, one could use the same classifier and processing in several plugins, but train with different datasets. For instance, one could use the same classifier and processing in several plugins, but train with different datasets.
This scenario is particularly useful for evaluation purposes. This scenario is particularly useful for evaluation purposes.
The only limitation is that the name of each plugin needs to be unique. The only limitation is that the name of each plugin needs to be unique.
Plugins Definitions Plugin Definition files
=================== =======================
The definition file contains all the attributes of the plugin, and can be written in YAML or JSON. The definition file contains all the attributes of the plugin, and can be written in YAML or JSON.
When the server is launched, it will recursively search for definition files in the plugin folder (the current folder, by default).
The most important attributes are: The most important attributes are:
* **name**: unique name that senpy will use internally to identify the plugin. * **name**: unique name that senpy will use internally to identify the plugin.
* **module**: indicates the module that contains the plugin code, which will be automatically loaded by senpy. * **module**: indicates the module that contains the plugin code, which will be automatically loaded by senpy.
* **version** * **version**
* extra_params: used to specify parameters that the plugin accepts that are not already part of the senpy API. Those parameters may be required, and have aliased names. For instance: * extra_params: to add parameters to the senpy API when this plugin is requested. Those parameters may be required, and have aliased names. For instance:
.. code:: yaml .. code:: yaml
@@ -68,10 +77,28 @@ The basic methods in a plugin are:
* __init__ * __init__
* activate: used to load memory-hungry resources * activate: used to load memory-hungry resources
* deactivate: used to free up resources * deactivate: used to free up resources
* analyse_entry: called in every user requests. It takes in the parameters supplied by a user and should yield one or more ``Entry`` objects. * analyse_entry: called in every user requests. It takes two parameters: ``Entry``, the entry object, and ``params``, the parameters supplied by the user. It should yield one or more ``Entry`` objects.
Plugins are loaded asynchronously, so don't worry if the activate method takes too long. The plugin will be marked as activated once it is finished executing the method. Plugins are loaded asynchronously, so don't worry if the activate method takes too long. The plugin will be marked as activated once it is finished executing the method.
Entries
=======
Entries are objects that can be annotated.
By default, entries are `NIF contexts <http://persistence.uni-leipzig.org/nlp2rdf/ontologies/nif-core/nif-core.html>`_ represented in JSON-LD format.
Annotations are added to the object like this:
.. code:: python
entry = Entry()
entry.vocabulary__annotationName = 'myvalue'
entry['vocabulary:annotationName'] = 'myvalue'
entry['annotationNameURI'] = 'myvalue'
Where vocabulary is one of the prefixes defined in the default senpy context, and annotationURI is a full URI.
The value may be any valid JSON-LD dictionary.
For simplicity, senpy includes a series of models by default in the ``senpy.models`` module.
Example plugin Example plugin
============== ==============
@@ -88,7 +115,7 @@ The definition file would look like this:
module: helloworld module: helloworld
version: 0.0 version: 0.0
threshold: 10 threshold: 10
description: Hello World
Now, in a file named ``helloworld.py``: Now, in a file named ``helloworld.py``:
@@ -97,11 +124,11 @@ Now, in a file named ``helloworld.py``:
#!/bin/env python #!/bin/env python
#helloworld.py #helloworld.py
from senpy.plugins import SenpyPlugin from senpy.plugins import AnalysisPlugin
from senpy.models import Sentiment from senpy.models import Sentiment
class HelloWorld(SenpyPlugin): class HelloWorld(AnalysisPlugin):
def analyse_entry(entry, params): def analyse_entry(entry, params):
'''Basically do nothing with each entry''' '''Basically do nothing with each entry'''
@@ -114,15 +141,112 @@ Now, in a file named ``helloworld.py``:
entry.sentiments.append(sentiment) entry.sentiments.append(sentiment)
yield entry yield entry
The complete code of the example plugin is available `here <https://lab.cluster.gsi.dit.upm.es/senpy/plugin-prueba>`__.
Loading data and files
======================
Most plugins will need access to files (dictionaries, lexicons, etc.).
It is good practice to specify the paths of these files in the plugin configuration, so the same code can be reused with different resources.
.. code:: yaml
name: dictworld
module: dictworld
dictionary_path: <PATH OF THE FILE>
The path can be either absolute, or relative.
From absolute paths
???????????????????
Absolute paths (such as ``/data/dictionary.csv`` are straightfoward:
.. code:: python
with open(os.path.join(self.dictionary_path) as f:
...
From relative paths
???????????????????
Since plugins are loading dynamically, relative paths will refer to the current working directory.
Instead, what you usually want is to load files *relative to the plugin source folder*, like so:
::
.
..
plugin.senpy
plugin.py
dictionary.csv
For this, we need to first get the path of your source folder first, like so:
.. code:: python
import os
root = os.path.realpath(__file__)
with open(os.path.join(root, self.dictionary_path) as f:
...
Docker image
============
Add the following dockerfile to your project to generate a docker image with your plugin:
.. code:: dockerfile
FROM gsiupm/senpy:0.8.8
This will copy your source folder to the image, and install all dependencies.
Now, to build an image:
.. code:: shell
docker build . -t gsiupm/exampleplugin
And you can run it with:
.. code:: shell
docker run -p 5000:5000 gsiupm/exampleplugin
If the plugin non-source files (:ref:`loading data and files`), the recommended way is to use absolute paths.
Data can then be mounted in the container or added to the image.
The former is recommended for open source plugins with licensed resources, whereas the latter is the most convenient and can be used for private images.
Mounting data:
.. code:: bash
docker run -v $PWD/data:/data gsiupm/exampleplugin
Adding data to the image:
.. code:: dockerfile
FROM gsiupm/senpy:0.8.8
COPY data /
F.A.Q. F.A.Q.
====== ======
What annotations can I use?
???????????????????????????
You can add almost any annotation to an entry.
The most common use cases are covered in the :doc:`apischema`.
Why does the analyse function yield instead of return? Why does the analyse function yield instead of return?
?????????????????????????????????????????????????????? ??????????????????????????????????????????????????????
This is so that plugins may add new entries to the response or filter some of them. This is so that plugins may add new entries to the response or filter some of them.
For instance, a `context detection` plugin may add a new entry for each context in the original entry. For instance, a `context detection` plugin may add a new entry for each context in the original entry.
On the other hand, a conveersion plugin may leave out those entries that do not contain relevant information. On the other hand, a conversion plugin may leave out those entries that do not contain relevant information.
If I'm using a classifier, where should I train it? If I'm using a classifier, where should I train it?
@@ -132,9 +256,9 @@ Training a classifier can be time time consuming. To avoid running the training
.. code:: python .. code:: python
from senpy.plugins import ShelfMixin, SenpyPlugin from senpy.plugins import ShelfMixin, AnalysisPlugin
class MyPlugin(ShelfMixin, SenpyPlugin): class MyPlugin(ShelfMixin, AnalysisPlugin):
def train(self): def train(self):
''' Code to train the classifier ''' Code to train the classifier
''' '''
@@ -151,12 +275,16 @@ Training a classifier can be time time consuming. To avoid running the training
def deactivate(self): def deactivate(self):
self.close() self.close()
You can speficy a 'shelf_file' in your .senpy file. By default the ShelfMixin creates a file based on the plugin name and stores it in that plugin's folder. You can specify a 'shelf_file' in your .senpy file. By default the ShelfMixin creates a file based on the plugin name and stores it in that plugin's folder.
I want to implement my service as a plugin, How i can do it? Shelves may get corrupted if the plugin exists unexpectedly.
???????????????????????????????????????????????????????????? A corrupt shelf prevents the plugin from loading.
If you do not care about the pickle, you can force your plugin to remove the corrupted file and load anyway, set the 'force_shelf' to True in your .senpy file.
This example ilustrate how to implement the Sentiment140 service as a plugin in senpy How can I turn an external service into a plugin?
?????????????????????????????????????????????????
This example ilustrate how to implement a plugin that accesses the Sentiment140 service.
.. code:: python .. code:: python
@@ -190,26 +318,30 @@ This example ilustrate how to implement the Sentiment140 service as a plugin in
yield entry yield entry
Where can I define extra parameters to be introduced in the request to my plugin? Can my plugin require additional parameters from the user?
????????????????????????????????????????????????????????????????????????????????? ??????????????????????????????????????????????????????????
You can add these parameters in the definition file under the attribute "extra_params" : "{param_name}". The name of the parameter has new attributes-value pairs. The basic attributes are: You can add extra parameters in the definition file under the attribute ``extra_params``.
It takes a dictionary, where the keys are the name of the argument/parameter, and the value has the following fields:
* aliases: the different names which can be used in the request to use the parameter. * aliases: the different names which can be used in the request to use the parameter.
* required: this option is a boolean and indicates if the parameters is binding in operation plugin. * required: if set to true, users need to provide this parameter unless a default is set.
* options: the different values of the paremeter. * options: the different acceptable values of the parameter (i.e. an enum). If set, the value provided must match one of the options.
* default: the default value of the parameter, this is useful in case the paremeter is required and you want to have a default value. * default: the default value of the parameter, if none is provided in the request.
.. code:: python .. code:: python
"extra_params": { extra_params
"language": { language:
"aliases": ["language", "l"], aliases:
"required": true, - language
"options": ["es","en"], - lang
"default": "es" - l
} required: true,
} options:
- es
- en
default: es
This example shows how to introduce a parameter associated with language. This example shows how to introduce a parameter associated with language.
The extraction of this paremeter is used in the analyse method of the Plugin interface. The extraction of this paremeter is used in the analyse method of the Plugin interface.
@@ -241,7 +373,6 @@ Additionally, with the ``--pdb`` option you will be dropped into a pdb post mort
senpy --pdb senpy --pdb
Where can I find more code examples? Where can I find more code examples?
???????????????????????????????????? ????????????????????????????????????

View File

@@ -1 +1,2 @@
sphinxcontrib-httpdomain>=1.4 sphinxcontrib-httpdomain>=1.4
nbsphinx

View File

@@ -1,74 +0,0 @@
Schema Examples
===============
All the examples in this page use the :download:`the main schema <_static/schemas/definitions.json>`.
Simple NIF annotation
---------------------
Description
...........
This example covers the basic example in the NIF documentation: `<http://persistence.uni-leipzig.org/nlp2rdf/ontologies/nif-core/nif-core.html>`_.
Representation
..............
.. literalinclude:: examples/example-basic.json
:language: json-ld
Sentiment Analysis
---------------------
Description
...........
Representation
..............
.. literalinclude:: examples/example-sentiment.json
:emphasize-lines: 5-10,25-33
:language: json-ld
Suggestion Mining
-----------------
Description
...........
Representation
..............
.. literalinclude:: examples/example-suggestion.json
:emphasize-lines: 5-8,22-27
:language: json-ld
Emotion Analysis
----------------
Description
...........
Representation
..............
.. literalinclude:: examples/example-emotion.json
:language: json-ld
:emphasize-lines: 5-8,25-37
Named Entity Recognition
------------------------
Description
...........
Representation
..............
.. literalinclude:: examples/example-ner.json
:emphasize-lines: 5-8,19-34
:language: json-ld
Complete example
----------------
Description
...........
This example covers all of the above cases, integrating all the annotations in the same document.
Representation
..............
.. literalinclude:: examples/example-complete.json
:language: json-ld

Binary file not shown.

Before

Width:  |  Height:  |  Size: 65 KiB

After

Width:  |  Height:  |  Size: 122 KiB

BIN
docs/senpy-framework.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 48 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 79 KiB

After

Width:  |  Height:  |  Size: 49 KiB

View File

@@ -1,20 +1,38 @@
What is Senpy? What is Senpy?
-------------- --------------
Senpy is an open source reference implementation of a linked data model for sentiment and emotion analysis services based on the vocabularies NIF, Marl and Onyx. Web services can get really complex: data validation, user interaction, formatting, logging., etc.
The figure below summarizes the typical features in an analysis service.
Senpy implements all the common blocks, so developers can focus on what really matters: great analysis algorithms that solve real problems.
The overall goal of the reference implementation Senpy is easing the adoption of the proposed linked data model for sentiment and emotion analysis services, so that services from different providers become interoperable. With this aim, the design of the reference implementation has focused on its extensibility and reusability. .. image:: senpy-framework.png
:width: 60%
:align: center
A modular approach allows organizations to replace individual components with custom ones developed in-house. Furthermore, organizations can benefit from reusing prepackages modules that provide advanced functionalities, such as algorithms for sentiment and emotion analysis, linked data publication or emotion and sentiment mapping between different providers.
Specifications Senpy for end users
============== ===================
The model used in Senpy is based on the following specifications: All services built using senpy share a common interface.
This allows users to use them (almost) interchangeably.
Senpy comes with a :ref:`built-in client`.
* Marl, a vocabulary designed to annotate and describe subjetive opinions expressed on the web or in information systems.
* Onyx, which is built one the same principles as Marl to annotate and describe emotions, and provides interoperability with Emotion Markup Language. Senpy for service developers
* NIF 2.0, which defines a semantic format and APO for improving interoperability among natural language processing services ============================
Senpy is a framework that turns your sentiment or emotion analysis algorithm into a full blown semantic service.
Senpy takes care of:
* Interfacing with the user: parameter validation, error handling.
* Formatting: JSON-LD, Turtle/n-triples input and output, or simple text input
* Linked Data: senpy results are semantically annotated, using a series of well established vocabularies, and sane default URIs.
* User interface: a web UI where users can explore your service and test different settings
* A client to interact with the service. Currently only available in Python.
Sharing your sentiment analysis with the world has never been easier!
Check out the :doc:`plugins` if you have developed an analysis algorithm (e.g. sentiment analysis) and you want to publish it as a service.
Architecture Architecture
============ ============
@@ -29,7 +47,5 @@ Senpy proposes a modular and dynamic architecture that allows:
The framework consists of two main modules: Senpy core, which is the building block of the service, and Senpy plugins, which consist of the analysis algorithm. The next figure depicts a simplified version of the processes involved in an analysis with the Senpy framework. The framework consists of two main modules: Senpy core, which is the building block of the service, and Senpy plugins, which consist of the analysis algorithm. The next figure depicts a simplified version of the processes involved in an analysis with the Senpy framework.
.. image:: senpy-architecture.png .. image:: senpy-architecture.png
:height: 400px :width: 100%
:width: 800px
:scale: 100 %
:align: center :align: center

58
docs/server.rst Normal file
View File

@@ -0,0 +1,58 @@
Server
======
The senpy server is launched via the `senpy` command:
.. code:: text
usage: senpy [-h] [--level logging_level] [--debug] [--default-plugins]
[--host HOST] [--port PORT] [--plugins-folder PLUGINS_FOLDER]
[--only-install]
Run a Senpy server
optional arguments:
-h, --help show this help message and exit
--level logging_level, -l logging_level
Logging level
--debug, -d Run the application in debug mode
--default-plugins Load the default plugins
--host HOST Use 0.0.0.0 to accept requests from any host.
--port PORT, -p PORT Port to listen on.
--plugins-folder PLUGINS_FOLDER, -f PLUGINS_FOLDER
Where to look for plugins.
--only-install, -i Do not run a server, only install plugin dependencies
When launched, the server will recursively look for plugins in the specified plugins folder (the current working directory by default).
For every plugin found, it will download its dependencies, and try to activate it.
The default server includes a playground and an endpoint with all plugins found.
Let's run senpy with the default plugins:
.. code:: bash
senpy -f . --default-plugins
Now go to `http://localhost:5000 <http://localhost:5000>`_, you should be greeted by the senpy playground:
.. image:: senpy-playground.png
:width: 100%
:alt: Playground
The playground is a user-friendly way to test your plugins, but you can always use the service directly: `http://localhost:5000/api?input=hello <http://localhost:5000/api?input=hello>`_.
By default, senpy will listen only on the `127.0.0.1` address.
That means you can only access the API from your (or localhost).
You can listen on a different address using the `--host` flag (e.g., 0.0.0.0).
The default port is 5000.
You can change it with the `--port` flag.
For instance, to accept connections on port 6000 on any interface:
.. code:: bash
senpy --host 0.0.0.0 --port 6000
For more options, see the `--help` page.

View File

@@ -1,76 +1,15 @@
Usage Usage
----- -----
The easiest and recommended way is to just use the command-line tool to load your plugins and launch the server. First of all, you need to install the package.
See :doc:`installation` for instructions.
Once installed, the `senpy` command should be available.
.. code:: bash .. toctree::
:maxdepth: 1
senpy server
SenpyClientUse
Or, alternatively: commandline
.. code:: bash
python -m senpy
This will create a server with any modules found in the current path.
Useful command-line options
===========================
In case you want to load modules, which are located in different folders under the root folder, use the next option.
.. code:: bash
python -m senpy -f .
The default port used by senpy is 5000, but you can change it using the option `--port`.
.. code:: bash
python -m senpy --port 8080
Also, the host can be changed where senpy is deployed. The default value is `127.0.0.1`.
.. code:: bash
python -m senpy --host 0.0.0.0
For more options, see the `--help` page.
Alternatively, you can use the modules included in senpy to build your own application.
Senpy server
============
Once the server is launched, there is a basic endpoint in the server, which provides a playground to use the plugins that have been loaded.
In case you want to know the different endpoints of the server, there is more information available in the NIF API section_.
CLI
===
This video shows how to use senpy through command-line tool.
https://asciinema.org/a/9uwef1ghkjk062cw2t4mhzpyk
Request example in python
=========================
This example shows how to make a request to the default plugin:
.. code:: python
from senpy.client import Client
c = Client('http://127.0.0.1:5000/api/')
r = c.analyse('hello world')
for entry in r.entries:
print('{} -> {}'.format(entry.text, entry.emotions))
.. _section: http://senpy.readthedocs.org/en/latest/api.html

8
docs/vocabularies.rst Normal file
View File

@@ -0,0 +1,8 @@
Vocabularies and model
======================
The model used in Senpy is based on the following vocabularies:
* Marl, a vocabulary designed to annotate and describe subjetive opinions expressed on the web or in information systems.
* Onyx, which is built one the same principles as Marl to annotate and describe emotions, and provides interoperability with Emotion Markup Language.
* NIF 2.0, which defines a semantic format and APO for improving interoperability among natural language processing services

7
k8s/README.md Normal file
View File

@@ -0,0 +1,7 @@
Deploy senpy to a kubernetes cluster.
Usage:
```
kubectl apply -f . -n senpy
```

26
k8s/senpy-deployment.yaml Normal file
View File

@@ -0,0 +1,26 @@
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: senpy-latest
spec:
replicas: 1
template:
metadata:
labels:
role: senpy-latest
app: test
spec:
containers:
- name: senpy-latest
image: gsiupm/senpy:latest
imagePullPolicy: Always
args:
- "--default-plugins"
resources:
limits:
memory: "512Mi"
cpu: "1000m"
ports:
- name: web
containerPort: 5000

14
k8s/senpy-ingress.yaml Normal file
View File

@@ -0,0 +1,14 @@
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: senpy-ingress
spec:
rules:
- host: latest.senpy.cluster.gsi.dit.upm.es
http:
paths:
- path: /
backend:
serviceName: senpy-latest
servicePort: 5000

12
k8s/senpy-svc.yaml Normal file
View File

@@ -0,0 +1,12 @@
---
apiVersion: v1
kind: Service
metadata:
name: senpy-latest
spec:
type: ClusterIP
ports:
- port: 5000
protocol: TCP
selector:
role: senpy-latest

View File

@@ -1,6 +1,6 @@
Flask>=0.10.1 Flask>=0.10.1
requests>=2.4.1 requests>=2.4.1
gevent>=1.1rc4 tornado>=4.4.3
PyLD>=0.6.5 PyLD>=0.6.5
six six
future future

View File

@@ -17,7 +17,6 @@
""" """
Sentiment analysis server in Python Sentiment analysis server in Python
""" """
from __future__ import print_function
from .version import __version__ from .version import __version__
import logging import logging

View File

@@ -22,35 +22,15 @@ the server.
from flask import Flask from flask import Flask
from senpy.extensions import Senpy from senpy.extensions import Senpy
from gevent.wsgi import WSGIServer
from gevent.monkey import patch_all
import logging import logging
import os import os
import sys
import argparse import argparse
import senpy import senpy
patch_all(thread=False)
SERVER_PORT = os.environ.get("PORT", 5000) SERVER_PORT = os.environ.get("PORT", 5000)
def info(type, value, tb):
if hasattr(sys, 'ps1') or not sys.stderr.isatty():
# we are in interactive mode or we don't have a tty-like
# device, so we call the default hook
sys.__excepthook__(type, value, tb)
else:
import traceback
import pdb
# we are NOT in interactive mode, print the exception...
traceback.print_exception(type, value, tb)
print
# ...then start the debugger in post-mortem mode.
# pdb.pm() # deprecated
pdb.post_mortem(tb) # more "modern"
def main(): def main():
parser = argparse.ArgumentParser(description='Run a Senpy server') parser = argparse.ArgumentParser(description='Run a Senpy server')
parser.add_argument( parser.add_argument(
@@ -94,28 +74,38 @@ def main():
action='store_true', action='store_true',
default=False, default=False,
help='Do not run a server, only install plugin dependencies') help='Do not run a server, only install plugin dependencies')
parser.add_argument(
'--threaded',
action='store_false',
default=True,
help='Run a threaded server')
parser.add_argument(
'--version',
'-v',
action='store_true',
default=False,
help='Output the senpy version and exit')
args = parser.parse_args() args = parser.parse_args()
if args.version:
print('Senpy version {}'.format(senpy.__version__))
exit(1)
logging.basicConfig() logging.basicConfig()
rl = logging.getLogger() rl = logging.getLogger()
rl.setLevel(getattr(logging, args.level)) rl.setLevel(getattr(logging, args.level))
app = Flask(__name__) app = Flask(__name__)
app.debug = args.debug app.debug = args.debug
if args.debug:
sys.excepthook = info
sp = Senpy(app, args.plugins_folder, default_plugins=args.default_plugins) sp = Senpy(app, args.plugins_folder, default_plugins=args.default_plugins)
if args.only_install: if args.only_install:
sp.install_deps() sp.install_deps()
return return
sp.activate_all() sp.activate_all()
http_server = WSGIServer((args.host, args.port), app) print('Senpy version {}'.format(senpy.__version__))
try: print('Server running on port %s:%d. Ctrl+C to quit' % (args.host,
print('Senpy version {}'.format(senpy.__version__)) args.port))
print('Server running on port %s:%d. Ctrl+C to quit' % (args.host, app.run(args.host,
args.port)) args.port,
http_server.serve_forever() threaded=args.threaded,
except KeyboardInterrupt: debug=app.debug)
print('Bye!')
http_server.stop()
sp.deactivate_all() sp.deactivate_all()

View File

@@ -26,6 +26,13 @@ API_PARAMS = {
"aliases": ["emotionModel", "emoModel"], "aliases": ["emotionModel", "emoModel"],
"required": False "required": False
}, },
"plugin_type": {
"@id": "pluginType",
"description": 'What kind of plugins to list',
"aliases": ["pluginType", "plugin_type"],
"required": True,
"default": "analysisPlugin"
},
"conversion": { "conversion": {
"@id": "conversion", "@id": "conversion",
"description": "How to show the elements that have (not) been converted", "description": "How to show the elements that have (not) been converted",
@@ -63,7 +70,7 @@ NIF_PARAMS = {
"aliases": ["f", "informat"], "aliases": ["f", "informat"],
"required": False, "required": False,
"default": "text", "default": "text",
"options": ["turtle", "text"], "options": ["turtle", "text", "json-ld"],
}, },
"intype": { "intype": {
"@id": "intype", "@id": "intype",

View File

@@ -25,6 +25,7 @@ from .version import __version__
from functools import wraps from functools import wraps
import logging import logging
import json
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -74,7 +75,7 @@ def basic_api(f):
@wraps(f) @wraps(f)
def decorated_function(*args, **kwargs): def decorated_function(*args, **kwargs):
raw_params = get_params(request) raw_params = get_params(request)
headers = {'X-ORIGINAL-PARAMS': raw_params} headers = {'X-ORIGINAL-PARAMS': json.dumps(raw_params)}
# Get defaults # Get defaults
web_params = parse_params({}, spec=WEB_PARAMS) web_params = parse_params({}, spec=WEB_PARAMS)
api_params = parse_params({}, spec=API_PARAMS) api_params = parse_params({}, spec=API_PARAMS)
@@ -92,6 +93,9 @@ def basic_api(f):
response = f(*args, **kwargs) response = f(*args, **kwargs)
except Error as ex: except Error as ex:
response = ex response = ex
logger.error(ex)
if current_app.debug:
raise
in_headers = web_params['inHeaders'] != "0" in_headers = web_params['inHeaders'] != "0"
expanded = api_params['expanded-jsonld'] expanded = api_params['expanded-jsonld']
@@ -121,7 +125,9 @@ def api():
@basic_api @basic_api
def plugins(): def plugins():
sp = current_app.senpy sp = current_app.senpy
dic = Plugins(plugins=list(sp.plugins.values())) ptype = request.params.get('plugin_type')
plugins = sp.filter_plugins(plugin_type=ptype)
dic = Plugins(plugins=list(plugins.values()))
return dic return dic

View File

@@ -1,6 +1,7 @@
import requests import requests
import logging import logging
from . import models from . import models
from .plugins import default_plugin_type
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -12,12 +13,15 @@ class Client(object):
def analyse(self, input, method='GET', **kwargs): def analyse(self, input, method='GET', **kwargs):
return self.request('/', method=method, input=input, **kwargs) return self.request('/', method=method, input=input, **kwargs)
def plugins(self, ptype=default_plugin_type):
resp = self.request(path='/plugins', plugin_type=ptype).plugins
return {p.name: p for p in resp}
def request(self, path=None, method='GET', **params): def request(self, path=None, method='GET', **params):
url = '{}{}'.format(self.endpoint, path) url = '{}{}'.format(self.endpoint, path)
response = requests.request(method=method, url=url, params=params) response = requests.request(method=method, url=url, params=params)
try: try:
resp = models.from_dict(response.json()) resp = models.from_dict(response.json())
resp.validate(resp)
except Exception as ex: except Exception as ex:
logger.error(('There seems to be a problem with the response:\n' logger.error(('There seems to be a problem with the response:\n'
'\tURL: {url}\n' '\tURL: {url}\n'

View File

@@ -5,8 +5,9 @@ It orchestrates plugin (de)activation and analysis.
from future import standard_library from future import standard_library
standard_library.install_aliases() standard_library.install_aliases()
from .plugins import SentimentPlugin, SenpyPlugin from . import plugins
from .models import Error, Entry, Results from .plugins import SenpyPlugin
from .models import Error, Entry, Results, from_string
from .blueprints import api_blueprint, demo_blueprint, ns_blueprint from .blueprints import api_blueprint, demo_blueprint, ns_blueprint
from .api import API_PARAMS, NIF_PARAMS, parse_params from .api import API_PARAMS, NIF_PARAMS, parse_params
@@ -21,11 +22,18 @@ import importlib
import logging import logging
import traceback import traceback
import yaml import yaml
import pip import subprocess
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
def log_subprocess_output(process):
for line in iter(process.stdout.readline, b''):
logger.info('%r', line)
for line in iter(process.stderr.readline, b''):
logger.error('%r', line)
class Senpy(object): class Senpy(object):
""" Default Senpy extension for Flask """ """ Default Senpy extension for Flask """
@@ -77,74 +85,105 @@ class Senpy(object):
else: else:
logger.debug("Not a folder: %s", folder) logger.debug("Not a folder: %s", folder)
def _find_plugin(self, params): def _find_plugins(self, params):
api_params = parse_params(params, spec=API_PARAMS) if not self.analysis_plugins:
algo = None
if "algorithm" in api_params and api_params["algorithm"]:
algo = api_params["algorithm"]
elif self.plugins:
algo = self.default_plugin and self.default_plugin.name
if not algo:
raise Error( raise Error(
status=404, status=404,
message=("No plugins found." message=("No plugins found."
" Please install one.").format(algo)) " Please install one."))
if algo not in self.plugins: api_params = parse_params(params, spec=API_PARAMS)
logger.debug(("The algorithm '{}' is not valid\n" algos = None
"Valid algorithms: {}").format(algo, if "algorithm" in api_params and api_params["algorithm"]:
self.plugins.keys())) algos = api_params["algorithm"].split(',')
elif self.default_plugin:
algos = [self.default_plugin.name, ]
else:
raise Error( raise Error(
status=404, status=404,
message="The algorithm '{}' is not valid".format(algo)) message="No default plugin found, and None provided")
if not self.plugins[algo].is_activated: plugins = list()
logger.debug("Plugin not activated: {}".format(algo)) for algo in algos:
raise Error( if algo not in self.plugins:
status=400, logger.debug(("The algorithm '{}' is not valid\n"
message=("The algorithm '{}'" "Valid algorithms: {}").format(algo,
" is not activated yet").format(algo)) self.plugins.keys()))
return self.plugins[algo] raise Error(
status=404,
message="The algorithm '{}' is not valid".format(algo))
def _get_params(self, params, plugin): if not self.plugins[algo].is_activated:
logger.debug("Plugin not activated: {}".format(algo))
raise Error(
status=400,
message=("The algorithm '{}'"
" is not activated yet").format(algo))
plugins.append(self.plugins[algo])
return plugins
def _get_params(self, params, plugin=None):
nif_params = parse_params(params, spec=NIF_PARAMS) nif_params = parse_params(params, spec=NIF_PARAMS)
extra_params = plugin.get('extra_params', {}) if plugin:
specific_params = parse_params(params, spec=extra_params) extra_params = plugin.get('extra_params', {})
nif_params.update(specific_params) specific_params = parse_params(params, spec=extra_params)
nif_params.update(specific_params)
return nif_params return nif_params
def _get_entries(self, params): def _get_entries(self, params):
entry = None
if params['informat'] == 'text': if params['informat'] == 'text':
results = Results()
entry = Entry(text=params['input']) entry = Entry(text=params['input'])
results.entries.append(entry)
elif params['informat'] == 'json-ld':
results = from_string(params['input'], cls=Results)
else: else:
raise NotImplemented('Only text input format implemented') raise NotImplemented('Informat {} is not implemented'.format(params['informat']))
yield entry return results
def _process_entries(self, entries, plugins, nif_params):
if not plugins:
for i in entries:
yield i
return
plugin = plugins[0]
specific_params = self._get_params(nif_params, plugin)
results = plugin.analyse_entries(entries, specific_params)
for i in self._process_entries(results, plugins[1:], nif_params):
yield i
def _process_response(self, resp, plugins, nif_params):
entries = resp.entries
resp.entries = []
for plug in plugins:
resp.analysis.append(plug.id)
for i in self._process_entries(entries, plugins, nif_params):
resp.entries.append(i)
return resp
def analyse(self, **api_params): def analyse(self, **api_params):
"""
Main method that analyses a request, either from CLI or HTTP.
It uses a dictionary of parameters, provided by the user.
"""
logger.debug("analysing with params: {}".format(api_params)) logger.debug("analysing with params: {}".format(api_params))
plugin = self._find_plugin(api_params) plugins = self._find_plugins(api_params)
nif_params = self._get_params(api_params, plugin) nif_params = self._get_params(api_params)
resp = Results() resp = self._get_entries(nif_params)
if 'with_parameters' in api_params: if 'with_parameters' in api_params:
resp.parameters = nif_params resp.parameters = nif_params
try: try:
entries = [] resp = self._process_response(resp, plugins, nif_params)
for i in self._get_entries(nif_params): self.convert_emotions(resp, plugins, nif_params)
entries += list(plugin.analyse_entry(i, nif_params))
resp.entries = entries
self.convert_emotions(resp, plugin, nif_params)
resp.analysis.append(plugin.id)
logger.debug("Returning analysis result: {}".format(resp)) logger.debug("Returning analysis result: {}".format(resp))
except Error as ex: except (Error, Exception) as ex:
if not isinstance(ex, Error):
ex = Error(message=str(ex), status=500)
logger.exception('Error returning analysis result') logger.exception('Error returning analysis result')
resp = ex raise ex
except Exception as ex:
logger.exception('Error returning analysis result')
resp = Error(message=str(ex), status=500)
return resp return resp
def _conversion_candidates(self, fromModel, toModel): def _conversion_candidates(self, fromModel, toModel):
candidates = self.filter_plugins(**{'@type': 'emotionConversionPlugin'}) candidates = self.filter_plugins(plugin_type='emotionConversionPlugin')
for name, candidate in candidates.items(): for name, candidate in candidates.items():
for pair in candidate.onyx__doesConversion: for pair in candidate.onyx__doesConversion:
logging.debug(pair) logging.debug(pair)
@@ -154,30 +193,32 @@ class Senpy(object):
# logging.debug('Found candidate: {}'.format(candidate)) # logging.debug('Found candidate: {}'.format(candidate))
yield candidate yield candidate
def convert_emotions(self, resp, plugin, params): def convert_emotions(self, resp, plugins, params):
""" """
Conversion of all emotions in a response. Conversion of all emotions in a response **in place**.
In addition to converting from one model to another, it has In addition to converting from one model to another, it has
to include the conversion plugin to the analysis list. to include the conversion plugin to the analysis list.
Needless to say, this is far from an elegant solution, but it works. Needless to say, this is far from an elegant solution, but it works.
@todo refactor and clean up @todo refactor and clean up
""" """
fromModel = plugin.get('onyx:usesEmotionModel', None)
toModel = params.get('emotionModel', None) toModel = params.get('emotionModel', None)
output = params.get('conversion', None)
logger.debug('Asked for model: {}'.format(toModel))
logger.debug('Analysis plugin uses model: {}'.format(fromModel))
if not toModel: if not toModel:
return return
try:
candidate = next(self._conversion_candidates(fromModel, toModel)) logger.debug('Asked for model: {}'.format(toModel))
except StopIteration: output = params.get('conversion', None)
e = Error(('No conversion plugin found for: ' candidates = {}
'{} -> {}'.format(fromModel, toModel))) for plugin in plugins:
e.original_response = resp try:
e.parameters = params fromModel = plugin.get('onyx:usesEmotionModel', None)
raise e candidates[plugin.id] = next(self._conversion_candidates(fromModel, toModel))
logger.debug('Analysis plugin {} uses model: {}'.format(plugin.id, fromModel))
except StopIteration:
e = Error(('No conversion plugin found for: '
'{} -> {}'.format(fromModel, toModel)))
e.original_response = resp
e.parameters = params
raise e
newentries = [] newentries = []
for i in resp.entries: for i in resp.entries:
if output == "full": if output == "full":
@@ -185,6 +226,9 @@ class Senpy(object):
else: else:
newemotions = [] newemotions = []
for j in i.emotions: for j in i.emotions:
plugname = j['prov:wasGeneratedBy']
candidate = candidates[plugname]
resp.analysis.append(candidate.id)
for k in candidate.convert(j, fromModel, toModel, params): for k in candidate.convert(j, fromModel, toModel, params):
k.prov__wasGeneratedBy = candidate.id k.prov__wasGeneratedBy = candidate.id
if output == 'nested': if output == 'nested':
@@ -193,13 +237,14 @@ class Senpy(object):
i.emotions = newemotions i.emotions = newemotions
newentries.append(i) newentries.append(i)
resp.entries = newentries resp.entries = newentries
resp.analysis.append(candidate.id) resp.analysis = list(set(resp.analysis))
@property @property
def default_plugin(self): def default_plugin(self):
candidate = self._default candidate = self._default
if not candidate: if not candidate:
candidates = self.filter_plugins(is_activated=True) candidates = self.filter_plugins(plugin_type='analysisPlugin',
is_activated=True)
if len(candidates) > 0: if len(candidates) > 0:
candidate = list(candidates.values())[0] candidate = list(candidates.values())[0]
logger.debug("Default: {}".format(candidate)) logger.debug("Default: {}".format(candidate))
@@ -258,6 +303,7 @@ class Senpy(object):
else: else:
th = Thread(target=act) th = Thread(target=act)
th.start() th.start()
return th
def deactivate_plugin(self, plugin_name, sync=False): def deactivate_plugin(self, plugin_name, sync=False):
try: try:
@@ -282,6 +328,7 @@ class Senpy(object):
else: else:
th = Thread(target=deact) th = Thread(target=deact)
th.start() th.start()
return th
@classmethod @classmethod
def validate_info(cls, info): def validate_info(cls, info):
@@ -295,13 +342,19 @@ class Senpy(object):
def _install_deps(cls, info=None): def _install_deps(cls, info=None):
requirements = info.get('requirements', []) requirements = info.get('requirements', [])
if requirements: if requirements:
pip_args = [] pip_args = ['pip']
pip_args.append('install') pip_args.append('install')
pip_args.append('--use-wheel') pip_args.append('--use-wheel')
for req in requirements: for req in requirements:
pip_args.append(req) pip_args.append(req)
logger.info('Installing requirements: ' + str(requirements)) logger.info('Installing requirements: ' + str(requirements))
pip.main(pip_args) process = subprocess.Popen(pip_args,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
log_subprocess_output(process)
exitcode = process.wait()
if exitcode != 0:
raise Error("Dependencies not properly installed")
@classmethod @classmethod
def _load_module(cls, name, root): def _load_module(cls, name, root):
@@ -366,23 +419,9 @@ class Senpy(object):
return self._plugin_list return self._plugin_list
def filter_plugins(self, **kwargs): def filter_plugins(self, **kwargs):
""" Filter plugins by different criteria """ return plugins.pfilter(self.plugins, **kwargs)
def matches(plug): @property
res = all(getattr(plug, k, None) == v for (k, v) in kwargs.items()) def analysis_plugins(self):
logger.debug( """ Return only the analysis plugins """
"matching {} with {}: {}".format(plug.name, kwargs, res)) return self.filter_plugins(plugin_type='analysisPlugin')
return res
if not kwargs:
return self.plugins
else:
return {n: p for n, p in self.plugins.items() if matches(p)}
def sentiment_plugins(self):
""" Return only the sentiment plugins """
return {
p: plugin
for p, plugin in self.plugins.items()
if isinstance(plugin, SentimentPlugin)
}

View File

@@ -214,6 +214,7 @@ class BaseModel(SenpyMixin, dict):
temp['@type'] = getattr(self, '@type') temp['@type'] = getattr(self, '@type')
except AttributeError: except AttributeError:
logger.warn('Creating an instance of an unknown model') logger.warn('Creating an instance of an unknown model')
super(BaseModel, self).__init__(temp) super(BaseModel, self).__init__(temp)
def _get_key(self, key): def _get_key(self, key):
@@ -236,7 +237,10 @@ class BaseModel(SenpyMixin, dict):
self.__setitem__(self._get_key(key), value) self.__setitem__(self._get_key(key), value)
def __delattr__(self, key): def __delattr__(self, key):
self.__delitem__(self._get_key(key)) try:
object.__delattr__(self, key)
except AttributeError:
self.__delitem__(self._get_key(key))
def _plain_dict(self): def _plain_dict(self):
d = {k: v for (k, v) in self.items() if k[0] != "_"} d = {k: v for (k, v) in self.items() if k[0] != "_"}
@@ -252,13 +256,32 @@ def register(rsubclass, rtype=None):
_subtypes[rtype or rsubclass.__name__] = rsubclass _subtypes[rtype or rsubclass.__name__] = rsubclass
def from_dict(indict): def from_dict(indict, cls=None):
target = indict.get('@type', None) if not cls:
if target and target in _subtypes: target = indict.get('@type', None)
cls = _subtypes[target] try:
else: if target and target in _subtypes:
cls = BaseModel cls = _subtypes[target]
return cls(**indict) else:
cls = BaseModel
except Exception:
cls = BaseModel
outdict = dict()
for k, v in indict.items():
if k == '@context':
pass
elif isinstance(v, dict):
v = from_dict(indict[k])
elif isinstance(v, list):
for ix, v2 in enumerate(v):
if isinstance(v2, dict):
v[ix] = from_dict(v2)
outdict[k] = v
return cls(**outdict)
def from_string(string, **kwargs):
return from_dict(json.loads(string), **kwargs)
def from_json(injson): def from_json(injson):
@@ -308,7 +331,7 @@ for i in [
_ErrorModel = from_schema('error') _ErrorModel = from_schema('error')
class Error(SenpyMixin, BaseException): class Error(SenpyMixin, Exception):
def __init__(self, message, *args, **kwargs): def __init__(self, message, *args, **kwargs):
super(Error, self).__init__(self, message, message) super(Error, self).__init__(self, message, message)
self._error = _ErrorModel(message=message, *args, **kwargs) self._error = _ErrorModel(message=message, *args, **kwargs)
@@ -337,5 +360,8 @@ class Error(SenpyMixin, BaseException):
def __delattr__(self, key): def __delattr__(self, key):
delattr(self._error, key) delattr(self._error, key)
def __str__(self):
return str(self.to_JSON(with_context=False))
register(Error, 'error') register(Error, 'error')

View File

@@ -9,11 +9,12 @@ import logging
import tempfile import tempfile
import copy import copy
from .. import models from .. import models
from ..api import API_PARAMS
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
class SenpyPlugin(models.Plugin): class Plugin(models.Plugin):
def __init__(self, info=None): def __init__(self, info=None):
""" """
Provides a canonical name for plugins and serves as base for other Provides a canonical name for plugins and serves as base for other
@@ -24,12 +25,24 @@ class SenpyPlugin(models.Plugin):
"information for the plugin.")) "information for the plugin."))
logger.debug("Initialising {}".format(info)) logger.debug("Initialising {}".format(info))
id = 'plugins/{}_{}'.format(info['name'], info['version']) id = 'plugins/{}_{}'.format(info['name'], info['version'])
super(SenpyPlugin, self).__init__(id=id, **info) super(Plugin, self).__init__(id=id, **info)
self.is_activated = False self.is_activated = False
def get_folder(self): def get_folder(self):
return os.path.dirname(inspect.getfile(self.__class__)) return os.path.dirname(inspect.getfile(self.__class__))
def activate(self):
pass
def deactivate(self):
pass
SenpyPlugin = Plugin
class AnalysisPlugin(Plugin):
def analyse(self, *args, **kwargs): def analyse(self, *args, **kwargs):
raise NotImplemented( raise NotImplemented(
'Your method should implement either analyse or analyse_entry') 'Your method should implement either analyse or analyse_entry')
@@ -48,30 +61,33 @@ class SenpyPlugin(models.Plugin):
for i in results.entries: for i in results.entries:
yield i yield i
def activate(self): def analyse_entries(self, entries, parameters):
pass for entry in entries:
logger.debug('Analysing entry with plugin {}: {}'.format(self, entry))
def deactivate(self): for result in self.analyse_entry(entry, parameters):
pass yield result
class SentimentPlugin(models.SentimentPlugin, SenpyPlugin): class ConversionPlugin(Plugin):
pass
class SentimentPlugin(models.SentimentPlugin, AnalysisPlugin):
def __init__(self, info, *args, **kwargs): def __init__(self, info, *args, **kwargs):
super(SentimentPlugin, self).__init__(info, *args, **kwargs) super(SentimentPlugin, self).__init__(info, *args, **kwargs)
self.minPolarityValue = float(info.get("minPolarityValue", 0)) self.minPolarityValue = float(info.get("minPolarityValue", 0))
self.maxPolarityValue = float(info.get("maxPolarityValue", 1)) self.maxPolarityValue = float(info.get("maxPolarityValue", 1))
class EmotionPlugin(models.EmotionPlugin, SenpyPlugin): class EmotionPlugin(models.EmotionPlugin, AnalysisPlugin):
def __init__(self, info, *args, **kwargs): def __init__(self, info, *args, **kwargs):
super(EmotionPlugin, self).__init__(info, *args, **kwargs) super(EmotionPlugin, self).__init__(info, *args, **kwargs)
self.minEmotionValue = float(info.get("minEmotionValue", -1)) self.minEmotionValue = float(info.get("minEmotionValue", -1))
self.maxEmotionValue = float(info.get("maxEmotionValue", 1)) self.maxEmotionValue = float(info.get("maxEmotionValue", 1))
class EmotionConversionPlugin(models.EmotionConversionPlugin, SenpyPlugin): class EmotionConversionPlugin(models.EmotionConversionPlugin, ConversionPlugin):
def __init__(self, info, *args, **kwargs): pass
super(EmotionConversionPlugin, self).__init__(info, *args, **kwargs)
class ShelfMixin(object): class ShelfMixin(object):
@@ -80,7 +96,12 @@ class ShelfMixin(object):
if not hasattr(self, '_sh') or self._sh is None: if not hasattr(self, '_sh') or self._sh is None:
self.__dict__['_sh'] = {} self.__dict__['_sh'] = {}
if os.path.isfile(self.shelf_file): if os.path.isfile(self.shelf_file):
self.__dict__['_sh'] = pickle.load(open(self.shelf_file, 'rb')) try:
self.__dict__['_sh'] = pickle.load(open(self.shelf_file, 'rb'))
except (IndexError, EOFError, pickle.UnpicklingError):
logger.warning('{} has a corrupted shelf file!'.format(self.id))
if not self.get('force_shelf', False):
raise
return self._sh return self._sh
@sh.deleter @sh.deleter
@@ -102,3 +123,40 @@ class ShelfMixin(object):
if hasattr(self, '_sh') and self._sh is not None: if hasattr(self, '_sh') and self._sh is not None:
with open(self.shelf_file, 'wb') as f: with open(self.shelf_file, 'wb') as f:
pickle.dump(self._sh, f) pickle.dump(self._sh, f)
default_plugin_type = API_PARAMS['plugin_type']['default']
def pfilter(plugins, **kwargs):
""" Filter plugins by different criteria """
if isinstance(plugins, models.Plugins):
plugins = plugins.plugins
elif isinstance(plugins, dict):
plugins = plugins.values()
ptype = kwargs.pop('plugin_type', default_plugin_type)
logger.debug('#' * 100)
logger.debug('ptype {}'.format(ptype))
if ptype:
try:
ptype = ptype[0].upper() + ptype[1:]
pclass = globals()[ptype]
logger.debug('Class: {}'.format(pclass))
candidates = filter(lambda x: isinstance(x, pclass),
plugins)
except KeyError:
raise models.Error('{} is not a valid type'.format(ptype))
else:
candidates = plugins
logger.debug(candidates)
def matches(plug):
res = all(getattr(plug, k, None) == v for (k, v) in kwargs.items())
logger.debug(
"matching {} with {}: {}".format(plug.name, kwargs, res))
return res
if kwargs:
candidates = filter(matches, candidates)
return {p.name: p for p in candidates}

View File

@@ -1,52 +0,0 @@
from senpy.plugins import EmotionConversionPlugin
from senpy.models import EmotionSet, Emotion, Error
import logging
logger = logging.getLogger(__name__)
class CentroidConversion(EmotionConversionPlugin):
def _forward_conversion(self, original):
"""Sum the VAD value of all categories found."""
res = Emotion()
for e in original.onyx__hasEmotion:
category = e.onyx__hasEmotionCategory
if category in self.centroids:
for dim, value in self.centroids[category].iteritems():
try:
res[dim] += value
except Exception:
res[dim] = value
return res
def _backwards_conversion(self, original):
"""Find the closest category"""
dimensions = list(self.centroids.values())[0]
def distance(e1, e2):
return sum((e1[k] - e2.get(self.aliases[k], 0)) for k in dimensions)
emotion = ''
mindistance = 10000000000000000000000.0
for state in self.centroids:
d = distance(self.centroids[state], original)
if d < mindistance:
mindistance = d
emotion = state
result = Emotion(onyx__hasEmotionCategory=emotion)
return result
def convert(self, emotionSet, fromModel, toModel, params):
cf, ct = self.centroids_direction
logger.debug('{}\n{}\n{}\n{}'.format(emotionSet, fromModel, toModel, params))
e = EmotionSet()
if fromModel == cf:
e.onyx__hasEmotion.append(self._forward_conversion(emotionSet))
elif fromModel == ct:
for i in emotionSet.onyx__hasEmotion:
e.onyx__hasEmotion.append(self._backwards_conversion(i))
else:
raise Error('EMOTION MODEL NOT KNOWN')
yield e

View File

@@ -0,0 +1,102 @@
from senpy.plugins import EmotionConversionPlugin
from senpy.models import EmotionSet, Emotion, Error
import logging
logger = logging.getLogger(__name__)
class CentroidConversion(EmotionConversionPlugin):
def __init__(self, info):
if 'centroids' not in info:
raise Error('Centroid conversion plugins should provide '
'the centroids in their senpy file')
if 'onyx:doesConversion' not in info:
if 'centroids_direction' not in info:
raise Error('Please, provide centroids direction')
cf, ct = info['centroids_direction']
info['onyx:doesConversion'] = [{
'onyx:conversionFrom': cf,
'onyx:conversionTo': ct
}, {
'onyx:conversionFrom': ct,
'onyx:conversionTo': cf
}]
if 'aliases' in info:
aliases = info['aliases']
ncentroids = {}
for k1, v1 in info['centroids'].items():
nv1 = {}
for k2, v2 in v1.items():
nv1[aliases.get(k2, k2)] = v2
ncentroids[aliases.get(k1, k1)] = nv1
info['centroids'] = ncentroids
super(CentroidConversion, self).__init__(info)
self.dimensions = set()
for c in self.centroids.values():
self.dimensions.update(c.keys())
self.neutralPoints = self.get("neutralPoints", dict())
if not self.neutralPoints:
for i in self.dimensions:
self.neutralPoints[i] = self.get("neutralValue", 0)
def _forward_conversion(self, original):
"""Sum the VAD value of all categories found weighted by intensity.
Intensities are scaled by onyx:maxIntensityValue if it is present, else maxIntensityValue
is assumed to be one. Emotion entries that do not have onxy:hasEmotionIntensity specified
are assumed to have maxIntensityValue. Emotion entries that do not have
onyx:hasEmotionCategory specified are ignored."""
res = Emotion()
maxIntensity = float(original.get("onyx:maxIntensityValue", 1))
for e in original.onyx__hasEmotion:
category = e.get("onyx:hasEmotionCategory", None)
if not category:
continue
intensity = e.get("onyx:hasEmotionIntensity", maxIntensity) / maxIntensity
if not intensity:
continue
centroid = self.centroids.get(category, None)
if centroid:
for dim, value in centroid.items():
neutral = self.neutralPoints[dim]
if dim not in res:
res[dim] = 0
res[dim] += (value - neutral) * intensity + neutral
return res
def _backwards_conversion(self, original):
"""Find the closest category"""
centroids = self.centroids
neutralPoints = self.neutralPoints
dimensions = self.dimensions
def distance_k(centroid, original, k):
# k component of the distance between the value and a given centroid
return (centroid.get(k, neutralPoints[k]) - original.get(k, neutralPoints[k]))**2
def distance(centroid):
return sum(distance_k(centroid, original, k) for k in dimensions)
emotion = min(centroids, key=lambda x: distance(centroids[x]))
result = Emotion(onyx__hasEmotionCategory=emotion)
result.onyx__algorithmConfidence = distance(centroids[emotion])
return result
def convert(self, emotionSet, fromModel, toModel, params):
cf, ct = self.centroids_direction
logger.debug(
'{}\n{}\n{}\n{}'.format(emotionSet, fromModel, toModel, params))
e = EmotionSet()
if fromModel == cf and toModel == ct:
e.onyx__hasEmotion.append(self._forward_conversion(emotionSet))
elif fromModel == ct and toModel == cf:
for i in emotionSet.onyx__hasEmotion:
e.onyx__hasEmotion.append(self._backwards_conversion(i))
else:
raise Error('EMOTION MODEL NOT KNOWN')
yield e

View File

@@ -0,0 +1,39 @@
---
name: Ekman2FSRE
module: senpy.plugins.conversion.emotion.centroids
description: Plugin to convert emotion sets from Ekman to VAD
version: 0.1
# No need to specify onyx:doesConversion because centroids.py adds it automatically from centroids_direction
centroids:
anger:
A: 6.95
D: 5.1
V: 2.7
disgust:
A: 5.3
D: 8.05
V: 2.7
fear:
A: 6.5
D: 3.6
V: 3.2
happiness:
A: 7.22
D: 6.28
V: 8.6
sadness:
A: 5.21
D: 2.82
V: 2.21
centroids_direction:
- emoml:big6
- emoml:fsre-dimensions
aliases: # These are aliases for any key in the centroid, to avoid repeating a long name several times
A: emoml:arousal
V: emoml:valence
D: emoml:dominance
anger: emoml:big6anger
disgust: emoml:big6disgust
fear: emoml:big6fear
happiness: emoml:big6happiness
sadness: emoml:big6sadness

View File

@@ -1,38 +1,44 @@
--- ---
name: Ekman2VAD name: Ekman2PAD
module: senpy.plugins.conversion.centroids module: senpy.plugins.conversion.emotion.centroids
description: Plugin to convert emotion sets from Ekman to VAD description: Plugin to convert emotion sets from Ekman to VAD
version: 0.1 version: 0.1
onyx:doesConversion: # No need to specify onyx:doesConversion because centroids.py adds it automatically from centroids_direction
- onyx:conversionFrom: emoml:big6 origin:
onyx:conversionTo: emoml:fsre-dimensions # Point in VAD space with no emotion (aka Neutral)
- onyx:conversionFrom: emoml:fsre-dimensions A: 5.0
onyx:conversionTo: emoml:big6 D: 5.0
V: 5.0
centroids: centroids:
emoml:big6anger: anger:
A: 6.95 A: 6.95
D: 5.1 D: 5.1
V: 2.7 V: 2.7
emoml:big6disgust: disgust:
A: 5.3 A: 5.3
D: 8.05 D: 8.05
V: 2.7 V: 2.7
emoml:big6fear: fear:
A: 6.5 A: 6.5
D: 3.6 D: 3.6
V: 3.2 V: 3.2
emoml:big6happiness: happiness:
A: 7.22 A: 7.22
D: 6.28 D: 6.28
V: 8.6 V: 8.6
emoml:big6sadness: sadness:
A: 5.21 A: 5.21
D: 2.82 D: 2.82
V: 2.21 V: 2.21
centroids_direction: centroids_direction:
- emoml:big6 - emoml:big6
- emoml:fsre-dimensions - emoml:pad
aliases: aliases: # These are aliases for any key in the centroid, to avoid repeating a long name several times
A: emoml:arousal A: emoml:arousal
V: emoml:valence V: emoml:valence
D: emoml:dominance D: emoml:dominance
anger: emoml:big6anger
disgust: emoml:big6disgust
fear: emoml:big6fear
happiness: emoml:big6happiness
sadness: emoml:big6sadness

View File

@@ -0,0 +1,30 @@
from senpy.plugins import AnalysisPlugin
from senpy.models import Entry
from nltk.tokenize.punkt import PunktSentenceTokenizer
from nltk.tokenize.simple import LineTokenizer
import nltk
class SplitPlugin(AnalysisPlugin):
def activate(self):
nltk.download('punkt')
def analyse_entry(self, entry, params):
chunker_type = params.get("delimiter", "sentence")
original_id = entry.id
original_text = entry.get("text", None)
if chunker_type == "sentence":
tokenizer = PunktSentenceTokenizer()
chars = tokenizer.span_tokenize(original_text)
for i, sentence in enumerate(tokenizer.tokenize(original_text)):
e = Entry()
e.text = sentence
e.id = original_id + "#char={},{}".format(chars[i][0], chars[i][1])
yield e
if chunker_type == "paragraph":
tokenizer = LineTokenizer()
chars = tokenizer.span_tokenize(original_text)
for i, paragraph in enumerate(tokenizer.tokenize(original_text)):
e = Entry()
e.text = paragraph
chars = [char for char in chars]
e.id = original_id + "#char={},{}".format(chars[i][0], chars[i][1])
yield e

View File

@@ -0,0 +1,18 @@
---
name: split
module: split
description: A sample plugin that chunks input text
author: "@militarpancho"
version: '0.1'
url: "https://github.com/gsi-upm/senpy"
requirements: {nltk}
extra_params:
delimiter:
aliases:
- type
- t
required: false
default: sentence
options:
- sentence
- paragraph

View File

@@ -37,6 +37,12 @@
"@type": "@id", "@type": "@id",
"@container": "@set" "@container": "@set"
}, },
"options": {
"@container": "@set"
},
"plugins": {
"@container": "@set"
},
"prov:wasGeneratedBy": { "prov:wasGeneratedBy": {
"@type": "@id" "@type": "@id"
}, },

View File

@@ -6,11 +6,10 @@
"properties": { "properties": {
"plugins": { "plugins": {
"type": "array", "type": "array",
"default": [],
"items": { "items": {
"$ref": "plugin.json" "$ref": "plugin.json"
} }
},
"@type": {
} }
} }
} }

View File

@@ -18,10 +18,16 @@
"type": "string" "type": "string"
}, },
"analysis": { "analysis": {
"type": "array",
"default": [], "default": [],
"type": "array",
"items": { "items": {
"$ref": "analysis.json" "anyOf": [
{
"$ref": "analysis.json"
},{
"type": "string"
}
]
} }
}, },
"entries": { "entries": {

View File

@@ -47,7 +47,7 @@
This website is the senpy Playground, which allows you to test the instance of senpy in this server. It provides a user-friendly interface to the functions exposed by the senpy API. This website is the senpy Playground, which allows you to test the instance of senpy in this server. It provides a user-friendly interface to the functions exposed by the senpy API.
</p> </p>
<p> <p>
Once you get comfortable with the parameters and results, you are encouraged to issue your own requests to the API endpoint, which should be <a href="/api">here</a>. Once you get comfortable with the parameters and results, you are encouraged to issue your own requests to the API endpoint. You can find examples of API URL's when you try out a plugin with the "Analyse!" button on the "Test it" tab.
</p> </p>
<p> <p>
These are some of the things you can do with the API: These are some of the things you can do with the API:

View File

@@ -8,8 +8,12 @@ DEFAULT_FILE = os.path.join(ROOT, 'VERSION')
def read_version(versionfile=DEFAULT_FILE): def read_version(versionfile=DEFAULT_FILE):
with open(versionfile) as f: try:
return f.read().strip() with open(versionfile) as f:
return f.read().strip()
except IOError: # pragma: no cover
logger.error('Running an unknown version of senpy. Be careful!.')
return '0.0'
__version__ = read_version() __version__ = read_version()

View File

@@ -11,4 +11,7 @@ max-line-length = 100
[bdist_wheel] [bdist_wheel]
universal=1 universal=1
[tool:pytest] [tool:pytest]
addopts = --cov=senpy --cov-report term-missing addopts = --cov=senpy --cov-report term-missing
[coverage:report]
omit = senpy/__main__.py

View File

@@ -0,0 +1,23 @@
from senpy.plugins import AnalysisPlugin
import multiprocessing
def _train(process_number):
return process_number
class AsyncPlugin(AnalysisPlugin):
def _do_async(self, num_processes):
pool = multiprocessing.Pool(processes=num_processes)
values = pool.map(_train, range(num_processes))
return values
def activate(self):
self.value = self._do_async(4)
def analyse_entry(self, entry, params):
values = self._do_async(2)
entry.async_values = values
yield entry

View File

@@ -0,0 +1,8 @@
---
name: Async
module: asyncplugin
description: I am async
author: "@balkian"
version: '0.1'
async: true
extra_params: {}

View File

@@ -4,4 +4,5 @@ from senpy.plugins import SentimentPlugin
class DummyPlugin(SentimentPlugin): class DummyPlugin(SentimentPlugin):
def analyse_entry(self, entry, params): def analyse_entry(self, entry, params):
entry.text = entry.text[::-1] entry.text = entry.text[::-1]
entry.reversed = entry.get('reversed', 0) + 1
yield entry yield entry

View File

@@ -1,8 +1,8 @@
from senpy.plugins import SenpyPlugin from senpy.plugins import AnalysisPlugin
from time import sleep from time import sleep
class SleepPlugin(SenpyPlugin): class SleepPlugin(AnalysisPlugin):
def activate(self, *args, **kwargs): def activate(self, *args, **kwargs):
sleep(self.timeout) sleep(self.timeout)

View File

@@ -19,6 +19,7 @@ def parse_resp(resp):
class BlueprintsTest(TestCase): class BlueprintsTest(TestCase):
def setUp(self): def setUp(self):
self.app = Flask("test_extensions") self.app = Flask("test_extensions")
self.app.debug = False
self.client = self.app.test_client() self.client = self.app.test_client()
self.senpy = Senpy() self.senpy = Senpy()
self.senpy.init_app(self.app) self.senpy.init_app(self.app)

View File

@@ -4,18 +4,21 @@ try:
except ImportError: except ImportError:
from mock import patch from mock import patch
import json
from senpy.client import Client from senpy.client import Client
from senpy.models import Results, Error from senpy.models import Results, Plugins, Error
from senpy.plugins import AnalysisPlugin, default_plugin_type
class Call(dict): class Call(dict):
def __init__(self, obj): def __init__(self, obj):
self.obj = obj.jsonld() self.obj = obj.serialize()
self.status_code = 200 self.status_code = 200
self.content = self.json() self.content = self.json()
def json(self): def json(self):
return self.obj return json.loads(self.obj)
class ModelsTest(TestCase): class ModelsTest(TestCase):
@@ -44,3 +47,19 @@ class ModelsTest(TestCase):
method='GET', method='GET',
params={'input': 'hello', params={'input': 'hello',
'algorithm': 'NONEXISTENT'}) 'algorithm': 'NONEXISTENT'})
def test_plugins(self):
endpoint = 'http://dummy/'
client = Client(endpoint)
plugins = Plugins()
p1 = AnalysisPlugin({'name': 'AnalysisP1', 'version': 0, 'description': 'No'})
plugins.plugins = [p1, ]
success = Call(plugins)
with patch('requests.request', return_value=success) as patched:
response = client.plugins()
assert isinstance(response, dict)
assert len(response) == 1
assert "AnalysisP1" in response
patched.assert_called_with(
url=endpoint + '/plugins', method='GET',
params={'plugin_type': default_plugin_type})

View File

@@ -10,7 +10,7 @@ except ImportError:
from functools import partial from functools import partial
from senpy.extensions import Senpy from senpy.extensions import Senpy
from senpy.models import Error, Results, Entry, EmotionSet, Emotion from senpy.models import Error, Results, Entry, EmotionSet, Emotion, Plugin
from flask import Flask from flask import Flask
from unittest import TestCase from unittest import TestCase
@@ -61,6 +61,19 @@ class ExtensionsTest(TestCase):
assert len(self.senpy.plugins) >= 3 assert len(self.senpy.plugins) >= 3
assert self.senpy.plugins["Sleep"].is_activated assert self.senpy.plugins["Sleep"].is_activated
def test_installing_nonexistent(self):
""" Fail if the dependencies cannot be met """
info = {
'name': 'TestPipFail',
'module': 'dummy',
'description': None,
'requirements': ['IAmMakingThisPackageNameUpToFail'],
'version': 0
}
root = os.path.join(self.dir, 'plugins', 'dummy_plugin')
with self.assertRaises(Error):
name, module = self.senpy._load_plugin_from_info(info, root=root)
def test_disabling(self): def test_disabling(self):
""" Disabling a plugin """ """ Disabling a plugin """
self.senpy.deactivate_all(sync=True) self.senpy.deactivate_all(sync=True)
@@ -96,19 +109,49 @@ class ExtensionsTest(TestCase):
assert r2.analysis[0] == "plugins/Dummy_0.1" assert r2.analysis[0] == "plugins/Dummy_0.1"
assert r1.entries[0].text == 'input' assert r1.entries[0].text == 'input'
def test_analyse_jsonld(self):
""" Using a plugin with JSON-LD input"""
js_input = '''{
"@id": "prueba",
"@type": "results",
"entries": [
{"@id": "entry1",
"text": "tupni",
"@type": "entry"
}
]
}'''
r1 = self.senpy.analyse(algorithm="Dummy",
input=js_input,
informat="json-ld",
output="tuptuo")
r2 = self.senpy.analyse(input="tupni", output="tuptuo")
assert r1.analysis[0] == "plugins/Dummy_0.1"
assert r2.analysis[0] == "plugins/Dummy_0.1"
assert r1.entries[0].text == 'input'
def test_analyse_error(self): def test_analyse_error(self):
mm = mock.MagicMock() mm = mock.MagicMock()
mm.analyse_entry.side_effect = Error('error on analysis', status=900) mm.id = 'magic_mock'
mm.analyse_entries.side_effect = Error('error on analysis', status=500)
self.senpy.plugins['MOCK'] = mm self.senpy.plugins['MOCK'] = mm
resp = self.senpy.analyse(input='nothing', algorithm='MOCK') try:
assert resp['message'] == 'error on analysis' self.senpy.analyse(input='nothing', algorithm='MOCK')
assert resp['status'] == 900 assert False
except Error as ex:
assert ex['message'] == 'error on analysis'
assert ex['status'] == 500
mm.analyse.side_effect = Exception('generic exception on analysis') mm.analyse.side_effect = Exception('generic exception on analysis')
mm.analyse_entry.side_effect = Exception( mm.analyse_entries.side_effect = Exception(
'generic exception on analysis') 'generic exception on analysis')
resp = self.senpy.analyse(input='nothing', algorithm='MOCK')
assert resp['message'] == 'generic exception on analysis' try:
assert resp['status'] == 500 self.senpy.analyse(input='nothing', algorithm='MOCK')
assert False
except Error as ex:
assert ex['message'] == 'generic exception on analysis'
assert ex['status'] == 500
def test_filtering(self): def test_filtering(self):
""" Filtering plugins """ """ Filtering plugins """
@@ -124,12 +167,13 @@ class ExtensionsTest(TestCase):
assert len(senpy.plugins) > 1 assert len(senpy.plugins) > 1
def test_convert_emotions(self): def test_convert_emotions(self):
self.senpy.activate_all() self.senpy.activate_all(sync=True)
plugin = { plugin = Plugin({
'id': 'imaginary', 'id': 'imaginary',
'onyx:usesEmotionModel': 'emoml:fsre-dimensions' 'onyx:usesEmotionModel': 'emoml:fsre-dimensions'
} })
eSet1 = EmotionSet() eSet1 = EmotionSet()
eSet1.prov__wasGeneratedBy = plugin['id']
eSet1['onyx:hasEmotion'].append(Emotion({ eSet1['onyx:hasEmotion'].append(Emotion({
'emoml:arousal': 1, 'emoml:arousal': 1,
'emoml:potency': 0, 'emoml:potency': 0,
@@ -145,19 +189,31 @@ class ExtensionsTest(TestCase):
'conversion': 'full'} 'conversion': 'full'}
r1 = deepcopy(response) r1 = deepcopy(response)
self.senpy.convert_emotions(r1, self.senpy.convert_emotions(r1,
plugin, [plugin, ],
params) params)
assert len(r1.entries[0].emotions) == 2 assert len(r1.entries[0].emotions) == 2
params['conversion'] = 'nested' params['conversion'] = 'nested'
r2 = deepcopy(response) r2 = deepcopy(response)
self.senpy.convert_emotions(r2, self.senpy.convert_emotions(r2,
plugin, [plugin, ],
params) params)
assert len(r2.entries[0].emotions) == 1 assert len(r2.entries[0].emotions) == 1
assert r2.entries[0].emotions[0]['prov:wasDerivedFrom'] == eSet1 assert r2.entries[0].emotions[0]['prov:wasDerivedFrom'] == eSet1
params['conversion'] = 'filtered' params['conversion'] = 'filtered'
r3 = deepcopy(response) r3 = deepcopy(response)
self.senpy.convert_emotions(r3, self.senpy.convert_emotions(r3,
plugin, [plugin, ],
params) params)
assert len(r3.entries[0].emotions) == 1 assert len(r3.entries[0].emotions) == 1
r3.jsonld()
# def test_async_plugin(self):
# """ We should accept multiprocessing plugins with async=False"""
# thread1 = self.senpy.activate_plugin("Async", sync=False)
# thread1.join(timeout=1)
# assert len(self.senpy.plugins['Async'].value) == 4
# resp = self.senpy.analyse(input='nothing', algorithm='Async')
# assert len(resp.entries[0].async_values) == 2
# self.senpy.activate_plugin("Async", sync=True)

View File

@@ -11,8 +11,12 @@ from senpy.models import (Emotion,
Entry, Entry,
Error, Error,
Results, Results,
Sentiment) Sentiment,
from senpy.plugins import SenpyPlugin Plugins,
Plugin,
from_string,
from_dict)
from senpy import plugins
from pprint import pprint from pprint import pprint
@@ -53,8 +57,8 @@ class ModelsTest(TestCase):
assert (received["entries"][0]["nif:isString"] != "Not testing") assert (received["entries"][0]["nif:isString"] != "Not testing")
def test_id(self): def test_id(self):
''' Adding the id after creation should overwrite the automatic ID """ Adding the id after creation should overwrite the automatic ID
''' """
r = Entry() r = Entry()
j = r.jsonld() j = r.jsonld()
assert '@id' in j assert '@id' in j
@@ -94,20 +98,32 @@ class ModelsTest(TestCase):
r.validate() r.validate()
def test_plugins(self): def test_plugins(self):
self.assertRaises(Error, SenpyPlugin) self.assertRaises(Error, plugins.Plugin)
p = SenpyPlugin({"name": "dummy", "version": 0}) p = plugins.Plugin({"name": "dummy",
"version": 0,
"extra_params": {
"none": {
"options": ["es", ],
"required": False,
"default": "0"
}
}})
c = p.jsonld() c = p.jsonld()
assert "info" not in c assert '@type' in c
assert "repo" not in c assert c['@type'] == 'plugin'
assert "extra_params" in c assert 'info' not in c
logging.debug("Framed:") assert 'repo' not in c
assert 'extra_params' in c
logging.debug('Framed:')
logging.debug(c) logging.debug(c)
p.validate() p.validate()
assert 'es' in c['extra_params']['none']['options']
assert isinstance(c['extra_params']['none']['options'], list)
def test_str(self): def test_str(self):
"""The string representation shouldn't include private variables""" """The string representation shouldn't include private variables"""
r = Results() r = Results()
p = SenpyPlugin({"name": "STR test", "version": 0}) p = plugins.Plugin({"name": "STR test", "version": 0})
p._testing = 0 p._testing = 0
s = str(p) s = str(p)
assert "_testing" not in s assert "_testing" not in s
@@ -143,3 +159,40 @@ class ModelsTest(TestCase):
print(t) print(t)
g = rdflib.Graph().parse(data=t, format='turtle') g = rdflib.Graph().parse(data=t, format='turtle')
assert len(g) == len(triples) assert len(g) == len(triples)
def test_plugin_list(self):
"""The plugin list should be of type \"plugins\""""
plugs = Plugins()
c = plugs.jsonld()
assert '@type' in c
assert c['@type'] == 'plugins'
def test_single_plugin(self):
"""A response with a single plugin should still return a list"""
plugs = Plugins()
p = Plugin({'id': str(1),
'version': 0,
'description': 'dummy'})
plugs.plugins.append(p)
assert isinstance(plugs.plugins, list)
js = plugs.jsonld()
assert isinstance(js['plugins'], list)
def test_from_string(self):
results = {
'@type': 'results',
'@id': 'prueba',
'entries': [{
'@id': 'entry1',
'@type': 'entry',
'text': 'TEST'
}]
}
recovered = from_dict(results)
assert isinstance(recovered, Results)
assert isinstance(recovered.entries[0], Entry)
string = json.dumps(results)
recovered = from_string(string)
assert isinstance(recovered, Results)
assert isinstance(recovered.entries[0], Entry)

View File

@@ -6,8 +6,9 @@ import shutil
import tempfile import tempfile
from unittest import TestCase from unittest import TestCase
from senpy.models import Results, Entry from senpy.models import Results, Entry, EmotionSet, Emotion
from senpy.plugins import SentimentPlugin, ShelfMixin from senpy.plugins import SentimentPlugin, ShelfMixin
from senpy.plugins.conversion.emotion.centroids import CentroidConversion
class ShelfDummyPlugin(SentimentPlugin, ShelfMixin): class ShelfDummyPlugin(SentimentPlugin, ShelfMixin):
@@ -83,7 +84,39 @@ class PluginsTest(TestCase):
res2 = a.analyse(input=1) res2 = a.analyse(input=1)
assert res2.entries[0].nif__isString == 2 assert res2.entries[0].nif__isString == 2
def test_two(self): def test_corrupt_shelf(self):
''' Reusing the values of a previous shelf '''
emptyfile = os.path.join(self.shelf_dir, "emptyfile")
invalidfile = os.path.join(self.shelf_dir, "invalid_file")
with open(emptyfile, 'w+b'), open(invalidfile, 'w+b') as inf:
inf.write(b'ohno')
files = {emptyfile: ['empty file', (EOFError, IndexError)],
invalidfile: ['invalid file', (pickle.UnpicklingError, IndexError)]}
for fn in files:
with open(fn, 'rb') as f:
msg, error = files[fn]
a = ShelfDummyPlugin(info={
'name': 'shelve',
'version': 'test',
'shelf_file': f.name
})
assert os.path.isfile(a.shelf_file)
print('Shelf file: %s' % a.shelf_file)
with self.assertRaises(error):
a.sh['a'] = 'fromA'
a.save()
del a._sh
assert os.path.isfile(a.shelf_file)
a.force_shelf = True
a.sh['a'] = 'fromA'
a.save()
b = pickle.load(f)
assert b['a'] == 'fromA'
def test_reuse_shelf(self):
''' Reusing the values of a previous shelf ''' ''' Reusing the values of a previous shelf '''
a = ShelfDummyPlugin(info={ a = ShelfDummyPlugin(info={
'name': 'shelve', 'name': 'shelve',
@@ -120,3 +153,52 @@ class PluginsTest(TestCase):
} }
}) })
assert 'example' in a.extra_params assert 'example' in a.extra_params
def test_conversion_centroids(self):
info = {
"name": "CentroidTest",
"description": "Centroid test",
"version": 0,
"centroids": {
"c1": {"V1": 0.5,
"V2": 0.5},
"c2": {"V1": -0.5,
"V2": 0.5},
"c3": {"V1": -0.5,
"V2": -0.5},
"c4": {"V1": 0.5,
"V2": -0.5}},
"aliases": {
"V1": "X-dimension",
"V2": "Y-dimension"
},
"centroids_direction": ["emoml:big6", "emoml:fsre-dimensions"]
}
c = CentroidConversion(info)
es1 = EmotionSet()
e1 = Emotion()
e1.onyx__hasEmotionCategory = "c1"
es1.onyx__hasEmotion.append(e1)
res = c._forward_conversion(es1)
assert res["X-dimension"] == 0.5
assert res["Y-dimension"] == 0.5
e2 = Emotion()
e2.onyx__hasEmotionCategory = "c2"
es1.onyx__hasEmotion.append(e2)
res = c._forward_conversion(es1)
assert res["X-dimension"] == 0
assert res["Y-dimension"] == 1
e = Emotion()
e["X-dimension"] = -0.2
e["Y-dimension"] = -0.3
res = c._backwards_conversion(e)
assert res["onyx:hasEmotionCategory"] == "c3"
e = Emotion()
e["X-dimension"] = -0.2
e["Y-dimension"] = 0.3
res = c._backwards_conversion(e)
assert res["onyx:hasEmotionCategory"] == "c2"