balkian.com

Sharing dotfiles

2015-04-10T17:47:00+02:00

Today's post is half a quick note, half public shaming. In other words, it is a reminder to be very careful with OAuth tokens and passwords.

As part of moving to emacs, I starting using the incredibly useful gh.el. When you first use it, the extension saves either your password or an OAuth token in your .gitconfig file. This is cool and convenient, unless you happen to be publishing your .gitconfig file in a public repo.

So, how can you still share your gitconfig without sharing your password/token with the rest of the world? Since Git 1.7.0, you can include other files in your gitconfig.

1 2	[include] path = ~/.gitconfig_secret

And now, in your .gitconfig_secret file, you just have to add this:

1
2
3

[github]
    user = balkian
    token = "< Your secret token >"

Zotero

2014-12-09T12:12:12+01:00

Zotero is an Open Source tool that lets you organise your bibliography, syncing it with the cloud. Unlike other alternatives such as Mendeley, Zotero can upload the attachments and data to a private cloud via WebDav.

If you use nginx as your web server, know that even though it provides partial support for webdav, Zotero needs more than that. Hence, you will need another webdav server, and optionally let nginx proxy to it. This short post provides the basics to get that set-up working under Debian/Ubuntu.

Setting up Apache

First we need to install Apache:

1	sudo apt-get install apache2

Change the head of "/etc/apache2/sites-enabled/000-default" to:

1	<VirtualHost *:880>

Then, create a file /etc/apache2/sites-available/webdav:

Alias /dav /home/webdav/dav
<Location /dav>
    Dav on
    Order Allow,Deny
    Allow from all
    Dav On
    Options +Indexes
    AuthType Basic
    AuthName DAV
    AuthBasicProvider file
    AuthUserFile /home/webdav/.htpasswd
    Require valid-user
</Location>

Ideally, you want your webdav folders to be private, adding authentication to them. So you need to create the webdav and zotero users and add the passwords to an htpasswd file. Even though you could use a single user, since you will be configuring several clients with your credentials I encourage you to create the zotero user as well. This way you can always change the password for zotero without affecting any other application using webdav.

sudo adduser webdav
sudo htpasswd -c /home/webdav/.htpasswd webdav
sudo htpasswd /home/webdav/.htpasswd zotero
sudo mkdir -p /home/webdav/dav/zotero

Enable the site and restart apache:

sudo a2enmod webdav
sudo a2enmod dav_fs
sudo a2ensite webdav
sudo service apache2 restart

At this point everything should be working at http://<your_host>:880/dav/zotero

Setting up NGINX

After the Apache side is working, we can use nginx as a proxy to get cleaner URIs. In your desired site/location, add this:

location /dav {
    client_max_body_size 20M;
    proxy_set_header X-Real-IP  $remote_addr;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header Host $host;
    proxy_pass http://127.0.0.1:880;
}

Now just reload nginx:

1	sudo service nginx force-reload

Extras

Zotero Reader - HTML5 client
Zandy - Android Open Source client

Proxies with Apache and python

2014-10-09T10:00:00+02:00

This is a quick note on proxying a local python application (e.g. flask) to a subdirectory in Apache. This assumes that the file wsgi.py contains a WSGI application with the name application. Hence, wsgi:application.

Gunicorn

<Location /myapp/>
    ProxyPass http://127.0.0.1:8888/myapp/
    ProxyPassReverse http://127.0.0.1:8888/myapp/
    RequestHeader set SCRIPT_NAME "/myapp/"
</Location>

Important: SCRIPT_NAME and the end of ProxyPass URL MUST BE THE SAME. Otherwise, Gunicorn will fail miserably.

Try it with:

1	venv/bin/gunicorn -w 4 -b 127.0.0.1:8888 --log-file - --access-logfile - wsgi:application

UWSGI

This is a very simple configuration. I will try to upload one with more options for uwsgi (in a .ini file).

<Location /myapp/>
    SetHandler uwsgi_handler
    uWSGISocker 127.0.0.1:8888
</Location>

Try it with:

1	uwsgi --socket 127.0.0.1:8888 -w wsgi:application

Extra: Supervisor

If everything went as expected, you can wrap your command in a supervisor config file and let it handle the server for you.

[unix_http_server]
file=/tmp/myapp.sock ; path to your socket file

[supervisord]
logfile = %(here)s/logs/supervisor.log
childlogdir = %(here)s/logs/

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
logfile = %(here)s/logs/supervisorctl.log
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket

[program:myapp]
command = venv/bin/gunicorn -w 4 -b 0.0.0.0:5000 --log-file %(here)s/logs/gunicorn.log --access-logfile - wsgi:application
directory = %(here)s
environment = PATH=%(here)s/venv/bin/
logfile = %(here)s/logs/myapp.log

Publishing on PyPi

2014-09-27T10:00:00+02:00

Developing a python module and publishing it on Github is cool, but most of the times you want others to download and use it easily. That is the role of PyPi, the python package repository. In this post I show you how to publish your package in less than 10 minutes.

Choose a fancy name

If you haven't done so yet, take a minute or two to think about this. To publish on PyPi you need a name for your package that isn't taken. What's more, a catchy and unique name will help people remember your module and feel more inclined to at least try it.

The package name should hint what your module does, but that's not always the case. That's your call. I personally put uniqueness and memorability over describing the functionality.

Create a .pypirc configuration file

[distutils] # this tells distutils what package indexes you can push to
index-servers =
    pypi # the live PyPI
    pypitest # test PyPI

[pypi] # authentication details for live PyPI
repository = https://pypi.python.org/pypi
username = { your_username }
password = { your_password } # not necessary

[pypitest] # authentication details for test PyPI
repository = https://testpypi.python.org/pypi
username = { your_username }

As you can see, you need to register both in the main pypi repository and the testing server. The usernames and passwords might be different, that is up to you!

Prepare your package

This should be the structure:

root-dir/    # Any name you want
    setup.py
    setup.cfg
    LICENSE.txt
    README.md
    mypackage/
        __init__.py
        foo.py
        bar.py
        baz.py

setup.cfg

1 2	[metadata] description-file = README.md

The markdown README is the de facto standard in Github, but you can also use rST (reStructuredText), the standard in the python community.

setup.py

from distutils.core import setup

setup(name = 'mypackage',
      packages = ['mypackage'], # this must be the same as the name above
      version = '{ version }',
      description = '{ description }',
      author = '{ name }',
      email = '{ email }',
      url = 'https://github.com/{user}/{package}', # URL to the github repo
      download\_url = 'https://github.com/{user}/{repo}/tarball/{version}',
      keywords = ['websockets', 'display', 'd3'], # list of keywords that represent your package
      classifiers = [], )

You might notice that the download_url points to a Github URL. We could host our package anywhere, but Github is a convenient option. To create the tarball and the zip packages, you only need to tag a tag in your repository and push it to github:

1 2	git tag {version} -m "{ Description of this tag/version}" git push --tags origin master

Push to the testing/main pypi server

It is advisable that you try your package on the test repository and fix any problems first. The process is simple: python setup.py register -r {pypitest/pypi} python setup.py sdist upload -r {pypitest/pypi}

If everything went as expected, you can now install your package through pip and browse your package's page. For instance, check my senpy package: https://pypi.python.org/pypi/senpy pip install senpy

Updating EuroLoveMap

2014-03-27T14:00:00+01:00

As part of the OpeNER hackathon we decided to build a prototype that would allow us to compare how different countries feel about several topics. We used the OpeNER pipeline to get the sentiment from a set of newspaper articles we gathered from media in several languages. Then we aggregated those articles by category and country (using the source of the article or the language it was written in), obtaining the "overall feeling" of each country about each topic. Then, we used some fancy JavaScript to make sense out of the raw information.

It didn't go too bad, it turns out we won.

Now, it was time for a face-lift. I used this opportunity to play with new technologies and improve it:

Using Flask, this time using python 3.3 and Bootstrap 3.0
Cool HTML5+JS cards (thanks to pastetophone)
Automatic generation of fake personal data to test the interface
Obfuscation of personal emails

The result can be seen here.

Publishing a Python 3 app on Heroku

1	mkvirtualenv -p /usr/bin/python3.3 eurolovemap

Since Heroku uses python 2.7 by default, we have to tell it which version we want, although it supports python 3.4 as well. I couldn't get python 3.4 working using the deadsnakes ppa, so I used python 3.3 instead, which works fine but is not officially supported. Just create a file named runtime.txt in your project root, with the python version you want to use:

1	python-3.3.1

Don't forget to freeze your dependencies so Heroku can install them: bash pip freze > requirements.txt

Publishing personal emails

There are really sophisticated and effective ways to obfuscate personal emails so that spammers cannot easily grab yours. However, this time I needed something really simple to hide our emails from the simplest form of crawlers. Most of the team are in academia somehow, so in the end all our emails are available in sites like Google Scholar. Anyway, nobody likes getting spammed so I settled for a custom Caesar cipher. Please, don't use it for any serious application if you are concerned about being spammed.

1 2	def blur_email(email): return "".join([chr(ord(i)+5) for i in email])

And this is the client side:

window.onload = function(){
     elems = document.getElementsByClassName('profile-email');
     for(var e in elems){
        var blur = elems[e].innerHTML;
        var email = "";
        for(var s in blur){
            var a = blur.charCodeAt(s)
            email = email+String.fromCharCode(a-5);
        }
        elems[e].innerHTML = email;
     }
}

Unfortunately, this approach does not hide your email from anyone using PhantomJS, ZombieJS or similar. For that, other approaches like generating a picture with the address would be necessary. Nevertheless, it is overkill for a really simple ad-hoc application with custom formatting and just a bunch of emails that would easily be grabbed manually.

Generation of fake data

To test the contact section of the site, I wanted to populate it with fake data. Fake-Factory is an amazing library that can generate fake data of almost any kind: emails, association names, acronyms... It even lets you localise the results (get Spanish names, for instance) and generate factories for certain classes (à la Django).

But I also wanted pictures, enter Lorem Pixel. With its API you can generate pictures of almost any size, for different topics (e.g. nightlife, people) and with a custom text. You can even use an index, so it will always show the same picture.

For instance, the picture below is served through Lorem Pixel.

By the way, if you only want cat pictures, take a look at Placekitten. And for NSFW text, there's the Samuel L. Jackson Ipsum

Remove git files with globbing

2013-08-22T23:14:00+02:00

A simple trick. If you want to remove all the '.swp' files from a git repository, just use:

1	git rm --cached '\\.swp'

Creating my web

2013-08-22T14:14:22+02:00

Finally, I've decided to set up a decent personal page. I have settled for github-pages because I like the idea of keeping my site in a repository and having someone else host and deploy it for me. The site will be really simple, mostly static files. Thanks to Github, Jekyll will automatically generate static pages for my posts every time I commit anything new to this repository.

But Jekyll can be used independently, so if I ever choose to host the site myself, I can do it quite easily. Another thing that I liked about this approach is that the generated html files can be used in the future, and I will not need Jekyll to serve it. Jekyll is really simple and most of the things are written in plain html. That means that everything could be easily reused if I ever choose to change to another blogging framework (e.g. pelical). But, for the time being, I like the fact that Github takes care of the compilation as well, so I can simply modify or add files through the web interface should I need to.

I hadn't played with HTML and CSS for a while now, so I also wanted to use this site as a playground. At some point, I realised I was doing mostly everything in plain HTML and CSS, and decided to keep it like that for as long as possible. As of this writing, I haven't included any Javascript code in the page. Probably I will use some to add my gists and repositories, but we will see about that.

I think the code speaks for itself, so you can check out my repository on Github. You can clone and deploy it easily like this:

1
2
3

git clone
https://github.com/balkian/balkian.github.com cd balkian.github.com
jekyll serve -w

I will keep updating this post with information about:

Some Jekyll plugins that might be useful
What CSS tricks I learnt
The webfonts I used
The badge on the left side of the page