Before we begin

Code available at:

https://github.com/balkian/lab-in-a-box

Live demos at:

https://github.todevnull.com

https://lab.todevnull.com

https://hub.todevnull.com

Feel free to log in, but try not to break them for now 😉

My name is Fernando and…

At Grupo de Sistemas Inteligentes

http://www.gsi.dit.upm.es

And I ❤ Docker

About this talk

Takeaway: you can set up a multi-user data analysis environment with isolation in minutes

Plus: using docker to perform and share experiments is even easier

Related Meetups:

Big Data and Machine Learning with Docker

Using Docker in Machine Learning Projects

Experiment, publish, repeat

Reproducibility

Obstacles

• Missing data
• Bleeding edge tools and libraries
• Throwaway software
  • Hacky
  • Little to no documentation
• Multiple languages

Obstacles

Is it a problem?

Jupyter notebooks

Jupyter architecture

Docker to the rescue

Jupyter/docker-stacks

Reproducible environment

docker run --rm -p 8888:8888 \
    -v $(WDIR)/:/home/jovyan/work/ \
    jupyter/scipy-notebook

And friendly, too

version: '2'
services:
  jupyter:
    image: jupyter/scipy-notebook
    volumes:
    - "./.nbconfig:/home/jovyan/.jupyter/nbconfig"
    - "./work:/home/jovyan/work/"
    ports:
    - "8888:8888""

docker-compose up

Related projects

• Using docker images to share trained systems

Requirements

• Shared environments
• Resource sharing
• Easy configuration
• Versioning
• Backups

And little to no overhead

Isolation

Jupyterhub

Authenticators

• Local
• OAuth
• LDAP
• JWT

Spawners

• Local
• Docker
• Kubernetes
• Marathon

More infrastructure

It’s demo time

https://github.todevnull.com https://github.com/balkian/lab-in-a-box

Zeppelin

• Alternative to Jupyter

CoCalc

• Alternative to Jupyter

Docker-Nvidia

• CUDA for docker

Jupyter Binder

• Custom Jupyter from git repositories

Knowledge-Repo

Lessons learned

• Docker + Docker-compose
  • Reproducible environments (partially)
  • Reduced tooling / experience
  • Ephemeral containers force you to automate/document installation
• Jupyterhub
  • Shared environments
  • Web interface (zero knowledge)

What’s missing?

• Roles and permissions

• Backups

• Ideas:
  • Kubernetes?
  • OpenShift?

Thanks for listening!

https://github.com/balkian/lab-in-a-box

jf.sanchez@upm.es