Title

---

# Python and Openstack
## Quick tour on some libraries

Sylvain Bauza

Python User Group Grenoble, 27/02/2014

---

# Agenda

1. Introduction
2. Pecan and WSME
3. Cliff
4. Stevedore
5. Oslo

---

# $ id

## At Bull, working on a FOSS project named [XLCloud](http://www.xlcloud.org)

## Openstack _Active Technical Contributor_, in particular with Compute and Reservation services

---

# But what the ... is Openstack ?

![Openstack Conceptual diagram](pyug_src/openstack-software-diagram.png)

---

# No, wait, Openstack is that !

![Openstack logical diagram](pyug_src/openstack-logical-diagram.jpg)

---

# Hold on, I saw that...

![Openstack Horizon Dashboard](pyug_src/openstack-horizon.jpg)]

---

# CI/CD are not offensive words

.left-column[
### Review system with Gerrit
]
.right-column[
![Review of a patch](pyug_src/gerrit_patch.jpg)]
]
---
class: img_col

# CI/CD are not offensive words

.left-column[
### Review system with Gerrit
### Zuul Pipeline
]
.right-column[
![Zuul overview](pyug_src/zuul_overview.jpg)]
]
---

# CI/CD are not offensive words

.left-column[
### Review system with Gerrit
### Zuul Pipeline
### Jenkins tests 
]
.right-column[
![Jenkins checks](pyug_src/gerrit_jenkins.jpg)]
]

---
# OK but Python by itself ?

## Per-project coding
---
# OK but Python by itself ?

## Per-project coding
## Efforts for sharing good practices
---
# OK but Python by itself ?

## Per-project coding
## Efforts for sharing good practices
## A set of common libraries containing code shared in Openstack
---
# OK but Python by itself ?

## Per-project coding
## Efforts for sharing good practices
## A set of common libraries containing code shared in Openstack
## ...and some other external libraries (say hello to Pypi.openstack.org)
---

## Pecan and WSME
[pɪˈkɑn] and [ˈwɪzˈme]

---

# Pecan

_"a very lightweight [HTTP] framework that provides object-dispatch style routing"_

- WSGI application running on top of a WSGI server (wsgiref, mod_wsgi, eventlet...)
- REST-style controllers
- Object routing
- Templating
- Hooks

---

# Configuration

```python
# Server Specific Configurations
server = {
    'port': '8080',
    'host': '0.0.0.0'
}

# Pecan Application Configurations
app = {
    'root': '${package}.controllers.root.RootController',
    'modules': ['${package}'],
    'static_root': '%(confdir)s/public',
    'template_path': '%(confdir)s/${package}/templates',
    'debug': True,
    'errors': {
        '404': '/error/404',
        '__force_dict__': True
    }
}
```
---

# Running the application

```python
pecan.configuration.set_config(dict(pecan_config), overwrite=True)

app = pecan.make_app(
    pecan_config.app.root,
    static_root=pecan_config.app.static_root,
    template_path=pecan_config.app.template_path,
    debug=CONF.debug,
    force_canonical=getattr(pecan_config.app, 'force_canonical', True),
    hooks=app_hooks,
    wrap_app=middleware.ParsableErrorMiddleware,
    guess_content_type_from_ext=False
)

wsgiref.simple_server.make_server(host, port, app).serve_forever()
```
---

# Take some REST

from mymodel import Author, Book

class BooksController(RestController):

@expose()
    def get(self, author_id, id):
        author = Author.get(author_id)
        if not author_id:
            abort(404)
        book = author.get_book(id)
        if not book:
            abort(404)
        return book.title

class AuthorsController(RestController):

books = BooksController()

@expose()
    def get(self, id):
        author = Author.get(id)
        if not author:
            abort(404)
        return author.name

class RootController(object):

authors = AuthorsController()
```
]
---
# Templating with Pecan

With Mako (default):
```python
class MyController(object):
    @expose('path/to/mako/template.html')
    def index(self):
        return dict(message='I am a mako template')
```
or with Kajiki:
```python
@expose('kajiki:path/to/kajiki/template.html')
def my_controller(self):
    return dict(message='I am a kajiki template')
```
or your own...:
```python
@expose()
def controller(self):
    return render('my_template.html', dict(message='I am the namespace'))
```
---

# Pecan Hooks, Captain !

```python
from pecan.hooks import PecanHook

class SimpleHook(PecanHook):

def before(self, state):
        print "\nabout to enter the controller..."

def after(self, state):
        print "\nmethod: \t %s" % state.request.method
        print "\nresponse: \t %s" % state.response.status
```
---

# WSME

_"Web Service Made Easy (WSME) simplify the writing of REST web services by providing simple yet powerful typing which removes the need to directly manipulate the request and the response objects"_

- Simple input validation with defined and extensible types 
- Native handling of different protocols (REST/Json, REST/XML, SOAP, etc.)
- Framework independance (Pecan, Flask, cornice...)
- Sphinx integration for automatic API documentation

---
# WSME types

- Native types : 
  - bytes, int, text (_Six.text_type_), float, bool, decimal, date, datetime, time, Arrays, Dicts
- User types : 
  - binary
  - Enum
  - UUIDType
  - StringType
- Complex types : should be inheriting from Base type. Eg.:

```
Gender = wsme.types.Enum(str, 'male', 'female')
Title = Ewsme.types.num(str, 'M', 'Mrs')

class Person(wsme.types.Base):
    lastname = wsme.types.wsattr(unicode, mandatory=True)
    firstname = wsme.types.wsattr(unicode, mandatory=True)

age = int
    gender = Gender
    title = Title

hobbies = [unicode]
``` 
---

# Example with Pecan

The previous example from Pecan becomes :
```python
from wsmeext.pecan import wsexpose

class BooksController(RestController):
    @wsexpose(Book, int, int)
    def get(self, author_id, id):
        # ..

@wsexpose(Book, int, int, body=Book)
    def put(self, author_id, id, book):
        # ..

class AuthorsController(RestController):
        books = BooksController()
```
---

# Sphinx documentation

- Extension for the Sphinx project
- Autodoc directives
- Samples to provide

Example: 
```rst
.. rest-controller:: climate.api.v2.controllers.lease:LeasesController
   :webprefix: /v2/leases

.. autotype:: climate.api.v2.controllers.lease.Lease
   :members:
```
---

## Cliff
[Command Line Interface Formulation Framework]

---

# What is Cliff ?

- Framework for creating pluggable CLI
- Objects model for commands
- Native formatters for list outputs (CSV, table...)
- Command Completion built-in
- Interactive mode

---

# Climb the Cliff

from cliff.app import App
from cliff.commandmanager import CommandManager

class DemoApp(App):

def __init__(self):
        super(DemoApp, self).__init__(
            description='cliff demo app',
            version='0.1',
            command_manager=CommandManager('cliff.demo'),
            )

def initialize_app(self, argv):
        # Before command processing

def prepare_to_run_command(self, cmd):
        # Before the command is run

def clean_up(self, cmd, result, err):
        # After the command is run

def main(argv=sys.argv[1:]):
    myapp = DemoApp()
    return myapp.run(argv)

if __name__ == '__main__':
    sys.exit(main(sys.argv[1:]))
```
]

---

# Show command

from cliff.show import ShowOne

class File(ShowOne):
    "Show details about a file"

def get_parser(self, prog_name):
        parser = super(File, self).get_parser(prog_name)
        parser.add_argument('filename', nargs='?', default='.')
        return parser

def take_action(self, parsed_args):
        stat_data = os.stat(parsed_args.filename)
        columns = ('Name',
                   'Size',
                   'UID',
                   'GID',
                   'Modified Time',
                   )
        data = (parsed_args.filename,
                stat_data.st_size,
                stat_data.st_uid,
                stat_data.st_gid,
                stat_data.st_mtime,
                )
        return (columns, data)
```
]

---

# Show command

from cliff.show import ShowOne

class File(ShowOne):
    "Show details about a file"

def get_parser(self, prog_name):
        parser = super(File, self).get_parser(prog_name)
        parser.add_argument('filename', nargs='?', default='.')
        return parser

.tile1[
```bash
$ cliffdemo file setup.py
+---------------+--------------+
|     Field     |    Value     |
+---------------+--------------+
| Name          | setup.py     |
| Size          | 5825         |
| UID           | 502          |
| GID           | 20           |
| Modified Time | 1335569964.0 |
+---------------+--------------+
```
]

---

# List command and formatters

```python
import logging
import os

from cliff.lister import Lister

class Files(Lister):
    """Show a list of files in the current directory.

The file name and size are printed by default.
    """

log = logging.getLogger(__name__)

def take_action(self, parsed_args):
        return (('Name', 'Size'),
                ((n, os.stat(n).st_size) for n in os.listdir('.'))
                )
```
---

# List command and formatters

from cliff.lister import Lister

class Files(Lister):
    """Show a list of files in the current directory.

The file name and size are printed by default.
    """

log = logging.getLogger(__name__)

def take_action(self, parsed_args):
        return (('Name', 'Size'),
                ((n, os.stat(n).st_size) for n in os.listdir('.'))
                )
```
]

.tile1[
```bash
$ cliffdemo files
+---------------+------+
|      Name     | Size |
+---------------+------+
| build         |  136 |
| cliffdemo.log | 2546 |
| Makefile      | 5569 |
| source        |  408 |
+---------------+------+

$ cliffdemo files -f csv
"Name","Size"
"build",136
"cliffdemo.log",2690
"Makefile",5569
"source",408
```
]
---

## Stevedore
[who's this guy ?]

---

# Stevedore

_"Stevedore, dockworker, docker, dock labourer, wharfie, and longshoreman can have various waterfront-related meanings concerning loading and unloading ships, according to place and country."_
.pull-right[http://en.wikipedia.org/wiki/Stevedore ]

- Dynamic code loading as plugins
- Use of setuptools for entry points
- Various loading patterns (Drivers, Hooks, Extensions)
- Plugins enabling either through installation, explicitely or self-enabled
- Manager classes wrapping .font_courier[pkg_resources]

---

# Loading Patterns

.right-column[
.center[
###__DriverManager__ class
![Driver Pattern](pyug_src/stevedore_driver.png)
]
]
---
# Loading Patterns

.left-column[
### Drivers : Single Name, Single Entry Point
### Hooks : Single Name, Many Entry Points
]

.right-column[
.center[
###__HookManager__ class
.img_full[
![Hook Pattern](pyug_src/stevedore_hook.png)
]
]
]
---
# Loading Patterns

.left-column[
### Drivers : Single Name, Single Entry Point
### Hooks : Single Name, Many Entry Points
### Extensions : Many Names, Many Entry Points
]

.right-column[
.center[
###__ExtensionManager__ class and subclasses (Named, Dispatch, Enabled)
![Extension Pattern](pyug_src/stevedore_extension.png)
]
]

---

# Example Base Plugin

```python
# stevedore/example/base.py
import abc

class FormatterBase(object):
    """Base class for example plugin used in the tutoral.
    """

__metaclass__ = abc.ABCMeta

def __init__(self, max_width=60):
        self.max_width = max_width

@abc.abstractmethod
    def format(self, data):
        """Format the data and return unicode text.

:param data: A dictionary with string keys and simple types as
                     values.
        :type data: dict(str:?)
        :returns: Iterable producing the formatted text.
        """
```
---

# Example Plugin

```python
# stevedore/example/simple.py
from stevedore.example import base

class Simple(base.FormatterBase):
    """A very basic formatter.
    """

def format(self, data):
        """Format the data and return unicode text.

:param data: A dictionary with string keys and simple types as
                     values.
        :type data: dict(str:?)
        """
        for name, value in sorted(data.items()):
            line = '{name} = {value}\n'.format(
                name=name,
                value=value,
            )
            yield line
```
---

# Example Plugin

class Simple(base.FormatterBase):
    """A very basic formatter.
    """

def format(self, data):
        """Format the data and return unicode text.

.tile1_left[
```
entry_points={
    'stevedore.example.formatter': [
        'simple = stevedore.example.simple:Simple',
        'field = stevedore.example.fields:FieldList',
        'plain = stevedore.example.simple:Simple',
    ],
},
```
]
---

# Loading the plugin
.left-column[
### Driver
]

from stevedore import driver

parser = argparse.ArgumentParser()
parser.add_argument(
    'format',
    nargs='?',
    default='simple',
    help='the output format',
)
parsed_args = parser.parse_args()

mgr = driver.DriverManager(
    namespace='stevedore.example.formatter',
    name=parser.format,
    invoke_on_load=True,
)
for chunk in mgr.driver.format(data):
    print(chunk, end='')
```
]
---
# Loading the plugin
.left-column[
### Driver
### Extension and Hook
]

mgr = extension.ExtensionManager(
    namespace='stevedore.example.formatter',
    invoke_on_load=True,
)

def format_data(ext, data):
    return (ext.name, ext.obj.format(data))

results = mgr.map(format_data, data)

for name, result in results:
    print('Formatter: {0}'.format(name))
    for chunk in result:
        print(chunk, end='')
    print('')
```
]
---
class: inverse, middle, center

### Oslo
[not the town...]

---

# Oslo

__Mission Statement__:
_"To produce a set of python libraries containing code shared by OpenStack projects. The APIs provided by these libraries should be high quality, stable, consistent, documented and generally applicable."_

- Openstack Program by itself, with a PTL (Project Tech Lead)
- Using Openstack CI
- Set of libraries internally and initially developed for Openstack
- Graduation through incubator and then PyPi
- Sync tool for incubated libraries (Git repository)
- Particularly various libraries

---

# Oslo libraries

from oslo.config import cfg

CONF = cfg.CONF
opts = [
    cfg.IntOpt('port', default=1234,
               help='Port that will be used to listen on'),
]

CONF.register_opts(opts)

def mymethod():
    print CONF.port
```
]
]
---

# Oslo libraries

.right-column[
.code12px[
```python
class RPCClient(object):
    def __init__(self, target):
        super(RPCClient, self).__init__()
        self._client = messaging.RPCClient(
            target=target,
            transport=messaging.get_transport(cfg.CONF),
        )

def cast(self, name, **kwargs):
        ctx = context.current()
        return self._client.cast(ctx.to_dict(), name, **kwargs)

def call(self, name, **kwargs):
        ctx = context.current()
        return self._client.call(ctx.to_dict(), name, **kwargs)

class RPCServer(service.Service):
    def __init__(self, target):
        super(RPCServer, self).__init__()
        self._server = messaging.get_rpc_server(
            target=target,
            transport=messaging.get_transport(cfg.CONF),
            endpoints=[ContextEndpointHandler(self, target)],
        )
```
]
]

---
# Oslo libraries

from setuptools import setup

setup(
    setup_requires=['pbr'],
    pbr=True,
)
```
```
[metadata]
name = climate
summary = Reservation Service for OpenStack clouds
(...)
home-page = https://launchpad.net/climate

[global]
setup-hooks = pbr.hooks.setup_hook

[files]
packages =
    climate

[entry_points]
console_scripts =
    (...)
```
]
]

---
# Oslo libraries

]

mylist = list() # BAD, shadows `list` built-in

class Foo(object):
    def list(self):
        return [1, 2, 3]

mylist = Foo().list() # OKAY, does not shadow built-in
```
```python
if not X is Y:  # BAD, intended behavior is ambiguous
    pass

if X is not Y:  # OKAY, intuitive
    pass
```
etc.

http://docs.openstack.org/developer/hacking/
]
]

---
# Oslo libraries

.left-full[
### Oslo.config
### Oslo.messaging
### Pbr
### Hacking
### Many many others... (rootwrap, cookiecutter, taskflow, fixtures, db, jsonutils, log, policy...)
]
---
# Oslo libraries

.left-full[
### Oslo.config
### Oslo.messaging
### Pbr
### Hacking
### Many many others... (test, rootwrap, cookiecutter, taskflow, fixtures, db, jsonutils, log, policy...)
### ... and the ones I just presented !
]
---
class: inverse

## References

- Pecan : http://pecan.readthedocs.org/en/latest/
- WSME : http://wsme.readthedocs.org/en/latest/
- Cliff : https://cliff.readthedocs.org/en/latest/
- Stevedore : http://stevedore.readthedocs.org/en/latest/
- Oslo : https://wiki.openstack.org/wiki/Oslo

- And Openstack code and reviews of course ! 
---
class: inverse, middle, center

### You reached it ! Questions tho ?