Involving to the end-user

The Knit Editor software aims to be installable and usable by end-users. In the whole summer of code, we focused on development and code documentation from the perspective of a developer. In this blog post we will discuss how the Knit Editor is presented to the end user. So, you reading this blog: Please comment with your thoughts on the sketches.

Currently, a site is in the making that shall present the Knit Editor software as is state of the art. The inspiration came from the talk by Tracy Osborn at PyCon 2016: “Web Design for Non Designers”. The site is currently in the making at fossasia.github.io/kniteditor. If you click on the following images, you get redirected to the implementation of the concept.

The Main Page

First thing that comes into view is the download button. This leads to the download site. Then, wen can see three popular use-cases of the knit editor. At the bottom, new developers can see that they can contribute.

End users are knitters of all ages. As tested with my mom, they expect the language to be at the top-right of the page.Main Page

Both, the download and the start developing button are highlighted in a different color to make certain that they are an action the user is expected to perform.

When you access the site you get automatically redirected to your browsers configured language.

The Download Site

When you click the download button on the main page, you reach the download site. Depending on the operating system information the browser sends, your download starts automatically, below, this is  sketched for Windows.

P1040129

Next to the download page, you may want to find other versions of the software or not. This is to be evaluated. Maybe a slightly less visible button is right for that or it can be left out. Usually, no-one uses the old software.

At the bottom, you can see that there is a predecessor of the software which is called “AYAB-Apparat”. Some people may expect to find this software, too.

The Developer Site

If you clicked “Start Developing” on the main page, you will be confronted with the site for development.

There are two ways main flavors of contributing. Either you translate or you write program code. Therefore, we have two buttons that skip to the corresponding sections.

P1040130

At the bottom, you can see that there are tutorials on how to set up the environment for development. Videos for this can be found under this Youtube playlist.

Summary

At the end of GSoC we should document the code. Since we did documentation-driven development, there was already a focus on the developers from the start. End-user involvement fell short during the development phase. “Documentation is the way of informing people.” – this is something I learned from a talk. Thus, I create the new site for the knit editor as a documentation about the project fit for non-developers.

Introducing new Developers to Your Project

You may want to introduce new developers into your project. This is either to get help or to allow the community to guide the project. In this blog post, I want to present how I believe an introduction can be designed.

(1) A Landing Page

For the knitting projects, I have designed a landing page, knitting.fossasia.org. It features all related projects with links to documentation and source code as well as a short summary what it is about. You can read more about the landing page in this blog post.

The idea is to to add short snippets of information that allow new developers to explore all available options to enter the projects. This may be contact information, documentation and videos.

(2) Setup Instructions

Especially for the projects I worked on, I created a introduction videos on how to set up your development environment. This could have been in text form, too.

The problem for new developers is often that the start takes a long time. Instead of focusing on the issue they would like to solve, it sometimes takes several hours to set up the environment with lots of possibilities to make mistakes.

In order to speed up the development environment setup, you can provide additional information. The CoderDojo Zen platform has its own landing page for new developers and a description of the steps to take to setup the development environment. In the case of the knitting projects, we have YouTube video tutorials which show the necessary steps.

(3) Introduction Events

You can participate in events to show your projects to other people.

With the knitting projects, we will participate at the Maker Faire Berlin 2016.There may be local user groups where you can present the work and the learnings from them.

Google Code-In is an excellent opportunity to allow new young developers to add code to your projects and document them. You can apply as a mentor and add tasks. A first task for beginners may be to get the project running on their computer. Another task could then be to either use it in a special way and document it or to solve an issue which is easy enough.

In the following Google Summer of Code your project can be used again. You can apply as a mentor for your project and help other students to work on what you left behind.

Summary

I listed some ways I will perform to get the projects I worked on into the community. I hope they may have been an inspiration for other people who read this. If you like to contribute new ways, you can comment ūüôā

Knit Editor Package Overview

In this years Google Summer of Code, we created several Python package. They are all available on the Python Package Index (PyPi) and installable via “pip install”. They are listed on knitting.fossasia.org but their interconnections are shown here.

In the following figure, you can see the different packages created during GSoC with a solid line. Other packages that are used by the “kniteditor” application are shown here with a dotted line.

 

Knit Editor Package Architecture
Knit Editor Package Architecture

Overall, five packages we created. The design is driven by responsibility. Thus the responsibilities of each packages should be clearly separated from the other packages. We describe the responsibilities of the packages as follows:

knittingpattern is the library for converting, loading, saving and manipulating knitting patterns. These patterns include in formation abour how to knit. Unlike a picture this includes more than a color: adding meshes, removing meshes, types of holes, the possibility of non-planar knit pieces ad more.

ObservableList is a list whose content can be observed. Whenever elements are added or removed, this list notifies the observers. This is used by the rows of instructions to provide a more convenient interface.

crc8 computes the crc8 hash from bytes. I did not find a Python library implementing ths functionality so I created it myself. The design follows the design of the hash functions in the Python standard library. This package is used by the AYABInterface package. Through creating a new package, this is also usable by other applications.

AYABInterface controls the knitting machines connected to the AYAB hack. Through the serial interface, it can send messages to the controllers and receive answers. It also provides hint for actions which the user should take in order to produce the desired outcome.

kniteditor contains the user interface to edit knitting patterns and control the knitting machines with the AYAB shield.

All these packages also include installation instructions.

AYABInterface – a Python Module for the AYAB shield

In the Google Summer of Code effort on knitting machines with AYAB, we created the AYABInterface module. This module allows us to control machines like the Brother KH-910. Click here to see it in action.

The development process for small changes worked like this:

  1. Talk about uncertainty in the issue.
  2. Write the specification.
  3. Write the tests.
  4. Write the code.

Of cause these steps were mixed but the objective is clear to first specify and then let the implementation follow.

From my perspective this made sure that other people can implement this protocol, too. After all, it should be specific enough now, to write source code for it.

The underlying Communication relies on a byte stream to communicate. This abstracts from the serial protocol and enables us to plug in e.g. sockets if someone wants to communicate via WIFI or LAN.

The communication walks through several states. They are documented in the states module. Here you can see an overview over the different states:

CommunicationStateDiagram

If you click it, you are redirected to an interactive documentation web page where you can click the different states and messages to view their implementation.

Summary

After all, communication is important, not only between the shield and the computer but also for the developers. Since the Shield is around for a while, this protocol is now documented in a way that allows us to use it also for other applications. A specification driven implementation allows us both:

  • a complete specification where we know what is possible to implement
  • an implementation that follows the specification and as such is exchangeable and a reference for other library developers.

 

New Landing Page for FOSSASIA Knitting Projects

Now, FOSSASIA has a new landing page for the different knitting projects, knitting.fossasia.org.

The objective of this page is to ease starting development on knitting projects. As such, it is listed on the labs site of FOSSASIA. The FOSSASIA labs give you the opportunity to get into development of open-source projects in the FOSSASIA community.

The site features a list of projects related to knitting – software and hardware projects. They all have Github issues attached and you can access them via a waffle link. There is a unified view on all issues for convenience.

In the section below, you can see how you can contribute. For translations, no coding is required but it is useful to know the Github pull-request workflow.

The next step is to start coding without understanding much of the inner workings of the project. This can be done by optimizing some metrics like code quality, health and coverage. This way, new developers setup the code base, get familiar with the code and have an easy first contribution.

To start implementing features, we have a list of issues attached. they are organized in waffle projects. They have tags like “easy” or “hard” to guide the choice. Easy issues can be worked on by developers who want to get into the code base and do not require much knowledge of the inner workings of the project.

In the bottom, there is also a community section that shows how to get in touch and a section for related work. This sections is extremely important as it allows us to expand the community to users, get inspiration. This can be seen as a repository for human interconnections. If a developer leaves the project, all commits remain. However, the connections to the people leave. In order to keep these connections, this section is created. It features a list of other sites, wiki pages and places t read and inform oneself.

Summary

Creating a community around project is new to me. I did this the first time in the Flappy-SVG project. With the knitting.fossasia.org site, I hope to see a community evolve around the knitting projects in FOSSASIA.

Knitting with the Knit Editor – First Success

With AYAB we are working on an editor which also has the ability to control a knitting machine. Here is the progress we made: one can control the needles and get some instructions on what to do:

Views of the Editor

The editor has several view. To begin with, you can either load or save patterns as knitting patterns or images:

1

This is the incomplete editor view, where you are able to add and remove instructions and rows and build your knit work:

2

Once, your pattern is done, you go over to the knit settings. Currently, there is only one ability to knit, knitting with the AYAB hack. You can choose your machine type and the connection and start knitting.

3

After you started knitting, you see the pattern which you want to knit.

4

Right next to the pattern, you can see the instructions for you. You can follow these to create the knit peace.

5

You can also open the settings menu by pressing F1. There you can choose your language:

6

At the current state, all this is rather sketchy. The basics work. For the best user experience, there is still a lot to do.

Awesome Kivy Revelations

We are using kivy for the GUI of the knit editor. It can be used to create Android, Windows, Linux, Mac and IOS apps. You can divide UI-design and code by using the kv language. In this blog post, I want to share some revelations I had when using kivy. This includes showing you how you can update translations automatically when the configuration changes and subtracting values. This awaits you:

Automatic Update of Translated Text in the Kivy UI

In projects like Django, there is the “_” function that allows translations of text. The implementation usually look calls a “gettext” function of some sort, which takes a string and returns its translation as a string. What we have in the kniteditor, is an observable translation, with the same interface:

def _(string):
    """Translate a string using the current language.
    :param str string: the string to translate
    :return: the translated string
    :rtype: str
    """
    return _locales.gettext(string)
_ = ObservableTranslation(_)

The difference is that the observable translation can be used like the “_” function but has additional methods that allow the UI to register and be notified when the language changes. When the language is changed, the global “gettext” is replaced by the “gettext” in the new language and, inthe last line, the observers are notified about the change.

def change_language_to(new_language):
    """Change the language to a language from the translations folder.

    :param str new_language: the language code of the new language
    """
    global _locales, _current_language
    _locales = gettext.translation(DOMAIN, _locale_dir,
                                   languages=[new_language])
    _current_language = new_language
    _.language_changed()

To see what this does, we can look at the whole implementation. I would like to give the whole picture, first, as it clarifies the context and discuss them below.

"""Observable Translations for kivy that change when the language changes.
The functionality of this module is highly inspired by
`kivy-gettext-example <https://github.com/tito/kivy-gettext-example>`.
"""
from kivy.lang import Observable


class ObservableTranslation(Observable):

    """This class allows kivy translations to be updated with the language."""

    def __init__(self, translate):
        """Create a new translation object with a translation function.
        :param translate: a callable that translates the text. 
          Even when the language is changed,
          it returns the text for the current language.
        """
        super().__init__()
        self._translate = translate
        self._observers = []

    def __call__(self, text):
        """Call this object to translate text.
        :param str text: the text to translate
        :return: the text translated to the current language
        """
        return self._translate(text)

    def fbind(self, name, func, args, **kwargs):
        """Add an observer. This is used by kivy."""
        self._observers.append((name, func, args, kwargs))

    def funbind(self, name, func, args, **kwargs):
        """Remove an observer. This is used by kivy."""
        key = (name, func, args, kwargs)
        if key in self._observers:
            self._observers.remove(key)

    def language_changed(self):
        """Update all the kv rules attached to this text."""
        for name, func, args, kwargs in self._observers:
            func(args, None, None)

__all__ = ["ObservableTranslation"]

The constructor takes the “_” function. When the object is called like a function, the “__call__” method is invoked, translating like “_”. If we only have these two methods, the observable translation works just like the “_” function.

“fbind” and “funbind” are the methods that are called when a translation is used in the kv language. They add and remove the observes from the list of observes.

“language_changed” walks through the observers and tells everyone of then to update. With this, you already have a updated translations when “change_language_to” is called.

Here is how you create a button in the kv language that uses the translation function:

#:import _ kniteditor.localization._
Root:
    Button:
        text: _("Load")

Updating the Translations From the Configuration

Ultimately, you want the translations to be changed, when the built-in kivy configuration changed. In the video, seconds, you have seen what it can look like.

LANGUAGE_SECTION = "language"
LANGUAGE_CODE = "current"
LANGUAGES = ["English", "Deutsch"]

class EditorWindow(App):

    """The editor window."""

    def build_config(self, config):
        """Build the configuration.

        :param kivy.config.ConfigParser config: the configuration parser
        .. seealso:: `Application Configuration
         <https://kivy.org/docs/api-kivy.app.html#application-configuration>`__
        """
        config.setdefaults(LANGUAGE_SECTION, {
            LANGUAGE_CODE: LANGUAGES[0]
        })

    def build(self):
        """Build the application."""
        self.update_language_from_config()

    def update_language_from_config(self):
        """Set the current language from the configuration.
        """
        config_language = self.config.get(LANGUAGE_SECTION, LANGUAGE_CODE)
        change_language_to(config_language)

    def build_settings(self, settings):
        """Create the applications settings dialog.

        :param  kivy.uix.settings.Settings settings: the settings 
          for this app
        .. seealso:: `Create a settings panel
          <https://kivy.org/docs/api-kivy.app.html#create-a-settings-panel>`__,
          :meth:`kivy.uix.settings.Settings.add_json_panel`,
          :mod:`kivy.uix.settings`
        """
        settings.add_json_panel(_('KnitEditor'), self.config,
                                data=self.settings_specification)

    @property
    def settings_specification(self):
        """The settings specification as JSON string.

        :rtype: str
        :return: a JSON string
        """
        settings = [
            {"type": "title",
             "title": _("KnitEditor")},

            {"type": "options",
             "title": _("Language"),
             "desc": _("Choose your language"),
             "section": LANGUAGE_SECTION,
             "key": LANGUAGE_CODE,
             "options": LANGUAGES},
        ]
        return json.dumps(settings)

    def on_config_change(self, config, section, key, value):
        """The configuration was changed.

        :param kivy.config.ConfigParser config: the configuration that was
          changed
        :param str section: the section that was changed
        :param str key: the key in the section that was changed
        :param value: the value this key was changed to
        """
        if section == LANGUAGE_SECTION and key == LANGUAGE_CODE:
            change_language_to(new_language)

“build_config” is calls when the application starts to fill the empty configuration with useful values.

“build” is called when the application creates its window, at this point in time, the translation is updated.

“build_settings” is called when the F1 key is pressed or the Android settings menu is activated. “settings_sepcification” is a JSON string used by kivy to build the settings.

“on_config_change” is called, when the configuration is updated. Through e.g. the settings dialog.

With these components together, you can have a updated language every time the language is changed. The video shows an application which uses this code.

Bonus Material: Kivy’s Update is Mighty

In the kv language, variables are accessed via identifiers and not via global variables. As such, the kv language can register every time a Property (like “StringProperty” and “ObjectProperty”) is used. It an register when they change. Thus you can automatically update formulas whenever a value changes.

in this example you can see how a value is subtracted but still, an automatic update is possible:

    Slider:
        min: 0
        max: 100
        value: 100 - other_slider.value

 

Credits

Thanks to Mathieu Virbel for the kivy-gettext-example. It was the basis of this work and an inspiration.

 

 

 

 

 

The new AYABInterface module

One create knit work with knitting machines and the AYAB shield. Therefore, the computer communicates with the machine. This communication shall be done, in the future, with this new library, the AYABInterface.

Here are some design decisions:

Complete vs. Incomplete

The idea is to have the AYAB seperated from the knittingpattern format. The knittingpattern format is an incomplete format that can be extended for any use case.  In contrast, the AYAB machine has a complete instruction set. The knittingpattern format is a means to transform these formats into different complete instruction sets. They should be convertible but not mixed.

Desciptive vs. Imperative

The idea is to be able to pass the format to the AYABInterface as a description. As much knowledge about the behavior is capsuled in the AYABInterface module. With this striving, we are less prone to intermix concerns across the applications.

Responsibilty Driven Design

I see these separated responsibilities:

  • A communication part focusing on the protocol to talk and the messages sent across the wire. It is an interpreter of the protocol, transforming it from bytes to objects.
  • A configuration that is passed to the interface
  • Different Machines types supported.
  • Actions the user shall perform.

Different Representations

I see these representations:

  • Commands are transferred across the wire. (PySerial)
  • For each movement of a carriage, the needles are used and put into a new position, B or D. (communication)
  • We would like to knit a list of rows with different colors. (interface)
    • Holes can be described by a list of orders in which meshes are moved to other locations, i.e. on needle 1 we can find mesh 1, on needle 2 we find mesh 2 first and then mesh 3, so mesh 2 and mesh 3 are knit together in the following step
  • The knitting pattern format.

Actions and Information for the User

The user should be informed about actions to take. These actions should not be in the form of text but rather in the form of an object that represents the action, i.e. [“move”, “this carriage”, “from right to left”]. This way, they can be adequately represented in the UI and translated somewhere central in the UI.

Summary

The new design separates concerns and allows testing. The bridge between the machine and the knittnigpattern format are primitive, descriptive objects such as lists and integers.

Transcript from the Python Toolbox 101

At the Python User Group Berlin, I lead a talk/discussion about free-of-charge tools for open-source development based on what we use GSoC. The whole content was in an Etherpad and people could add their ideas.

Because there are a lot of tools, I thought, I would share it with you. Maybe it is of use. Here is the talk:


Python Users Berlin 2016/07/14 Talk & Discussion

 

START: 19:15
Agenda 1min END: 19:15
======
– Example library
– What is code
– Version Control
  РPython Package Index
– …, see headings
– discussion: write down, what does not fit into my structure
Example Library (2min)  19:17
======================
What is Code (2min) 19:19
===================
.. note:: This frames our discussion
– Source files .py, .pyw
– tests
– documentation
– quality
– readability
– bugs and problems
– <3
Configurationsfiles plain Text for editing
Version Control (2min) 19:21
======================
.. note:: Sharing and Collaboration
– no Version Control:
  РDropbox
  РGoogle drive
  РTelekom cloud
  Рftp, windows share
– Version Control Tools:
  Рgit
    Рhttps://github.com
    Рhttps://gitlab.com
    Рgitweb own server
¬†¬†¬† –¬†
  Рmecurial
    Рhttps://bitbucket.org
  Рsvn
    Рhttp://sourceforge.net/
  Рperforce (proprietary)
  
  
  
  
  
  
Python Package Index (3min) 19:24
—————————
.. note:: Shipping to the users
hosts python packages you develop.
Example: “knittingpattern” package
pip
Installation from Pypi:
    $ python3 -m pip install knittingpattern # Linux
    > py -3.4 -m pip install knittingpattern # Windows
Documentation upload included!
Documentation (3min) 19:27
====================
.. note:: Inform users
I came across a talk:
Documentation can be:
– tutorials
– how to
– introduction to the community/development process
– code documentation!!!
– chat
–¬†
Building the documentation (3min)  19:30
———————————
Formats:
– HTML
– PDF
– reRST
– EPUB
– doc strings in source code
– test?
Tools:
– Sphinx
– doxygen
– doc strings
  Рstandard how to put in docstrings in Python
¬†¬†¬† –¬†
Example: Sphinx  3min 19:33
~~~~~~~~~~~~~~~
– Used for Python
– Used for knittingpattern
Python file:
Documentation file with sphinx.ext.autodoc:
Built documentation:
    See the return type str, Intersphinx can reference across documentations.
    Intersphinx uses objects inventory, hosted with the documentation:
Testing the documentation:
    РTODO: link
      Рevertying is included in the docs
      Рeverything that is public is documented
      
      syntax
      Рnumpy 
      Рgoogle 
      Рsphinx
Hosting the Documentation (3min) 19:36
——————————–
Tools:
– pythonhosted
  only latest version
– readthedocs.io
  several branches, versions, languages
– wiki pages
–¬†
Code Testing 2min 19:38
============
.. note:: Tests show the presence of mistakes, not their absence.
What can be tested:
– features
Рstyle: pep8, pylint, 
– documentation
– complexity
–¬†
Testing Features with unit tests 4min 19:42
——————————–
code:
¬†¬†¬† def fib(i): …
Tools with different styles
– unittest
  
    import unittest
    from fibonacci import fib
    class FibonacciTest(unittest.TestCase):
        def testCalculation(self):
            self.assertEqual(fib(0), 0)
            self.assertEqual(fib(1), 1)
            self.assertEqual(fib(5), 5)
            self.assertEqual(fib(10), 55)
            self.assertEqual(fib(20), 6765)
¬†¬†¬† if __name__ == “__main__”:¬†
        unittest.main()
 
– doctest
    import doctest
    def fib(n):
¬†¬†¬†¬†¬†¬†¬† “””¬†
        Calculates the n-th Fibonacci number iteratively  
        >>> fib(0)
        0
        >>> fib(1)
        1
        >>> fib(10) 
        55
        >>> fib(15)
        610
        >>> 
¬†¬†¬†¬†¬†¬†¬† “””
        a, b = 0, 1
        for i in range(n):
            a, b = b, a + b
        return a
¬†¬†¬† if __name__ == “__main__”:¬†
        doctest.testmod()
– pytest (works with unittest)
    import pytest
    from fibonacci import fib
    
¬†¬†¬† @pytest.mark.parametrize(“parameter,value”,[(0, 0), (1, 1), (10, 55), (15, 610)])
    def test_fibonacci(parameter, value):
        assert fib(parameter) == value
– nose tests?
– …
– pyhumber
Рassert in code,  PyHamcrest
– Behaviour driven development
  Рhuman test
Automated Test Run & Continuous Integration 2min 19:44
===========================================
.. note:: 
Several branches:
– production branch always works
– feature branches
– automated test before feature is put into production
Tools running tests 6min 19:50
——————-
– Travis CI for Mac, Ubuntu
– Appveyor for Windows
Host yourself:
– buildbot
– Hudson
– Jenkins
– Teamcity
– circle CI
  + selenium for website test
–¬†
– …?????!!!!!!
Tools for code quality 4min 19:54
———————-
– landscape
  complexity, style, documentation
  Рlibraries are available separately
    Рflake8
    Рdestinate
    Рpep257
– codeclimate
  code duplication, code coverage
  Рlibraries are available separately
– PyCharm
  Рintegrated what landscape has 
  Р+ complexity
Bugs, Issues, Pull Requests, Milestones 4min 19:58
=======================================
.. note:: this is also a way to get people into the project
1. find bug
2. open issue if big bug, discuss
3. create pull request
4. merge
5. deploy
– github
  issue tracker
                 http://www.appveyor.com/docs/branches#build-on-tags-github-and-gitlab-only
– waffle.io – scrumboard
  merge several github issues tracker
– Redmine
JIRA
Рtrac 
– github issues + zenhub integrated in github
– gitlab
– gerrit framework that does alternative checking https://www.gerritcodereview.com/
  1. propose change
  2. test
  3. someone reviews the code
      РX people needed
  QT company uses it
Localization 2min 20:00
============
crowdin.com
    Crowdsourced translation tool:
    
Discussion
– spellchecker is integrated in PyCharm
  Рcharacter set
  Рnew vocabulary
  Рnot for continuous integration (CI)
– Emacs
¬† –¬†
Рpylint plugin 
   Рnot all languages?
– readthedocs
  Рadd github project, 
  Рhosts docs
– sphinx-plugin?
– PyCon testing talk:
    РHypothesis package
      Рtries to break your code
¬†¬†¬†¬†¬† – throws in a lot of edge cases (huge number, nothing, …)
      -> find obscure edge cases
      
Did someone create a Pylint plugin
– question:
    Рcyclomatic code complexity
    Рwhich metrics tools do you know?
¬†¬†¬† –
Virtual Environment:
    nobody should install everything in the system
    -> switch between different python versions
    Рpython3-venv
      Рslightly different than virtual-env(more mature)
Beginners:
    Windows:
        install Anaconda

Deploying a Kivy Application with PyInstaller for Mac OSX with Travis CI to Github

In this sprint for the kniteditor library we focused on automatic deployment for Windows and Mac. The idea: whenever a tag is pushed to Github, a new travis build is triggered. The new built app is uploaded to Github as an “.dmg” file.

Travis

Travis is configured with the “.travis.yml” file which you can see here:

language: python

# see https://docs.travis-ci.com/user/multi-os/
matrix:
  include:
    - os: linux
      python: 3.4
    - os: osx
      language: generic
  allow_failures:
    - os: osx

install:
  - if [ "$TRAVIS_OS_NAME" == "osx"   ] ; then   mac-build/install.sh ; fi

script:
  - if [ "$TRAVIS_OS_NAME" == "osx"   ] ; then   mac-build/test.sh ; fi

before_deploy:
  - if [ "$TRAVIS_OS_NAME" == "osx"   ] ; then cp mac-build/dist/KnitEditor.dmg /Users/travis/KnitEditor.dmg ; fi

deploy:
  # see https://docs.travis-ci.com/user/deployment/releases/
  - provider: releases
    api_key:
      secure: v18ZcrXkIMgyb7mIrKWJYCXMpBmIGYWXhKul4/PL/TVpxtg2f/zfg08qHju7mWnAZYApjTV/EjOwWCtqn/hm2CfPFo=
    file: /Users/travis/KnitEditor.dmg
    on:
      tags: true
      condition:  "\"$TRAVIS_OS_NAME\" == \"osx\""
      repo: AllYarnsAreBeautiful/kniteditor

Note that it builds both Linux and OSX. Thus, for each step one must distinguish. Here, only the OSX parts are shown. These steps are executed:

  1. Installation. The app and dmg files are built.
  2. Testing. The tests are shipped with the app in our case. This allows us to execute them at many more locations – where the user is.
  3. Before Deploy. Somehow Travis did not manage to upload from the original location. Maybe it was a bug. Thus, a absolute path was created for the use in (4).
  4. Deployment to github. In this case we use an API key. One could also use a password.

Installation:

#!/bin/bash
#
# execute with --user to pip install in the user home
#
set -e

HERE="`dirname \"$0\"`"
USER="$1"
cd "$HERE"

brew update

echo "# install python3"
brew install python3
echo -n "Python version: "
python3 --version
python3 -m pip install --upgrade pip

echo "# install pygame"
python3 -m pip uninstall -y pygame || true
# locally compiled pygame version
# see https://bitbucket.org/pygame/pygame/issues/82/homebrew-on-leopard-fails-to-install#comment-636765
brew install sdl sdl_image sdl_mixer sdl_ttf portmidi
brew install mercurial || true
python3 -m pip install $USER hg+http://bitbucket.org/pygame/pygame

echo "# install kivy dependencies"
brew install sdl2 sdl2_image sdl2_ttf sdl2_mixer gstreamer

echo "# install requirements"
python3 -m pip install $USER -I Cython==0.23 \
                       --install-option="--no-cython-compile"
USE_OSX_FRAMEWORKS=0 python3 -m pip install $USER kivy
python3 -m pip uninstall -y Cython==0.23
python3 -m pip install $USER -r ../requirements.txt
python3 -m pip install $USER -r ../test-requirements.txt
python3 -m pip install $USER PyInstaller

./build.sh $USER

The first step is to update brew. It cost me 4 hours to find this bug, 2 hours to work around it before. If brew is not updated, Python 3.4 is installed instead of Python 3.5.

Then, Python, Pygame as the window provider for Kivy is installed, and the other requirements. It goes on with the build step. While installation is executed once on a personal Mac, the build step is executed several times, when the source code is changed.

#!/bin/bash
#
# execute with --user to make pip install in the user home
#
set -e

HERE="`dirname \"$0\"`"
USER="$1"
cd "$HERE"

(
  cd ..

  echo "# removing old installation of kniteditor"
  python3 -m pip uninstall -y kniteditor || true

  echo "# build the distribution"
  python3 -m pip install $USER wheel
  python3 setup.py sdist --formats=zip
  python3 setup.py bdist_wheel
  python3 -m pip uninstall -y wheel

  echo "# install the current version from the build"
  python3 -m pip install $USER --upgrade dist/kniteditor-`linux-build/package_version`.zip

  echo "# install test requirements"
  python3 -m pip install $USER --upgrade -r test-requirements.txt
)

echo "# build the app"
# see https://pythonhosted.org/PyInstaller/usage.html
python3 -m PyInstaller -d -y KnitEditor.spec

echo "# create the .dmg file"
# see http://stackoverflow.com/a/367826/1320237
KNITEDITOR_DMG="`pwd`/dist/KnitEditor.dmg"
rm -f "$KNITEDITOR_DMG"
hdiutil create -srcfolder dist/KnitEditor.app "$KNITEDITOR_DMG"

echo "The installer can be found in \"$KNITEDITOR_DMG\"."

In the first steps we install the kniteditor from the built “sdist”¬† zip file. This way we can uninstall it with pip. Also, the dependencies are installed. Then, PyInstaller is invoked with a spec and then the .dmg file is created.

The spec looks like this:

# -*- mode: python -*-

import sys
site_packages = [path for path in sys.path if path.rstrip("/\\").endswith('site-packages')]
print("site_packages:", site_packages)

from kivy.tools.packaging.pyinstaller_hooks import get_deps_all, \
    hookspath, runtime_hooks

block_cipher = None

added_files = [(site_packages_, ".") for site_packages_ in site_packages]

kwargs = get_deps_all()
kwargs["datas"] = added_files
kwargs["hiddenimports"] += ['queue', 'unittest', 'unittest.mock']


a = Analysis(['_KnitEditor.py'],
             pathex=[],
             binaries=None,
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             hookspath=hookspath(),
             runtime_hooks=runtime_hooks(),
             **kwargs)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          exclude_binaries=True,
          name='KnitEditorX',
          debug=False,
          strip=False,
          upx=True,
          console=True )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               name='KnitEditor')
app = BUNDLE(coll,
             name='KnitEditor.app',
             icon=None,
             bundle_identifier="com.ayab-knitting.KnitEditor")

Note, that all files in all site-packages are included. This way, we do not need to cope with missing modules. Also, there are three different names for

  • the entry script “_KnitEditor.py”
  • the executable “KnitEditorX”
  • the library “kniteditor”

While on the command line, OSX is case sensitive, it is not sensitive on the file system. Thus, if one of the names is the same, we can get errors durig the PyInstaller build.

Lessons learned

Do “brew update” on travis.

Use absolute paths for deployment on Mac OSX travis.

Never use the same names in PyInstaller for the main script, a library and the executable. Otherwise you get a “not a directory” or “not a file” error.

Travis OSX build max out from time to time. It is much faster to have a Mac computer there, to create the scripts.