Shell hacks

Custom Shell

Working with database models needs a lot of use of the flask shell. You can access it with:

python manage.py shell

The default shell is quite unintuitive. It doesn’t pretty print the outputs and has no support for auto-completion. I’ve installed IPython that takes care of that. But still working with models means writing a lot of import statements. Plus if there was some change in the code related to the app, then the shell had to restarted again so the changes could be loaded. Meaning writing the import statements again.

We were using Flask-Script and I wanted to run a custom shell that imports all the required modules, models and helper functions, so I don’t have to write them over and over. Some of them were as long as:

from open_event.models.users_events_roles import UsersEventsRoles

So I created a custom shell command with different context that overrides the default shell provided by the Flask-Script Manager. It was pretty easy with Flask-Script. One thing I had to keep in mind is that it needed to be in a different file than manage.py. Since manage.py was committed to source repo, changes to it would be tracked. So I needed a different file that could be excluded from the source repo. I created an smg.py that imported the Manager from the open_event module and overrides the shell command.

from flask_script import Shell

from open_event import manager
from open_event.models.user import User
from open_event.models.event import Event
from open_event.helpers.data import save_to_db, delete_from_db

def _make_context():
    return dict(
        uq=User.query,
        eq=Event.query,
        su=User.query.get(1),
        User=User,
        Event=Event,
        savetodb=save_to_db,
        deletefromdb=delete_from_db
    )


if __name__ == "__main__":
    manager.add_command('shell', Shell(make_context=_make_context))
    manager.run()

Place this smg.py file in the same directory as manage.py, so you can access it with python smg.py shell.

The code is pretty simple to understand. We import the Shell class from flask_script, create its object with our context and then add it to the manager as a command. _make_context contains what I usually like to have in my shell. It must always return a dictionary. The keys of this dictionary would be available as statements inside the shell with their values specified here.

This helps a lot. Most of the time I would be working with the super_admin user, and I would need its User object from time to time. The super_admin user is always going to be the first user (User object with id 1). So instead of from open_event.models.user import User; su = User.query.get(1) I could just use the su variable. Models like User and Event are also readily available (so are their base queries). This is the default context that I always keep, but many times you need more models than the ones specified here. Like when I was working with the permissions system.

from open_event.models.users_events_roles import UsersEventsRoles
from open_event.models.service import Service
from open_event.models.role import Role

def _make_context():
    return dict(
        # usual stuff
        UER=UsersEventsRoles,
        Service=Service,
        Role=Role
    )

You can even write a script that fetches all the database models (instance of sqlalchemy Model class) and then add them to the _make_context dictionary. I like to keep it minimum and differently named, so there are no conflicts of classes when try to auto-complete.

One more thing, you need to exclude the smg.py file so that git doesn’t track it. You can simply add it in the .git/info/exclude file.

I also wrote some useful one-liner bash commands.

Revision History

Every time someone updates the database models, he needs to migrate and provide the migrations file to others by committing it to the source. These files help us upgrade our already defined database. We work with Alembic. Regarding alembic revisions (migration files) you can keep two things in mind. One is the Head, that keeps track of the latest revision, and another is Current, that specifies what revision your tables in the database are based on. If your current revision is not a head, it means your database tables are not up-to-date. You need to upgrade (python manage.py db upgrade). The can be multiple heads in the revision history. python manage.py heads displays all of them. The current revision can be fetched with python manage.py current. I wanted something that automatically checks if current is at a head.

python manage.py db history | grep --color "$(python manage.py db current 2> /dev/null)|$(python manage.py db heads)|$"

You can specify it as an alias in .bashrc. This command displays the revision history with the head and current revision being colored.

Screenshot from 2016-07-11 16:34:18

You can identify the head as it is always appended by “(head)”. The other colored revision would be the current.

You can also reverse the output, by printing bottom to up. This is helpful when the revision history gets large and you have to scroll up to see the head.

alias rev='python manage.py db history | tac | grep --color "$(python manage.py db current 2> /dev/null)|$(python manage.py db heads)|$"'

Screenshot from 2016-07-11 16:42:30.png

Current Revision changer

The current revision is maintained at the database in the one-column alembic_version table. If for some reason the migration file for current revision no longer exists (like when changing to a branch at git that doesn’t have the file), alembic would raise an error. So to change the revision at the database I wrote a bash function.

change_db_current () {
    echo "Current Revision: "
    read current_rev
    echo "New Revision: "
    read new_rev

    echo 'c test \ update alembic_version set version_num='"'"$new_rev"'"' where version_num='"'"$current_rev"'"';' | sudo su - postgres -c psql
}

Enter the current revision hash (displayed in the alembic error) and the revision you need to point the current to, and it will update the table with the new revision. Sick right! It’s pretty stupid actually. It was when I didn’t know how to stamp in alembic. Anyways, it was useful some time ago.

Self-Contained Folder For Webapp

The first version of Open-event-webapp will be a generator that will create the web app.

This week I have worked on various OTS issues that will become the basis for the web app. The OpenTechSummit web app works along with Open-event-scraper.

The web app can be generated in any empty repository and can be hosted with gh-pages by just running build.sh  file from the Open-event-scraper. This is the build.sh file I have written for doing this.

build file
build.sh

 

How can you create the webapp from scraper in your repository ?

 

1 . Replace the URL of the repository in git clone command.

git clone – – depth=1 < destination repo url > ots-repo

2 . Run the file build.sh from  Open-event-scraper.

./build.sh

 

Utility for transferring content using build.sh

 

The build.sh file is written to make a folder that is self-sufficient. It means it can be taken anywhere and it should work on its own.

The build.sh file first creates a clone of the destination repository in the local Open-event-scraper. It itself make the required folders inside the cloned repository and runs generator.js which provides index.html file according to the template schedule.tpl. The resync command that is known as remote sync is used to transfer the files remotely. A programm folder is created automatically that includes all the CSS, JS,  JSON and other important files used to run the web app.

Finally, the programm folder along with all necessary files is pushed to gh-pages branch of the destination repository.

Working Example

 

To create the web app I have replaced the destination URL as shown in the image.

4

After running the build.sh we will get the output as shown :

1 .

8
Programm folder ( self-sufficient)

2.

9
Sub folders inside Programm folder

That’s how a folder is generated which contains all the necessary files needed to run the web app.

Enhancement of steam-shell plugin and Import from git

(ˢᵒᶜⁱᵉᵗʸserver) aims to be a platform for developing collaborative applications.
sTeam server project repository: sTeam.

Indentation of output made independent of the screen-width.

There were errors encountered in the pull request. These needed to be modified. The notable one’s were the Indentation of the output displayed by the look command in the steam-shell. The earlier display was taken into consideration that the default size of the linux terminal is 80 char long. However this can differ from individual to individual terminal settings. Therefore the module reference of pike was studied and a method to display the contents in the output of ls command format was written down. Thus,  now the output is independent of the screen width of an individual.

Issue. Github Issue Github PR
Indentation of output in steal-shell. Issue-24 PR-42

Indentation

Also there were minor errors in the edit.pike script. These were resolved too. The edit.pike was not able to exit the program. Therefore an exit(0) call was made. This resulted in the edit.pike being able to exit successfully but a new error was introduced in the steam-shell.pike. When the edit.pike command was called inside it, the steam-shell.pike would exit on successfully completing the edit command. The steam-shell should not exit in this case. There were changes made in the VisTeam.pike, edit.pike and applauncher.pike in order to get the utility working correctly.

Issue. Github Issue Github PR
Edit.pike hangs on closing Issue-29 PR-44
Edit command closes the steam-shell abruptly Issue-43 PR-44

The steam-shell vim plugin was modified and the support for multi line commands was added. Now the user can type the commands in multipe lines, format them and then execute them. The output will be displayed in a new tab.

Issue. Github Issue Github PR
Integrate sTeam-shell into Vi. Issue-37 PR-41

An example of it can be seen below:-

ListGates
Output:

OutputMult

The import-from-git.pike script is used to import contents from a normal git repository into the steam directory. The import from git script was studied and understood initially. The issue’s listed were replicated in the system and a possible solution for them was found out.

The script has support for importing multiple documents at the same time. Also the objects imported supported only text mime type. The script was modified to support other mime types by initially detecting them through an external process. However later in the server/factories/Document_factory.pike file the auto detecting of mime type was found out and used in the code. Now the mime types are detected and updated successfully for documents of other mime types like MP3, jpg, png etc

Issue. Github Issue Github PR
Auto-detect Mime type. Issue-18 PR-45

ImportMimeType

The contents of objects of types other than text mime type were not been read by the existing process. As a result this resulted into creation of empty objects with the set mime type. The process to read the contents was modified and the contents of objects of other types were set accordingly. The content is read in the form of a string. Since every object can be read as a string.

However in the import-from-git an issue was encountered. Indexing over Null value with set_contents for every object created by the script. On running the script again the earlier object is successfully imported but the same error is encountered over the next object to be imported. This issue would be looked down and tried to be solved. Also support for importing single objects will be provided.

Error1

 

Checkout the FOSSASIA Idea’s page for more information on projects supported by FOSSASIA.

More about sTeam’s command line interface

This post will take you through some interesting work that I have been doing for my project under FOSSASIA. This project is being done under the google summer of code platform. I have been working on various code scripts as my project is all about improving the tools of the sTeam collaboration platform. But this particular script is a command line interface that I am thrilled to work on.

If you are not familiar with what sTeam is, you can look it up here, sTeam

Let me start with how sTeam originally works for any user. This picture will give you a clear view about it.

 

If you are new to the sTeam platform, this script will take you through the sTeam commands and the interface, also making sure, you have fun in the process. It lets the user play with sTeam and get to know more about how it works. One other interesting concept that we have tried to integrate here is that of MUDs(Multi user dungeons/dimensions). Some of you may already be familiar with the concept of MUDs( others, please look it up here MUD). We have tried to bring the MUD kind of experience to this sTeam interface (like I said, thrilling!).

I have seen people go through a lot of tutorials on git, and the commands(sometimes complex) through which it operates, and yet, people have difficulty learning it. What if it had a MUD type of interaction, which would just let you type “create file” “commit file” “get commit content” “upload file” “rename file” “goto repo”, etc. Wouldnt it be much easier that way? Well, sTeam is going through all the extensive tool development so that it can make a user-friendly interaction. Here are some screenshots of this interface,

 

steamshell2.png

steamshell3.png

steamshell_edit.png

 

This interface is under heavy development and plans to make sTeam more and more interesting for its users. If you have any questions on sTeam, please feel free to contact me, or join us on our daily scrum meetings on IRC (#steam-devel) or join #fossasia and ping us with a question.

If you are looking for a open source project and want to contribute to FOSSASIA, Please have a look at FOSSASIA labs.

Improve the command line tools of the sTeam collaboration platform

First of all, I really thank Google for organizing such an event for students in the summer and I am fortunate to be a part of such a program. I would also like to thank FOSSASIA for accepting my proposal for the project and for providing me with the experience of working with them.

When you see your name on the list

Name on the list? Google’s server being stacked up with lots and lots of request, Students impatiently clicking the refresh button to see whether their names are present in the list, a lot of students, a lot of projects, and suddenly, that one refresh, when the list loads with the projects matching the student names. (CTRL+F) ”Trilok Tourani” and there it was,

  • Organization : FOSSASIA
  • Project : Improve the command line tools of the sTeam collaboration platform
  • Student : Trilok Tourani
  • Mentors : Martin Bahr(Working mentor) , Chris Angelico, Aruna Herath, Markus(Backup mentors)
  • Status : Accepted

Dumbstruck for a minute, but when it hit me, that I got selected, the happiness was beyond what I had expected. I went on the IRC and thanked my mentors. All I knew was I had a project in hand and a long way ahead to perform beyond their expectations.


About the project

What is sTeam?

sTeam is a collaboration platform which helps people to share their documents, chat with them, have a look at their virtual workarea, and ease the sharing/developing in big groups. You can also look at the current development on the sTeam web interface here. It is still in the development phase, so please feel free to provide us with any ideas to improve. To know more about sTeam, please visit societyserver.

sTeam is completely built on pike programming language. To know more about pike programming, please visit Pike. If you already know about pike, and are good at it, please visit Fossasia’s IRC channel #fossasia and ask for tasks/bugs on this project to solve right away. If you are looking for something other than sTeam, FOSSASIA has a wide range of projects in various fields, and also welcomes any new project ideas.


My part of the project

I would like people to know the kind of work that I am doing currently for sTeam and show them how interesting it is to develop for it.

My project is to develop the command line tools to ease the work for the developers, and people who choose to work with command line, over a web interface.

These tools include,

  • Exporting your documents to your Github repository with just a command.
  • Importing document from Github right into sTeam for people to see.
  • A easy to use debug client that helps you while developing for this platform, or to see the various documents/containers you have in which room, move them, copy them, use them, all with commands similar to the Linux command line.
  • A way of to edit your documents directly with command line editors, without having to go to the web, type in the url, clicking on a document, and then finally editing it.
  • Chat with your friends/group members over the IRC.
  • and more being added to the list….(Please contact us if you have any)

These tools are already built up, but with more development, they will be very efficient and easy to use commands which a normal user can type in and get things done, and this is where my part comes in.

For some of these tools, which are already built, some development is needed. While some other tools, I have to develop by myself and enhance the command line usage for sTeam.

These are few of the commands with their screenshots to help you understand exactly what these tools are meant for,

The sTeam debug client

The editing documents client,

and edit the document in a simple command-line editor (used vim here),

Exporting documents from sTeam to git (version-wise)

Importing documents to sTeam from git (version-wise)

If you really like what I am doing, or are interested in developing for sTeam, please join our IRC channel #fossasia or join [email protected] .

If you are looking for some other project, FOSSASIA has a lot of projects to be worked on. Please visit FOSSASIA for more info.