open event server installation and docker-compose

Installing Open Event Server 

I’m going to walk you through the process of installing Open Event Server and possible issues you might encounter, this will probably be helpful if you’re stuck with it, but most importantly in building some sort of installer as I’ve stated numerous times before.

Requirements for its installation

Debian based distro or unix-like with aptitude package manager

Enough memory and storage, (depends on whether you put the database on your system or not)

Knowledge of UNIX-like systems

First, here are the commands, I tried to make them as easy as a copy-paste but it probably won’t work depending on it, so I’ll talk to you about what each thing does

apt-get update
apt-get install -y locales git sudo
locale-gen en_US.UTF-8
if [ -z "$LANG" ];then export LANG=en_US.UTF-8;fi

export DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true
git clone https://github.com/kreijstal-contributions/open-event-api.git --depth=1
cd open-event-server
#libffi6 libffi-dev
apt-get install -y redis-server tmux  libssl-dev vim postgresql postgresql-contrib build-essential python3-dev libpq-dev libevent-dev libmagic-dev python3-pip wget ca-certificates python3-venv curl && update-ca-certificates && apt-get clean -y

service redis-server start
service  postgresql start
cp .env.example .env
#sed -i -e 's/SERVER_NAME/#SERVER_NAME/g' .env
#The above was used because before the repository came with an annoying SERVER_NAME which only listened to localhost, so if you accessed with 127.0.0.1, it didn’t work
#pip install virtualenv
#python 2.7.14
#virtualenv .

python3 -m venv .
#pip install pip==9.0.1
source bin/activate
#pip3 install -r requirements/tests.txt
pip3 install --no-cache-dir -r requirements.txt
#pip install eventlet
cat << EOF | su - postgres -c psql
-- Create the database user:
CREATE USER john WITH PASSWORD 'start';

-- Create the database:
CREATE DATABASE oevent OWNER=john
                                  LC_COLLATE='en_US.utf8'
                                  LC_CTYPE='en_US.utf8'
                                  ENCODING='UTF8'

TEMPLATE=template0;
EOF
python3 create_db.py admin@admin.com admin
python3 manage.py db stamp head
python3 manage.py runserver 

 

Ok, let’s start with the beginning, we assume here you know how to open a linux terminal, and that you’re on linux.

Description of commands

Updating the repo

apt-get update

Ok, so let’s start with some assumptions, you’re either on ubuntu, or debian, apt-get is on every debian-based distro, so you can do this on linux-mint as well, but if you’re using other distro you might have to consult with your package manager guide where to get the packets that are needed.

Unfortunately the open-event-server has dependencies and in order to get them we’ve used the package manager of ubuntu, that means some packages might not be possible to be found on every distro, and there is currently not support for that.

Installing git and sudo

apt-get install -y locales git sudo

Ok, let’s also assume you’re on a barebones ubuntu installation, as many people are so let’s work on some very basic requirements that you probably already fulfil, git is not installed by default, so here we install it, we also install sudo, and the -y flag is to avoid being asked for confirmation.

Setting the locales

locale-gen en_US.UTF-8
if [ -z "$LANG" ];then export LANG=en_US.UTF-8;fi

Okay so basically some very bare bones linux distributions don’t have the UTF-8 locale enabled, here we generate it in case it hasn’t, and if the $LANG variable doesn’t exist, then we add the en_US locale.

Really ensuring it works everywhere, especially because once you start installing packages you might get prompted or errors might come up
Cloning the repo

Explanation

git clone https://github.com/fossasia/open-event-server.git --depth=1

Here we clone the repo with git clone from github.

However we use the –depth=1 flag, that means we only and just only get the latest version, git was designed as a version control tool, not as a source host, so when you clone a project with the purpose of installing it, you’re downloading the whole revision history, and some revision histories can get pretty, pretty large, slowing the download time.

If you’re a contributor and you want to revert some git commits, or you want to debug and see what causes something, then feel free to download everything, but if you’re downloading the git with the purpose to use the software, you really don’t need to download the revision history.

cd open-event-server

A bit of trivia for those who didn’t know, cd means “Change Directory”, here we’re changing our current directory to open-event-server

#libffi6 libffi-dev
apt-get install -y redis-server tmux  libssl-dev vim postgresql postgresql-contrib build-essential python3-dev libpq-dev libevent-dev libmagic-dev python3-pip wget ca-certificates python3-venv curl && update-ca-certificates && apt-get clean -y


Okay, here we use apt-get to install redi-server, which is a dependency for open-event, and postgresql, we also download python3 plus some important packets like virtualenv for python3.

Starting services

In this specific case we’re installing these servers on our server but it doesn’t necessarily have to be that way, if you have an external postgres or redis server you can use that as well.
service redis-server start

We start the redis-server

service  postgresql start

We start the postgres service

cp .env.example .env

Virtual python enviroments

python3 -m venv .

We initialize the virtualenv on the current directory, that is .

source bin/activate

We activate the virtualenv generated by .venv

pip3 install --no-cache-dir -r requirements.txt

Postgres

cat << EOF | su - postgres -c psql
-- Create the database user:
CREATE USER john WITH PASSWORD 'start';

-- Create the database:
CREATE DATABASE oevent OWNER=john
                                 LC_COLLATE='en_US.utf8'
                                 LC_CTYPE='en_US.utf8'
                                 ENCODING='UTF8'
TEMPLATE=template0;
EOF

A very clever use of command line pipes, basically psql is the command you use to interact with the database, but it is usually interactive, however on the command line you can very easily pipe typed commands using cat, this is equivalent as if we executed su – postgres -c psql and then typed everything until EOF, so next time you need to automate something that prompts you from input, you can use this!

Also psql won’t generally accept connections from anyone buy postgres, even if you’re root that’s why we’re using su – postgres -c psql  which means, run psql as user postgres, you don’t need to use su, if you know how to change users with sudo

Open event

python3 create_db.py admin@admin.com admin
python3 manage.py db stamp head
python3 manage.py runserver

Using docker-compose install both versions

I’ll explain now, how to do it using docker-compose, which concurrently boots both server and frontend.

git clone https://github.com/Kreijstal/oevent-docker-compose.git
cd oevent-docker-compose
bash clone.sh
docker-compose build

docker-compose run v1 python create_db.py

You will be prompted to add admin account for the v1 server. After you do that just do

docker-compose up

(v2 log in settings you can change it in docker-compose.yml)
And that’s it! Way faster than the other way, requires even less code, and almost always succeed), the catch is that you have to have docker installed in your machine

Continue Readingopen event server installation and docker-compose

Get My Badges from Badgeyay API

Badgeyay is no longer a simple badge generator. It has more cool features than before.

Badgeyay now supports a feature that shows your badges. It is called ‘my-badges’ component. To get this component work, we need to design a backend API to deliver the badges produced by a particular user.

Why do we need such an API?

The main aim of Badgeyay has changed from being a standard and simple badge generator to a complete suite that solves your badge generation and management problem. So to tackle the problem of managing the produced badges per user, we need to define a separate route and schema that delivers the generated badges.

Adding the functionality to backend

Let us see how we implemented this functionality into the backend of the project.

Step 1 : Adding a route

This step involves adding a separate route that provides with the generated output of the badges linked with the user account.

@router.route(‘/get_badges’, methods=[‘GET’])
def get_badges():
input_data = request.args
user = User.getUser(user_id=input_data.get(
‘uid’))
badges = Badges().query.filter_by(creator=user)
return jsonify(UserBadges(many=True).dump(badges).data)

This route allows us to get badges produced by the user as a JSON API data object. This object is fed to the frontend to render the badges as cards.

Step 2 : Adding a relevant Schema

After creating a route we need to add a relevant schema that will help us to deliver the badges generated by the user to the Ember JS frontend so that it can be consumed as JSON API objects and shown to the user.

class UserBadges(Schema):
class Meta:
type_ =
‘user-badges’
self_view =
‘generateBadges.get_badges’
kwargs = {
‘id’: ‘<id>’}

id = fields.Str(required=True, dump_only=True)
image = fields.Str(required=
True)
csv = fields.Str(required=
True)
badge_id = fields.Str(required=
True)
text_color = fields.Str(required=
True)
badge_size = fields.Str(required=
True)
user_id = fields.Relationship(
self_url=
‘/api/upload/get_file’,
self_url_kwargs={
‘file_id’: ‘<id>’},
related_url=
‘/user/register’,
related_url_kwargs={
‘id’: ‘<id>’},
include_resource_linkage=
True,
type_=
‘User’
)

This is the ‘UserBadge’ schema that produces the output results of the GET request on the route.

Finally, once this is done we can fire up a GET request on our deployment to receive results. The command that you need to run is given below.

$ ~ curl -X GET http://localhost:5000/api/get_badges?uid={user_id}

Further Improvements

We are working on adding multiple routes and adding modifications to database models and schemas so that the functionality of Badgeyay can be extended to a large extent. This will help us in making this badge generator even better.

Resources

 

Continue ReadingGet My Badges from Badgeyay API

Implementing User Stats for SUSI.AI Admin Panel

SUSI.AI has an admin panel where users with roles operator or above get an overview of various stats and manage other users and skills. The admin panel allows the system admins to get a list of all users registered on susi and also provide an option to change their user roles as well. The admin tab provides us with the statistics of its users. In this post we will discuss how this is implemented on susi server.

Continue ReadingImplementing User Stats for SUSI.AI Admin Panel

Custom Colored Images with Badgeyay

Backend functionality of any Badge generator is to generate badges as per the requirements of the user. Currently Badgeyay is capable of generating badges by the following way:

  • Adding or Selecting a Pre-defined Image from the given set
  • Uploading a new image and then using it as a background

Well, badgeyay has been missing a functionality of generating Custom Colored images.

What is meant by Custom Colored Badges?

Currently, there are a set of 7 different kind of pre-defined images to choose from. But let’s say that a user want to choose from the images but doesn’t like any of the color. Therefore we provide the user with an additional option of applying custom background-color for their badges. This allows Badgeyay to deliver a more versatile amount of badges than ever before.

Adding the functionality to backend

Lets see how this functionality has been implemented in the backend of the project.

Step 1 :  Adding a background-color route to backend

Before generating badges, we need to know that what is the color that the user wants on the badge. Therefore we created a route that gathers the color and saves the user-defined.svg into that particular color.

@router.route(‘/background_color’, methods=[‘POST’])
def background_color():
try:
data = request.get_json()[‘data’][‘attributes’]
bg_color = data[‘bg_color’]
except Exception:
return ErrorResponse(PayloadNotFound().message, 422, {‘Content-Type’: ‘application/json’}).respond()

svg2png = SVG2PNG()

bg_color = ‘#’ + str(bg_color)
user_defined_path = svg2png.do_svg2png(1, bg_color)
with open(user_defined_path, “rb”) as image_file:
image_data = base64.b64encode(image_file.read())
os.remove(user_defined_path)

try:
imageName = saveToImage(imageFile=image_data.decode(‘utf-8’), extension=”.png”)
except Exception:
return ErrorResponse(ImageNotFound().message, 422, {‘Content-Type’: ‘application/json’}).respond()

uid = data[‘uid’]
fetch_user = User.getUser(user_id=uid)
if fetch_user is None:
return ErrorResponse(UserNotFound(uid).message, 422, {‘Content-Type’: ‘application/json’}).respond()

file_upload = File(filename=imageName, filetype=’image’, uploader=fetch_user)
file_upload.save_to_db()
return jsonify(ColorImageSchema().dump(file_upload).data)

Step 2: Adding Schema for background-color to backend

To get and save values from and to database, we need to have some layer of abstraction and so we use schemas created using marshmallow_jsonapi

class ColorImageSchema(Schema):
class Meta:
type_ = ‘bg-color’
self_view = ‘fileUploader.background_color’
kwargs = {‘id’: ‘<id>’}

id = fields.Str(required=True, dump_only=True)
filename = fields.Str(required=True)
filetype = fields.Str(required=True)
user_id = fields.Relationship(
self_url=’/api/upload/background_color’,
self_url_kwargs={‘file_id’: ‘<id>’},
related_url=’/user/register’,
related_url_kwargs={‘id’: ‘<id>’},
include_resource_linkage=True,
type_=’User’
)

Now we have our schema and route done, So we can move forward with the logic of making badges.

Step 3 : Converting the SVG to PNG and adding custom color

Now we have the user-defined color for the badge background, but we still need a way to apply it to the badges. It is done using the following code below.

def do_svg2png(self, opacity, fill):
“””
Module to convert svg to png
:param `opacity` – Opacity for the output
:param `fill` –  Background fill for the output
“””
filename = os.path.join(self.APP_ROOT, ‘svg’, ‘user_defined.svg’)
tree = parse(open(filename, ‘r’))
element = tree.getroot()
# changing style using XPath.
path = element.xpath(‘//*[@id=”rect4504″]’)[0]
style_detail = path.get(“style”)
style_detail = style_detail.split(“;”)
style_detail[0] = “opacity:” + str(opacity)
style_detail[1] = “fill:” + str(fill)
style_detail = ‘;’.join(style_detail)
path.set(“style”, style_detail)
# changing text using XPath.
path = element.xpath(‘//*[@id=”tspan932″]’)[0]
# Saving in the original XML tree
etree.ElementTree(element).write(filename, pretty_print=True)
print(“done”)
png_name = os.path.join(self.APP_ROOT, ‘static’, ‘uploads’, ‘image’, str(uuid.uuid4())) + “.png”
svg2png(url=filename, write_to=png_name)
return png_name

Finally , we have our badges generating with custom colored background.

Here is a sample image:

Resources

 

Continue ReadingCustom Colored Images with Badgeyay

How to Change a Password and Forgot Password Feature in the SUSI.AI Server

The accounting system of SUSI.AI provides its users the option to change the password of their accounts. This features gives us two options, either we can change our password by entering the older password or if the user forgot the password, they are provided a link on their email address through which we can change our password. Using either option, the user has to authenticate themselves before they can actually change their passwords. If the user has the current password, it is considered as a parameter of authentication. In other case the user has to check their email account for the link which also confirms the authenticity of user. In this post we will discuss how both options works on SUSI.

Continue ReadingHow to Change a Password and Forgot Password Feature in the SUSI.AI Server

Youtube search as a Console Service Endpoint in SUSI.AI

SUSI.AI now has the ability to search and play any song or video directly in the webclient and the speaker. When a user asks for a query regarding playing a song, the clients sends a search request to the server. In this post I will discuss about the server side implementation of the Youtube search. Every time a request is made by any client, the client sends a query request to the server in the form of a json object. For more on the working on the webclient side can be seen here.

The endpoint for youtube search is  http://api.susi.ai/susi/console.json

Continue ReadingYoutube search as a Console Service Endpoint in SUSI.AI

Delete User Account Service using LoginService.java API in SUSI.AI

SUSI.AI has an api to handle all account related services called accounts.susi.ai. This API provides services like change password and delete account. the main goal of accounts.susi.ai is to centralise the accounts’ related settings from web, android and iOS clients into a single place like other accounting services. In this post we will discuss one of these features which is the delete user account from SUSI.AI.

The api accounts.susi.ai has a react file DeleteAccount.react.js. This file handle the delete account service and interacts with the LoginServce.java api, which is the core file for authentication. I will discuss in details how these two interact with each other and with other files.

Continue ReadingDelete User Account Service using LoginService.java API in SUSI.AI

Drafts of SUSI Chatbots

While creating a SUSI bot using bot wizard, a user has the option to save a draft of the bot at any step. Then he/she can continue building the bot from there. We can do four operations on a draft: Storing, fetching, editing and deleting.

Storing a draft:

For the storage of drafts, a GET request is made to the storeDraft.json API. The code of chatbot is passed in ‘object’ parameter. This code must be in JSON format. A sample API call looks like this:

https://api.susi.ai/cms/storeDraft.json?object={"test":"123"}

The JSON passed in ‘object’ has the following format:

{
   “group”: /*group_here*/
   “language”: /*language_here*/
   “name”: /*bot_name_here*/
   “buildCode”: /*code_in_build_tab_here*/
   “designCode”: /*code_in_design_tab_here*/
   “configCode”: /*code_in_config_tab_here*/
}

While storing the draft, we can either specify an ID ourselves as a parameter named ‘id’ in the API call url or we can get a randomly generated id from the server itself.
The hex codes in designCode includes ‘#’ which can not be passed in the url. Hence, we remove it while saving a draft and then add it back while editing it.

Fetching a draft:

For fetching drafts, a GET request is made to the readDraft.json API. We can either simply call the API without any other parameters which will return all the chatbots of the user. The API call for this looks like this:

https://api.susi.ai/cms/readDraft.json

We can fetch a particular draft by specifying the ID of draft in the API call like this:

https://api.susi.ai/cms/readDraft.json?id=abcdef12

Editing a draft:

The drafts are displayed on the botbuilder page. From there a user can open a draft simply by clicking on it. This uses the ID of that draft to fetch its details by making a GET request to readDraft.json API as specified above.
For applying the details fetched to bot wizard, we pass the draft ID in url of bot wizard as “/botbuilder/botwizard?draftID=abcdef12”.  Then we fetch this ID using “this.getQueryStringValue(‘draftID’)” and this is followed by fetching all details of bot.
After fetching details of the chatbot, all the details are updated in the state of Botwizard component and hence user can continue making SUSI bot.

Deleting a draft:

For the deletion of drafts, a GET request is made to the deleteDraft.json API. We need to specify the id of the draft that we want to delete in the ‘id’ parameter. So, the API call looks like this:

https://api.susi.ai/cms/deleteDraft.json?id=abcdef12

Resources

Continue ReadingDrafts of SUSI Chatbots

Fetching Bots from SUSI.AI Server

Public skills of SUSI.AI are stored in susi_skill_cms repository. A private skill is different from a public skill. It can be viewed, edited and deleted only by the user of who created it. The purpose of private skill is that this acts as a chatbot. All the information of the bot like design, configuration etc is stored within the private skill itself. In order to show the lists of chatbots a user has created to him/her, we need to fetch these lists from the server. The API for fetching private skills is in ListSkillService.java servlet. This servlet is used for both the purposes for fetching public skills and private skills.

How are bots fetched?

All the private skills are stored in data/chatbot/chatbot.json location in the SUSI server. Hence, we can fetch these skills from here.
The API call to fetch private skills is:

https://api.susi.ai/cms/getSkillList.json?private=1&access_token=accessTokenHere

The API call has two parameters:

  1. private = 1: This parameter tells the servlet that we’re asking for private skills because same API is used for fetching public skills as well.
  2. access_token : We have to pass the access-token in order to authenticate the user and access the chatbots of the specific user because the chatbots are stored according to user id of the users in chatbot.json. To get the user id, we need access-token.

For fetching the chatbots from chatbot.json, we firstly authenticate the user. Then we check if the user has any chatbots or not. If the user has chatbots then we loop through chatbot.json to find the chatbots of user by using user id. After getting chatbots details, we simply present it in json structure. You can easily understand this through the following code snippets. For authenticating:

String userId = null;
String privateSkill = call.get("private", null);
if (call.get("access_token", null) != null) {
    ClientCredential credential = new ClientCredential(ClientCredential.Type.access_token, call.get("access_token", null));
    Authentication authentication = DAO.getAuthentication(credential);
    // check if access_token is valid
    if (authentication.getIdentity() != null) {
        ClientIdentity identity = authentication.getIdentity();
        userId = identity.getUuid();
    }
}

You can see that the parameter private is stored in privateSkill. We check if privateSkill is not null followed by checking if the user id provided is valid or not. When both the checks are successful, we create a JSON object in which we store all the bots. To do this, we loop through the chatbot.json file to find the user id provided in API call. If the user id doesn’t exist in chatbot.json file then a message “User has no chatbots.” is displayed. If the user id is found then we again loop through all the chatbots and put their name as the value of key “name”,  language as the value of key “language” and group as the value of key “group”. All these key value pairs for the chatbots are pushed in an array and finally that array is added to the JSON object. This JSON is then received as response of the API call. The following code will demonstrate this:

if(privateSkill != null) {
    if(userId != null) {
        JsonTray chatbot = DAO.chatbot;
        JSONObject result = new JSONObject();
        JSONObject userObject = new JSONObject();
        JSONArray botDetailsArray = new JSONArray();
        JSONArray chatbotArray = new JSONArray();
        for(String user_id : chatbot.keys())
        {
            if(user_id.equals(userId)) {
                userObject = chatbot.getJSONObject(user_id);
                Iterator chatbotDetails = userObject.keys();
                List<String> chatbotDetailsKeysList = new ArrayList<String>();
                while(chatbotDetails.hasNext()) {
                    String key = (String) chatbotDetails.next();
                    chatbotDetailsKeysList.add(key);
                }
                for(String chatbot_name : chatbotDetailsKeysList)
                {
                    chatbotArray = userObject.getJSONArray(chatbot_name);
                    for(int i=0; i<chatbotArray.length(); i++) {
                        String name = chatbotArray.getJSONObject(i).get("name").toString();
                        String language = chatbotArray.getJSONObject(i).get("language").toString();
                        String group = chatbotArray.getJSONObject(i).get("group").toString();
                        JSONObject botDetails = new JSONObject();
                        botDetails.put("name", name);
                        botDetails.put("language", language);
                        botDetails.put("group", group);
                        botDetailsArray.put(botDetails);
                        result.put("chatbots", botDetailsArray);
                    }
                }
            }
        }
        if(result.length()==0) {
            result.put("accepted", false);
            result.put("message", "User has no chatbots.");
            return new ServiceResponse(result);
        }
        result.put("accepted", true);
        result.put("message", "All chatbots of user fetched.");
        return new ServiceResponse(result);
    }
}

So, if chatbot.json contains:

{"c9b58e182ce6466e413d5acafae906ad": {"chatbots": [
 {
   "name": "testing111",
   "language": "en",
   "group": "Knowledge"
 },
 {
   "name": "DNS_Bot",
   "language": "en",
   "group": "Knowledge"
 }
]}}

Then, the JSON received at http://api.susi.ai/cms/getSkillList.json?private=1&access_token=JwTlO8gdCJUns569hzC3ujdhzbiF6I is:

{
  "session": {"identity": {
    "type": "email",
    "name": "test@whatever.com",
    "anonymous": false
  }},
  "chatbots": [
    {
      "name": "testing111",
      "language": "en",
      "group": "Knowledge"
    },
    {
      "name": "DNS_Bot",
      "language": "en",
      "group": "Knowledge"
    }
  ],
  "accepted": true,
  "message": "All chatbots of user fetched."
}

References:

Continue ReadingFetching Bots from SUSI.AI Server

How User preferences data is stored in Chat.susi.ai using Accounting Model

Like any other accounting services SUSI.AI also provides a lot of account preferences. Users can select their language, timezone, themes, speech settings etc. This data helps users to customize their experience when using SUSI.AI.

In the web client these user preferences are fetch from the server by UserPreferencesStore.js and the user identity is fetched by UserIdentityStore.js. These settings are then exported to the settings.react.js file. This file is responsible for the settings page and takes care of all user settings. Whenever a user changes a setting, it identifies the changes and show an option to save these changes. These changes are then updated on the server using the accounting model of SUSI.AI. Let’s take a look at each file discussed above in detail.

Continue ReadingHow User preferences data is stored in Chat.susi.ai using Accounting Model