Automatically deploy SUSI Web Chat on surge after Travis passes

We are using surge from the very beginning of this SUSI web chat and SUSI skill cms projects development. We used surge for provide preview links for Pull requests. Surge is really easy tool to use. We can deploy our static web pages really easily and quickly.  But If user had to change something in pull request user has to deploy again in surge and update the link. If we can connect this operation with travis ci we can minimise re-works. We can embed the deploying commands inside the travis.yml. We can tell travis to make a preview link (surge deployment) if test cases are passed by embedding the surge deployment commands inside the travis.yml like below. This is travis.yml file sudo: required dist: trusty language: node_js node_js: - 6 script: - npm test after_success: - bash ./surge_deploy.sh - bash ./deploy.sh cache: directories: - node_modules branches: only: - master Surge deployment commands are inside the “surge_deploy.sh” file. In that we have to check the status of the pull request whether it is passing test cases or not. We can do it like below. if [ "$TRAVIS_PULL_REQUEST" == "false" ]; then echo "Not a PR. Skipping surge deployment" exit 0 fi Then we have to install surge in the environment. Then after install all npm packages and run build. npm i -g surge npm install npm run build Since there is a issue with displaying moving to child routes we have to take a copy of index.html file and name it as a 404.html. cp ./build/index.html ./build/404.html Then make two environment variables for your surge email address and surge token export SURGE_LOGIN=fossasiasusichat@example.com # surge Token (run ‘surge token’ to get token) export SURGE_TOKEN=d1c28a7a75967cc2b4c852cca0d12206 Now we have to make the surge deployment URL (Domain). It should be unique so we made a URL that contains pull request number. export DEPLOY_DOMAIN=https://pr-${TRAVIS_PULL_REQUEST}-susi-web-chat.surge.sh surge --project ./build/ --domain $DEPLOY_DOMAIN; Since all our static contents which made after the build process are in “build” folder we have to tell surge to get static html files from that. Now make a pull request. you would find the deployment link in travis ci report after travis passed. Expand the output of the surge_deploy.sh You will find the deployment link as we defined in the surge_deploy.sh file References: Integrating with travis ci - https://surge.sh/help/integrating-with-travis-ci React Routes to Deploy 404 page on gh-pages and surge - https://blog.fossasia.org/react-routes-to-deploy-404-page-on-gh-pages-and-surge/

Continue ReadingAutomatically deploy SUSI Web Chat on surge after Travis passes

Improving Loklak apps site

In this blog I will be describing some of the recent improvements made to the Loklak apps site. A new utility script has been added to automatically update the loklak app wall after a new app has been made. Invalid app query in app details page has been handled gracefully. A proper message is shown when a user enters an invalid app name in the url of the details page. Tests has been added for details page. Developing updatewall script This is a small utility script to update Loklak wall in order to expose a newly created app or update an existing app. Before moving into the working of this script let us discuss how Loklak apps site tracks all the apps and their details. In the root of the project there is a file names apps.json. This file contains an aggregation of all the app.json files present in the individual apps. Now when the site is loaded, index.html loads the Javascript code present in app_list.js. This app_list.js file makes an ajax call to root apps.json files, loads all the app details in a list and attaches this list to the AngularJS scope variable. After this the app wall consisting of various app details is rendered using html. So whenever a new app is created, in order to expose the app on the wall, the developer needs to copy the contents of the application’s app.json and paste it in the root apps.json file. This is quite tedious on the part of the developer as for making a new app he will first have to know how the site works which is not all directly related to his development work. Next, whenever he updates the app.json of his app, he needs to again update apps.json file with the new data. This newly added script (updatewall) automates this entire process. After creating a new app all that the developer needs to do is run this script from within his app directory and the app wall will be updated automatically. Now, let us move into the working of this script. The basic workflow of the updatewall script can be described as follows. The script loads the json data present in the app.json file of the app under consideration. Next it loads the json data present in the root apps.json file. if __name__ == '__main__': #open file containg json object json_list_file = open(PATH_TO_ROOT_JSON, 'r') #load json object json_list = json.load(json_list_file, object_pairs_hook=OrderedDict) json_list_file.close() app_json_file = open(PATH_TO_APP_JSON, 'r') app_json = json.load(app_json_file, object_pairs_hook=OrderedDict) app_json_file.close() #method to update Loklak app wall expose_app(json_list, app_json) When we are loading the json data we are using object_pairs_hook in order to load the data into an OrderedDict rather than a normal python dictionary. We are doing this so that the order of the dictionary items are maintained. Once the data is loaded we invoke the expose method. def expose_app(json_list, app_json): #if app is already present in list then fetch that app app = getAppIfPesent(json_list, app_json) #if app is not present then add…

Continue ReadingImproving Loklak apps site

Using Protractor for UI Tests in Angular JS for Loklak Apps Site

Loklak apps site’s home page and app details page have sections where data is dynamically loaded from external javascript and json files. Data is fetched from json files using angular js, processed and then rendered to the corresponding views by controllers. Any erroneous modification to the controller functions might cause discrepancies in the frontend. Since Loklak apps is a frontend project, any bug in the home page or details page will lead to poor UI/UX. How do we deal with this? One way is to write unit tests for the various controller functions and check their behaviours. Now how do we test the behaviours of the site. Most of the controller functions render something on the view. One thing we can do is simulate the various browser actions and test site against known, accepted behaviours with Protractor. What is Protractor Protractor is end to end test framework for Angular and AngularJS apps. It runs tests against our app running in browser as if a real user is interacting with our browser. It uses browser specific drivers to interact with our web application as any user would. Using Protractor to write tests for Loklak apps site First we need to install Protractor and its dependencies. Let us begin by creating an empty json file in the project directory using the following command. echo {} > package.json Next we will have to install Protractor. The above command installs protractor and webdriver-manager. After this we need to get the necessary binaries to set up our selenium server. This can be done using the following. ./node_modules/protractor/bin/webdriver-manager update ./node_modules/protractor/bin/webdriver-manager start Let us tidy up things a bit. We will include these commands in package.json file under scripts section so that we can shorten our commands. Given below is the current state of package.json { "scripts": { "start": "./node_modules/http-server/bin/http-server", "update-driver": "./node_modules/protractor/bin/webdriver-manager update", "start-driver": "./node_modules/protractor/bin/webdriver-manager start", "test": "./node_modules/protractor/bin/protractor conf.js" }, "dependencies": { "http-server": "^0.10.0", "protractor": "^5.1.2" } } The package.json file currently holds our dependencies and scripts. It contains command for starting development server, updating webdriver and starting webdriver (mentioned just before this) and command to run test. Next we need to include a configuration file for protractor. The configuration file should contain the test framework to be used, the address at which selenium is running and path to specs file. // conf.js exports.config = { framework: "jasmine", seleniumAddress: "http://localhost:4444/wd/hub", specs: ["tests/home-spec.js"] }; We have set the framework as jasmine and selenium address as http://localhost:4444/wd/hub. Next we need to define our actual file. But before writing tests we need to find out what are the things that we need to test. We will mostly be testing dynamic content loaded by Javascript files. Let us define a spec. A spec is a collection of tests. We will start by testing the category name. Initially when the page loads it should be equal to All apps. Next we test the top right hand side menu which is loaded by javascript using topmenu.json file. it("should have a category name", function()…

Continue ReadingUsing Protractor for UI Tests in Angular JS for Loklak Apps Site

Preparing for Automatic Publishing of Android Apps in Play Store

I spent this week searching through libraries and services which provide a way to publish built apks directly through API so that the repositories for Android apps can trigger publishing automatically after each push on master branch. The projects to be auto-deployed are: Open Event Orga App Open Event Android PSLab Android Loklak Wok Android Phimpe Android SUSI Android I had eyes on fastlane for a couple of months and it came out to be the best solution for the task. The tool not only allows publishing of APK files, but also Play Store listings, screenshots, and changelogs. And that is only a subset of its capabilities bundled in a subservice supply. There is a process before getting started to use this service, which I will go through step by step in this blog. The process is also outlined in the README of the supply project. Enabling API Access The first step in the process is to enable API access in your Play Store Developer account if you haven’t done so. For that, you have to open the Play Dev Console and go to Settings > Developer Account > API access. If this is the first time you are opening it, you’ll be presented with a confirmation dialog detailing about the ramifications of the action and if you agree to do so. Read carefully about the terms and click accept if you agree with them. Once you do, you’ll be presented with a setting panel like this: Creating Service Account As you can see there is no registered service account here and we need to create one. So, click on CREATE SERVICE ACCOUNT button and this dialog will pop up giving you the instructions on how to do so: So, open the highlighted link in the new tab and Google API Console will open up, which will look something like this: Click on Create Service Account and fill in these details: Account Name: Any name you want Role: Project > Service Account Actor And then, select Furnish a new private key and select JSON. Click CREATE. A new JSON key will be created and downloaded on your device. Keep this secret as anyone with access to it can at least change play store listings of your apps if not upload new apps in place of existing ones (as they are protected by signing keys). Granting Access Now return to the Play Console tab (we were there in Figure 2 at the start of Creating Service Account), and click done as you have created the Service Account now. And you should see the created service account listed like this: Now click on grant access, choose Release Manager from Role dropdown, and select these PERMISSIONS: Of course you don’t want the fastlane API to access financial data or manage orders. Other than that it is up to you on what to allow or disallow. Same choice with expiry date as we have left it to never expire. Click on ADD USER and…

Continue ReadingPreparing for Automatic Publishing of Android Apps in Play Store

Writing Selenium Tests for Checking Bookmark Feature and Search functionality in Open Event Webapp

We integrated Selenium Testing in the Open Event Webapp and are in full swing in writing tests to check the major features of the webapp. Tests help us to fix the issues/bugs which have been solved earlier but keep on resurging when some new changes are incorporated in the repo. I describe the major features that we are testing in this. Bookmark Feature The first major feature that we want to test is the bookmark feature. It allows the users to mark a session they are interested in and view them all at once with a single click on the starred button. We want to ensure that the feature is working on all the pages. Let us discuss the design of the test. First, we start with tracks page. We select few sessions (2 here) for test and note down their session_ids. Finding an element by its id is simple in Selenium can be done easily. After we find the session element, we then find the mark button inside it (with the help of its class name) and click on it to mark the session. After that, we click on the starred button to display only the marked sessions and proceed to count the number of visible elements on the page. If the number of visible session elements comes out to be 2 (the ones that we marked), it means that the feature is working. If the number deviates, it indicates that something is wrong and the test fails. Here is a part of the code implementing the above logic. The whole code can be seen here // Returns the number of visible session elements on the tracks page TrackPage.getNoOfVisibleSessionElems = function() { return this.findAll(By.className('room-filter')).then(this.getElemsDisplayStatus).then(function(displayArr) { return displayArr.reduce(function(counter, value) { return value == 1 ? counter + 1 : counter; }, 0); }); }; // Bookmark the sessions, scrolls down the page and then count the number of visible session elements TrackPage.checkIsolatedBookmark = function() { // Sample sessions having ids of 3014 and 3015 being checked for the bookmark feature var sessionIdsArr = ['3014', '3015']; var self = this; return self.toggleSessionBookmark(sessionIdsArr).then(self.toggleStarredButton.bind(self)).then(function() { return self.driver.executeScript('window.scrollTo(0, 400)').then(self.getNoOfVisibleSessionElems.bind(self)); }); }; Here is the excerpt of code which matches the actual number of visible session elements to the expected number. You can view the whole test script here //Test for checking the bookmark feature on the tracks page it('Checking the bookmark toggle', function(done) { trackPage.checkIsolatedBookmark().then(function(num) { assert.equal(num, 2); done(); }).catch(function(err) { done(err); }); }); Now, we want to test this feature on the other pages: schedule and rooms page. We can simply follow the same approach as done on the tracks page but it is time expensive. Checking the visibility of all the sessions elements present on the page takes quite some time due to a large number of sessions. We need to think of a different approach.We had already marked two elements on the tracks page. We then go to the schedule page and click on the starred mode. We calculate the current height of the page. We…

Continue ReadingWriting Selenium Tests for Checking Bookmark Feature and Search functionality in Open Event Webapp

Using Cron Scheduling to automatically run background jobs

Cron scheduling is nothing new. It is basically a concept in which the system keeps executing lines of code every few seconds, minutes, or hours. It is required in many large applications which require some automation in their working. I had to use it for two purposes in our Open-Event application. To automatically delete the items in the trash after a period of 30 days. To automatically send after event mails to the speakers and organizers when an event gets completed 1. Delete items in Trash system So the deleted items get stored in the trash of the admin. However if the items are not deleted or no action is performed on them then they should be deleted automatically if they stay in the trash for more than 30 days. I have used a framework - apscheduler (Advanced Python Scheduler) for this.The code is like this from apscheduler.schedulers.background import BackgroundScheduler def empty_trash(): with app.app_context(): print 'HELLO' events = Event.query.filter_by(in_trash=True) users = User.query.filter_by(in_trash=True) sessions = Session.query.filter_by(in_trash=True) for event in events: if datetime.now() - event.trash_date >= timedelta(days=30): DataManager.delete_event(event.id) for user in users: if datetime.now() - user.trash_date >= timedelta(days=30): transaction = transaction_class(Event) transaction.query.filter_by(user_id=user.id).delete() delete_from_db(user, "User deleted permanently") for session in sessions: if datetime.now() - session.trash_date >= timedelta(days=30): delete_from_db(session, "Session deleted permanently") trash_sched = BackgroundScheduler(timezone=utc) trash_sched.add_job(empty_trash, 'cron', day_of_week='mon-fri', hour=5, minute=30) trash_sched.start() The trash_sched is initialized first. It is given an instance of BackgroundScheduler() meaning it will always run in the background as long as the application server is running. The second line defines the trigger which is given to the scheduler meaning at what time intervals do we want the scheduler to execute the function. trash_sched.add_job(empty_trash, 'cron', day_of_week='mon-fri', hour=5, minute=30) Here the scheduler adds the function to the job list. The function is empty_trash and the trigger given here is 'cron'. The following line: day_of_week='mon-fri', hour=5, minute=30 sets the time at which the sheduler executes the job. The above line means that the job will be executed every day from Monday to Friday at 5:30 AM. Now coming to the function which is to be executed: def empty_trash(): with app.app_context(): events = Event.query.filter_by(in_trash=True) users = User.query.filter_by(in_trash=True) sessions = Session.query.filter_by(in_trash=True) for event in events: if datetime.now() - event.trash_date >= timedelta(days=30): DataManager.delete_event(event.id) for user in users: if datetime.now() - user.trash_date >= timedelta(days=30): transaction = transaction_class(Event) transaction.query.filter_by(user_id=user.id).delete() delete_from_db(user, "User deleted permanently") for session in sessions: if datetime.now() - session.trash_date >= timedelta(days=30): delete_from_db(session, "Session deleted permanently") There are three items in the trash: events, users and sessions. We get all the items in the trash by the query.all() method. Each model: Events, Users and Sessions has a column trash_date = db.Column(db.DateTime) The date on which the item is moved to the trash is stored in the trash_date column. And the following line: if datetime.now() - event.trash_date >= timedelta(days=30) checks if the item has been in the trash for more than 30 days. If yes then it is deleted automatically. This function is constantly executed by the apscheduler according to the time settings given by us. Thus we do…

Continue ReadingUsing Cron Scheduling to automatically run background jobs

Open Event Apk generator

So we made this apk generator currently hosted on a server (http://192.241.232.231) which let’s you generate an android app for your event in 10 minutes out of which the server takes about 8 minutes to build :P . So, essentially you just have to spare 2 minutes and just enter 3 things(email, Desired app’s name and Api link). Isn’t this cool? So how exactly do we do this? At the backend, we are running a python scripts with some shell scripts where the python script is basically creating directories, interacting with our firebase database to get the data entered by a user. So we made these scripts to first of all to clone the open event android repo, then customise and change the code in the repo according to the parameters entered by the organiser earlier(shown in the image). After the code has been changed by the scripts to customise the app according to the event the app will be used for, we move on to the script to build the apk, where we build and sign the apk in release mode using signing configs and release keys from the server because we don’t the organiser to generate keys and store it on the server to avoid the hassle and also the privacy concerns involving the keys. So this is when the apk is generated on the server. Now you have questions like the apk is generated but how do I get it? Do I have to wait on the same page for 10 minutes while the apk is being sent? The answer is no and yes. What I mean by this is that you can wait to download the apk if you want but we’ll anyways send it to your email you specified on the apk generator website. This is the sample Email we got when we were testing the system So it’s an end to end complete solution from which you can easily get an android app for your event in just 2 minutes. Would love to hear feedback and reviews. Please feel free to contact me @ manan13056@iiitd.ac.in or on social media’s(Twitter, Facebook). Adios! P.S. : Here is the link to the scripts we're using.

Continue ReadingOpen Event Apk generator