json

Read more about the article Integrating Firebase Cloud Functions In Badgeyay

Integrating Firebase Cloud Functions In Badgeyay

Badgeyay is an open source project developed by FOSSASIA Community for generating badges for conferences and events. The Project is divided into two parts frontend, which is in ember, and backend, which is in flask. Backend uses firebase admin SDK (Python) and Frontend uses firebase javascript client with emberfire wrapper for ember. Whenever an user signs up on the website, database listener that is attached to to the Model gets triggered and uses flask-mail for sending welcome mail to the user and in case of email and password signup, verification mail as well. Problem is sending mail using libraries is a synchronous process and takes a lot of processing on the server. We can use messaging queues like RabbitMQ and Redis but that will be burden as server cost will increase. The workaround is to remove the code from the server and create a firebase cloud function for the same task. Firebase cloud functions lets you run backend code on the cloud and can be triggered with HTTP events or listen for the events on the cloud, like user registration. Procedure Firebase uses our Gmail ID for login, so make sure to have a Gmail ID and on the first sight we will be greeted with Firebase console, where we can see our created or imported firebase apps. Create the app by clicking on the Add Project Icon and write the name of the application (e.g. Test Application) and select the region, in my case it is India. Firebase will automatically generated an application ID for the app. Click on Create Project to complete creation of project After Completion, click on the project to enter into the project. You will be greeted with an overview saying to integrate firebase with your project. We will click on the Add Firebase to web App and save the config as JSON in a file as clientKey.json for later use. Now we need to install the firebase tools on our local machine so for that execute npm i -g firebase-tools   Now login from the CLI so that firebase gets token for the Gmail ID of the user and can access the firebase account of that Gmail ID. firebase login   After giving permissions to the firebase CLI from your Gmail account in the new tab opened in browser, create a folder named cloud_functions in the project directory and in that execute firebase init   Select only functions from the list of options by pressing space. After this select the project from the list where you want to use the cloud function. You can skip the step if you later want to add the cloud function to project by selecting don’t setup a default project and can later be used by command firebase use --add Choose the language of choice If you want, you can enforce eslint on the project and after this the cloud function is set up and the directory structure looks as follows. We will write our cloud function in index.js.…

Continue ReadingIntegrating Firebase Cloud Functions In Badgeyay
Read more about the article Ember Data Integration In Badgeyay Frontend

Ember Data Integration In Badgeyay Frontend

Badgeyay is an open source utility to develop badges for events and tech conferences. Badgeyay project is divided into two components. Frontend part is designed with ember and backend part is designed with Flask and database as PostgreSQL and Firebase as PaaS. After refactoring the backend API for generation of badges, now it is time to consume the API in frontend by ember, and the way to consume the api in ember front--end is with the use of in built ember-data library. Ember data behaves in a way similar to server side ORM’s (Object Relational Mappers). It is a very versatile library and can be equipped with variety of backend services. It can be used with REST as well as sockets and other transfer protocols for communication. For better understanding the working of ember data, let’s see how to use the same to consume the File Upload endpoint in the backend. Procedure Enabling CORS on server, to allow cross-domain requests to the API. from flask_cors import CORS CORS(app, resources={r"*": {"origins": "*"}}) Creating Adapter for the model in frontend. In our case it is csv-file. In the adapter we need to specify the host and the path, because our backend api is not running on the same port. import DS from 'ember-data'; const { RESTAdapter } = DS; export default RESTAdapter.extend({ host : 'http://localhost:5000', pathForType : () => { return 'api/upload/file'; } }); After creating the adapter we need to create the record in the controller of the respective component. The record is like an object of a class, which when pushed to store will make a network request to backend (POST) and fetch the response from the backend. Backend response will provide the id to save in store import Controller from '@ember/controller'; import { inject as service } from '@ember/service'; export default Controller.extend({ routing : service('-routing'), actions : { mutateCSV(csvData) { let csv_ = this.get('store').createRecord('csv-file', { csvFile : csvData, extension : 'csv' }); csv_.save(); }, mutateText(txtData) { console.log(txtData); } } }); Model for the csv-file import DS from 'ember-data'; const { Model, attr } = DS; export default Model.extend({ csvFile : attr('string'), extension : attr('string') }); Next is to create serializers for the model. Serializers gets triggered at two moments, first when the data is sent to the server and second when data is received from the server. Each time an independent function gets executed. As the naming conventions of the functions pretty much explains their role, but for the sake of clarification serialize function gets executed when we send request to the server and normalizeResponse gets executed when we are getting response from the server. import DS from 'ember-data'; const { JSONAPISerializer } = DS; export default JSONAPISerializer.extend({ serialize(snapshot, options) { let json = this._super(...arguments); json.csvFile = { 'csvFile' : json.data.attributes['csv-file'], 'extension' : json.data.attributes.extension }; delete json.data; return json; }, normalizeResponse(store, primaryModelClass, payload, id, requestType) { return payload; } }); After receiving the response a promise is returned by the push method to save the record in the store…

Continue ReadingEmber Data Integration In Badgeyay Frontend
Read more about the article Store User’s Personal Information with SUSI

Store User’s Personal Information with SUSI

In this blog, I discuss how SUSI.AI stores personal information of it’s users. This personal information is mostly about usernames/links to different websites like LinkedIn, GitHub, Facebook, Google/Gmail etc. To store such details, we have a dedicated API. Endpoint is : https://api.susi.ai/aaa/storePersonalInfo.json In this API/Servlet, storing the details and getting the details, both the aspects are covered. At the time of making the API call, user has an option either to ask the server for a list of available store names along with their values or request the server to store the value for a particular store name. If a store name already exists and a client makes a call with new/updated value, The servlet will update the value for that particular store name. The reason you are looking at minimal user role as USER is quite obvious, i.e. these details correspond to a particular user. Hence neither we want someone writing such information anonymously nor we want this information to be visible to anonymous user until allowed by the user. In the next steps, we start evaluating the API call made by the client. We look at the combination of the parameters present in the request. If the request is to fetch list of available stores, server first checks if Accounting object even has a JSONObject for “stores” or not. If not found, it sends an error message “No personal information is added yet.” and error code 420. Prior to all these steps, server first generates an accounting object for the user. If found, details are encoded as JSONObject’s parameter. Look at the code below to understand things fairly. Accounting accounting = DAO.getAccounting(authorization.getIdentity()); if(post.get("fetchDetails", false)) { if(accounting.getJSON().has("stores")){ JSONObject jsonObject = accounting.getJSON().getJSONObject("stores"); json.put("stores", jsonObject); json.put("accepted", true); json.put("message", "details fetched successfully."); return new ServiceResponse(json); } else { throw new APIException(420, "No personal information is added yet."); } } If the request was not to fetch the list of available stores, It means client wants server to save a new field or update a previous value for that of a store name. A combination of If-else evaluates whether the call even contains required parameters. if (post.get("storeName", null) == null) { throw new APIException(422, "Bad store name encountered!"); } String storeName = post.get("storeName", null); if (post.get("value", null) == null) { throw new APIException(422, "Bad store name value encountered!"); } If request contains all the required data, then store name & value are extracted as key-value pair from the request. In the next steps, since the server is expected to store list of the store names for a particular user, First the identity is gathered from the already present authorization object in “serviceImpl” method. If the server finds a “null” identity, It throws an error with error code 400 and error message “Specified User Setting not found, ensure you are logged in”. Else, server first checks if a JSONObject with key “stores” exists or not. If not, It will create an object and will put the key value pair in the new JSONObject. Otherwise…

Continue ReadingStore User’s Personal Information with SUSI
Read more about the article Open Event API Server: Implementing FAQ Types

Open Event API Server: Implementing FAQ Types

In the Open Event Server, there was a long standing request of the users to enable the event organisers to create a FAQ section. The API of the FAQ section was implemented subsequently. The FAQ API allowed the user to specify the following request schema {  "data": {    "type": "faq",    "relationships": {      "event": {        "data": {          "type": "event",          "id": "1"        }      }    },    "attributes": {      "question": "Sample Question",      "answer": "Sample Answer"    }  } }   But, what if the user wanted to group certain questions under a specific category. There was no solution in the FAQ API for that. So a new API, FAQ-Types was created. Why make a separate API for it? Another question that arose while designing the FAQ-Types API was whether it was necessary to add a separate API for it or not. Consider that a type attribute was simply added to the FAQ API itself. It would mean the client would have to specify the type of the FAQ record every time a new record is being created for the same. This would mean trusting that the user will always enter the same spelling for questions falling under the same type. The user cannot be trusted on this front. Thus the separate API made sure that the types remain controlled and multiple entries for the same type are not there. Helps in handling large number of records: Another concern was what if there were a large number of FAQ records under the same FAQ-Type. Entering the type for each of those questions would be cumbersome for the user. The FAQ-Type would also overcome this problem Following is the request schema for the FAQ-Types API {  "data": {    "attributes": {      "name": "abc"    },    "type": "faq-type",    "relationships": {      "event": {        "data": {          "id": "1",          "type": "event"        }      }    }  } }   Additionally: FAQ to FAQ-type is a many to one relation. A single FAQ can only belong to one Type The FAQ-type relationship will be optional, if the user wants different sections, he/she can add it ,if not, it’s the user’s choice. Related links Best Practices for Designing an Effective FAQ Page: https://1stwebdesigner.com/faq-page-design/ Fancy FAQs with jQuery Sliders: https://davidwalsh.name/jquery-sliders

Continue ReadingOpen Event API Server: Implementing FAQ Types
Read more about the article List all the Users Registered on SUSI.AI

List all the Users Registered on SUSI.AI

In this blog, I'll be telling on how SUSI admins can access list of all the registered users from SUSI-server. Following this, they may modify/edit user role of any registered user. What is User Role? A UserRole defines the servlet access right. Not all users are allowed to access all the data and services. For  example, To list all the users, minimal user role expected is ADMIN. This classification of users are inspired by the wikipedia User Access Levels, see https://en.wikipedia.org/wiki/Wikipedia:User_access_levels.While querying SUSI, Users are classified into 7 different categories, namely : BOT ANONYMOUS USER   REVIEWER ACCOUNTCREATOR ADMIN BUREAUCRAT * Please see that these are as of the date of publish of this blog. These are subject to change, which is very unlikely. All the users who are not logged in but interacting with SUSI are anonymous users. These are only subject to chat with SUSI, login, signup or may use forgot password service. Once a user login to the server, a token is generated and sent back to client to maintain the identity, hence acknowledging them. Privileged users are those who have special rights with them. These are more like moderators with much special rights than any other user. At the top level of the hierarchy are the admins. These users have more rights than anyone. They can change role of any other user, override decision of any privileged user as well. Let us now look at the control flow of this. First things first, make a component of User List in the project. Let us name it ListUsers and since it has to be accessible by those users who possess ADMIN rights, you will find it enclosed in Admin package in components folder. Open up index.js file, import Listusers component  and add route to it in the following way : ...//other import statements import ListUser from "./components/Admin/ListUser/ListUser"; ...//class definition and other methods <Route path="/listUser" component={ListUser}/> …//other routes defined Find a suitable image for “List Users” option and add the option for List Users in static appbar component along with the image. We have used Material UI’s List image in our project. ...// other imports import List from 'material-ui/svg-icons/action/list'; …Class and method definition <MenuItem primaryText="List Users" onTouchTap={this.handleClose} containerElement={<Link to="/listUser" />} rightIcon={<List/>} /> ...//other options in top right corner menu Above code snippet will add an option to redirect admins to ‘/listUsers’ route. Let us now have a closer look at functionality of both client and server. By now you must have known what ComponentDidMount does. {If not, I’ll tell you. This is a method which is given first execution after the page is rendered. For more information, visit this link}. As mentioned earlier as well that this list will be available only for admins and may be even extended for privileged users but not for anonymous or any other user, an AJAX call is made to server in ComponentDidMount of ‘listuser’ route which returns the base user role of current user. If user is an Admin, another method,…

Continue ReadingList all the Users Registered on SUSI.AI
Read more about the article How to Send a Script as Variable to the Meilix ISO with Travis and Meilix Generator

How to Send a Script as Variable to the Meilix ISO with Travis and Meilix Generator

We wanted to add more features to Melix Generator web app to be able to customize Meilix ISO with more features so we thought of sending every customization we want to apply as a different variable and then use the scripts from Meilix Generator repo to generate ISO but that idea was bad as many variables are to be made and need to be maintained on both Heroku and Travis CI and keep growing with addition of features to web app. So we thought of a better idea of creating a combined script with web app for each feature to be applied to ISO and send it as a variable to Travis CI. Now another problem was how to send script as a variable after generating it as json do not support special characters inside the script. We tried escaping the special characters and the data was successfully sent to Travis CI and was shown in config but when setting that variable as an environment variable in Travis CI the whole value of variable was not taken as we had spaces in the script. So to eliminate that problem we encoded the variable in the app as base64 and sent it to Travis CI and used it using following code. To generate the variable from script. with open('travis_script_1.sh','rb') as f: os.environ["TRAVIS_SCRIPT"] = str(base64.b64encode(f.read()))[1:]   For this we have to import base64 module and open the script generated in binary mode and using base64 we encode the script and using Travis CI API we send variable as script to the Travis CI to build the ISO with script in chroot we were also required to make changes in Meilix to be able to decode the script and then copy it into chroot during the ISO build. sudo su <<EOF echo "$TRAVIS_SCRIPT" > edit/meilix-generator.sh mv browser.sh edit/browser.sh EOF   Using script inside chroot. chmod +x meilix-generator.sh browser.sh echo "$(<meilix-generator.sh)" #to test the file ./browser.sh rm browser.sh Resources Base64 python documentation from docs.python.org Base64 bash tutorial from scottlinux.com by Scott Miller su in a script from unix.stackexchange.com answered by Ankit

Continue ReadingHow to Send a Script as Variable to the Meilix ISO with Travis and Meilix Generator
Read more about the article SUSI.AI User Roles and How to Modify Them

SUSI.AI User Roles and How to Modify Them

In this blog, I discuss what is ‘user-role’ in SUSI.AI, what are the various roles and how SUSI admins can modify/update a user’s roles. What is User Role? A UserRole defines the servlet access right. Not all users are allowed to access all the data and services. For  example, To list all the users, minimal user role expected is ADMIN. This classification of users are inspired by the wikipedia User Access Levels, see https://en.wikipedia.org/wiki/Wikipedia:User_access_levels.While querying SUSI, Users are classified into 7 different categories, namely : BOT ANONYMOUS USER   REVIEWER ACCOUNTCREATOR ADMIN BUREAUCRAT * Please see that these are as of the date of publish of this blog. These are subject to change, which is very unlikely. If SUSI is active as a bot on some bot integrated platform (like line or kik), the user role assigned to it will be that of BOT. This user role just has technical access to the server. All the users who are not logged in but interacting with SUSI are ANONYMOUS users. These are only subject to chat, login and signup. They may use forgot password service and reset password services as well. Once a user login to the server, a token is generated and sent back to client to maintain the identity, hence acknowledging them as USER(s). Users with role assigned as “REVIEWERS” are expected to manage the Skill CMS. There might be some dispute or conflict in a skill. REVIEWERS then take the access of skill data and finalise the conflict there itself for smooth functionality. ADMIN users are those who have special rights with them. These are more like moderators with much special rights than any other user. At the top level of the hierarchy are the BUREAUCRATS. These users have more rights than anyone. They can change role of any other user, override decision of any ADMIN user as well. Both admins and bureaucrats have the access to all the settings file on the server. They not only can look at the list, but also download and upload them. Now these users also have right to upgrade or downgrade any other user as well. All these user roles are defined in UserRole.java file. In each request received by the server, the user role of user making the request is compared with the minimal user role in getMinimalUserRole() method. This method is defined in AbstractAPIHandler which validates if a user is allowed to access a particular servlet or not. private void process(HttpServletRequest request, HttpServletResponse response, Query query) throws ServletException, IOException { // object initialisation and comparsions // user authorization: we use the identification of the user to get the assigned authorization Authorization authorization = DAO.getAuthorization(identity); if (authorization.getUserRole().ordinal() < minimalUserRole.ordinal()) { response.sendError(401, "Base user role not sufficient. Your base user role is '" + authorization.getUserRole().name() + "', your user role is '" + authorization.getUserRole().getName() + "'"); return; } // evaluations based on other request parameters. } Now that we know about what User Roles actually are, let us look at how…

Continue ReadingSUSI.AI User Roles and How to Modify Them
Read more about the article Fetching Metadata in Loklak Server

Fetching Metadata in Loklak Server

In Loklak Server multiscrapers are working fine but there was a need to setup metadata framework to be embedded with the data. Metadata outputs the parameters passed, number of hits on the webpage to fetch results and number of results outputted. There is no metadata framework for TwitterScraper. Metadata is collected but there are 2 issues: 1) the metadata fields are directly feeded while outputing data. 2) Every Scraper had different metadata fields or none. To improve this for multiscraper system, I embedded metadata by configuring in the BaseScraper class and in PostTimeline iterator. If the metadata is directly collected in BaseScraper itself, then it will become non-of-developer-concern while working on scrapers and he can concentrate on improving scrapers. These are the following changes I made in code: 1) Input Get-Parameters For scrapers, one of the metadata field was input parameters. I directly added them in metadata block. protected Post getMetadata() { Post metadata = new Post(true); metadata.put("hits", this.hits); metadata.put("count", this.count); metadata.put("scraper", this.scraperName); metadata.put("input_parameters", this.extra); return metadata; }   2) Hits and Counts Hits refer to number of times Loklak Server made a hit to the target website where as Counts refer to number of posts scraped by the scraper. To fetching these data was easy. For count, I added a method putData in BaseScraper. It shall be used to create list of posts instead of directly creating the list. Here I have added counter which counts the posts. protected Post putData(Post typeArray, String key, JSONArray postList) { this.count = this.count + postList.length(); typeArray.put(key, postList); return typeArray; }   For hits, I just counted the number of times the URL was fed into ClientConnection method. public Post getDataFromConnection(String url, String type) throws IOException { // This adds to hits count even if connection fails this.hits++; ClientConnection connection = new ClientConnection(url); . . .   3) For multiscrapers in Search Endpoint This was a bit tricky task. For creating metadata block for all the scrapers, I had to fetch metadata block of all the scrapers, process them and then output with the results. I added this to PostTimeline iterator and implemented in a loop when a scraper outputs data. public void collectMetadata(JSONObject metadata) { // INITIALIZE PARAMETERS int hits = 0; int count = 0; Set scrapers = new HashSet<String>(); // GET LIST OF KEYS IN SCRAPER List<String> listKeys = new ArrayList<String>(this.posts.keySet()); int n = listKeys.size(); for (int i = 0; i < n; i++) { // FETCH METADATA POST FROM SCRAPED DATA Post postMetadata = (Post) this.posts.get(listKeys.get(i)).get("metadata"); hits = hits + Integer.parseInt(String.valueOf(postMetadata.get("hits"))); count = count + Integer.parseInt(String.valueOf(postMetadata.get("count"))); scrapers.add(postMetadata.get("scraper")); } // SET OUTPUT metadata.put("hits", hits); metadata.put("count", count); metadata.put("scraper_count", scrapers.size()); metadata.put("scrapers", scrapers); }   References Crawlers and Metadata Extraction (Stuff that needs to be solved): https://vimeo.com/53109189 Why Metadata? https://www.villanovau.com/resources/bi/metadata-importance-in-data-driven-world/#.WZmbMKvhXeQ

Continue ReadingFetching Metadata in Loklak Server
Read more about the article API to List All Users on SUSI.AI

API to List All Users on SUSI.AI

In this blog, I discuss how the SUSI server helps in listing out all the users registered on it. The only role Susi server plays is, Whenever it receives a request at http://api.susi.ai/aaa/getUsers.json The server evaluate the parameters in the request, validates them and notify the user accordingly. API needs 2 parameters, out of which access-token is a necessary. 2nd parameter has to be one from the given list : Parameter Data type getPageCount boolean GetUserCount boolean Page integer On the basis of this 2nd parameter, server gets to know what does the client with given access-token is requesting. Server evaluates the access-token and validates that if the access token belongs to a user with user role atleast ADMIN, then the request is valid and proceed further with fetching the data in next step. Otherwise, server responds with error code “401” and error message “Base user role not sufficient”. It is advisable for clients that before redirecting users to admin panel or any other service, Please hit http://api.susi.ai/aaa/showAdminService.json And check that whether the user logged in is allowed to access the admin panel or not. The servlet /showAdminService.json is quite easy to understand for even those new to programming. Coming back to our topic, by now, server knows that this client is authorized to access the user list. But what all information does server needs to provide? In response to this request, server encodes following attributes in the JSON Array {which is part of JSON object} and sends it to user : Attribute Description Name Email-Id of the user Anonymous Is this user anonymous or not User Role User Role of the user Confirmed User has verified account or not Last Login IP Last IP from which login was requested Last Login Time Time when last login request was made Signup Time When did the user signed up First things first, check if enough parameters are provided or not. If not, respond with error stating “Bad Request. No parameter present”. Otherwise, server does a general iteration which has to be done irrespective of the 2nd parameter. First of all, get a list of all the authorized users using getAuthorizedClients method of Data Access Object class. This method picks up all the keys from authorized file {which are nothing but identification of clients from which requests are received}. Though it, skips those key which are host addresses (which can not be used to identify a user), it does includes all the email ids {which are obvious identification of users}. public static Collection<ClientIdentity> getAuthorizedClients() { ArrayList<ClientIdentity> i = new ArrayList<>(); for (String id: authorization.keys()) { if(id.contains("host")) continue; i.add(new ClientIdentity(id)); } return i; } In next steps, the collection is converted to suitable data types over which iterations are easy and can be converted to JSON objects and Arrays easily. After this, server evaluates which parameter is requested in the request. Let us pick each case one by one for simplicity. Client has requested number of pages in the request. Server finds the…

Continue ReadingAPI to List All Users on SUSI.AI
Read more about the article Change Password for SUSI Accounts Using Access Token and Email-ID

Change Password for SUSI Accounts Using Access Token and Email-ID

In this blog, I discuss how the SUSI server synchronizes with SUSI Accounts and SUSI webchat for users to Change Password. When a user logs in, the clients store the email id of the user along with the access token in cookies. These are stored once the client gets a positive login response from the server. Both of these are required at the time of making the final call. Web clients store the email id and access token in the following way. cookies.set('loggedIn', loggedIn, { path: '/', maxAge: time }); cookies.set('emailId', email, { path: '/', maxAge: time }); First, the client has to ask the user to enter their current password. A javascript test is used to validate that at least 6 characters must be entered by the user. A similar test is run on the new password. But while confirming the password, client checks whether the user has entered the same password as new password or not. These are just the basics. In next stage (which is achieved only when all the above conditions are met), client encodes the email id (which it gets from cookies), current password, new password and the access token (which it again extracts from cookies). Now, Client just has to make an ajax request to the server. The response is processed and sent back to the client. Let us now look at PasswordChange Servlet. The base user role is defined as USER. Initial steps of the servlet are to extract the values form the request it receives. The values extracted from the request are in turn used to make a client’s identity. Before that, server checks if current and new password have same values or not. If not, then server returns a JSON response to user stating, “Your current password and new password matches”. Otherwise, it will continue its control flow as it is. Look at the code snippet below: if(password.equals(newpassword)){ result.put("message", "Your current password and new password matches"); result.put("accepted", false); return new ServiceResponse(result); } The reader here may think that they have discovered a hack. But they have not. Why? Because this is just the first step. In later stages, the hash of passwords are used to match to see whether the passwords match or not. To obtain a proper client identity, first a Client credentials object is made with support from the email id which is received in ‘changepassword’ attribute. Using the ClientCredentials object made above, an object of Authentication class is made. This object uses a method defined in its class to return a valid client identity. Using the client identity, value of password hash is extracted from the database along with the salt used to hash the password. If any error is encountered while extracting the client’s password hash value and/or salt value, an error is thrown towards the client, with a message stating “invalid credentials”. ClientCredential pwcredential = new ClientCredential(ClientCredential.Type.passwd_login, useremail); Authentication authentication = DAO.getAuthentication(pwcredential); ClientCredential emailcred = new ClientCredential(ClientCredential.Type.passwd_login, authentication.getIdentity().getName()); ClientIdentity identity = authentication.getIdentity(); String passwordHash; String salt;…

Continue ReadingChange Password for SUSI Accounts Using Access Token and Email-ID