Correct the API for Downloading GitHub Content in SUSI.AI android app

The content from github in the SUSI.AI android app is downloaded through simple links and the data is parsed through them and is used in the app depending upon what needs to be done with that data at that time.

A simple example for this that was used in the app was  :

private val imageLink = “https://raw.githubusercontent.com/fossasia/susi_skill_data/master/models/general/”

Above is the link that is used to download and display the images of the skills in the app. All the api calls that generally take place in SUSI are through the SUSI server, and making the call to display the images for the skills which takes place through the github links should be replaced by making the calls to the SUSI server instead, as this is a terrible programming style, with this style the project cannot be cloned from other developers and it cannot be moved to other repositories.

We see that there was a huge programming style issue present in the android app and hence, it was fixed by adding the API that calls the SUSI server for external source images and removing the existing implementation that downloads the image from github directly.

Below is an example of the link to the API call made in the app that was needed for the request to be made to the SUSI server :

${BaseUrl.SUSI_DEFAULT_BASE_URL}/cms/getImage.png?model=${skillData.model}&language=${skillData.language}&group=${skillData.group}&image=${skillData.image}

The link is displayed in the kotlin string interpolation manner, here is what the actual URL would look like :

https://api.susi.ai/cms/getImage.pngmodel=${skillData.model}&language=${skillData.language}&group=${skillData.group}&image=${skillData.image}

Here the values with ‘$’ symbol are the parameters for the API taken from the SkillData.kt file and are put inside the link so that the image needed can be extracted.

Now, since we use this link to set the images, to avoid the duplicate code, an Object class was made for this purpose. The object class contained two functions, one for setting the image and one for parsing the skilldata object and forming a URL out of it. Here is the code for the object class :

object Utils {

  fun setSkillsImage(skillData: SkillData, imageView: ImageView) {
      Picasso.with(imageView.context)
              .load(getImageLink(skillData))
              .error(R.drawable.ic_susi)
              .fit()
              .centerCrop()
              .into(imageView)
  }

  fun getImageLink(skillData: SkillData): String {
      val link = “${BaseUrl.SUSI_DEFAULT_BASE_URL}/cms/getImage.png?model=${skillData.model}&language=${skillData.language}&group=${skillData.group}&image=${skillData.image}”
              .replace(” “,“%20”)
      Timber.d(“SUSI URI” + link)
      return link
  }
}

setSkillsImage() method sets the image in the ImageView and the getImageLink() method returns the image formed out of the SkillData object.

References

 

Continue ReadingCorrect the API for Downloading GitHub Content in SUSI.AI android app

Adding a feature to delete skills from skill page for admins

SUSI Skill CMS has evolved drastically over the past few months with not only the introduction of skill metrics, skill analytics and powerful sorting features and interactive skill view types we needed the SUSI admins to be able to delete skills directly from the skills page and hence the skill can be deleted without visiting the admin service and then locating the skill and deleting it. This feature can be useful when a skill live on the system needs to be removed instantaneously for any reason like the API used by the skill going down or if it is a redundant skill or anything else. This feature was much needed for the admins and thus it was implemented.

About the API

An API is developed at the server so from the client we call this API to fetch data from the server and plug this data into the chart we wish to render.

Endpoint :

/cms/deleteSkill.json?access_token={access_token}&model={model}&group={group}&language={language}&skill={skill}

 

Parameters :

  • Model
  • Group
  • Skill
  • Language
  • Feedback
  • Access token (taken from the session of the logged in user)

Sample API call :

/cms/deleteSkill.json?access_token=ANecfA1GjP4Bkgv4PwjL0OAW4kODzW&model=general&group=Knowledge&language=en&skill=whois

Displaying a button with delete icon on skill page

The option to delete skill should be available at the skill page for each skill so we add a button with a delete icon for this in the group of edit skills and skill version buttons, clicking over this button will open up a confirmation dialog with two actions notable the delete/confirm button which deletes the skills and the cancel button which can be useful in case the user changes their mind. On clicking the delete button the request to delete the skill is sent to the server and thus the skill is deleted.

Import some required components

import Dialog from 'material-ui/Dialog';
import FlatButton from 'material-ui/FlatButton';
import Cookies from 'universal-cookie';
import $ from 'jquery';

 

Adding some variables to the component state which will help us decide when the delete skill dialog is to be shown.

this.state = {
   ...
   showDeleteDialog: false,
   ...
}

 

Display the delete skill button only when the user is logged in user has admin rights.

{
   cookies.get(showAdmin) ? (
           ...
   ): ''
}

 

Adding some JSX to the component’s render function which includes a div in the skill page section and the Dialog component for the delete skill and some actions which in our case is the the confirmation to delete skill and to cancel the skill deletion in case the user changes their mind. Also a tooltip is shown which appears on hovering over the delete skill button.

<div className="skillDeleteBtn">
 <FloatingActionButton
        onClick={this.handleDeleteToggle}
        data-tip="Delete Skill"
        backgroundColor={colors.header}
 >
        <DeleteBtn />
 </FloatingActionButton>
 <ReactTooltip effect="solid" place="bottom" />
 <Dialog
        title="Delete Skill"
        actions={deleteDialogActions}
        modal={false}
        open={this.state.showDeleteDialog}
        onRequestClose={this.handleDeleteToggle}
 >
        <div>
         Are you sure about deleting{' '}
         <span style={{ fontWeight: 'bold' }}>
           {this.state.skill_name}
         </span>?
        </div>
 </Dialog>
</div>

 

Clicking the delete skill button will change the state variable which decides whether the dialog is to be shown or not.

handleDeleteToggle = () => {
 this.setState({
        showDeleteDialog: !this.state.showDeleteDialog,
 });
};

 

Adding submit and cancel actions for the dialog menu and send them to the dialog as a prop.

const deleteDialogActions = [
 <FlatButton
        label="Delete"
        key="delete"
        style={{ color: 'rgb(66, 133, 244)' }}
        onClick={this.deleteSkill}
 />,
 <FlatButton
        label="Cancel"
        key="cancel"
        style={{ color: 'rgb(66, 133, 244)' }}
        onClick={this.handleDeleteToggle}
 />,
];

Hitting the endpoint for skill deletion.

Adding onClick event handlers for dialog actions, for the cancel button we simply toggle the view of the delete skill dialog and for the submit button hits the endpoint for skill deletion and then we display a snack bar message about the status of request we submit if it succeeded or failed. Once the submit button is clicked we hit the delete skill endpoint by supplying appropriate params and post the request for skill deletion by calling the API through AJAX with appropriate params and thus this concludes the skill deletion workflow.

Build the URL for the AJAX request.

let deleteUrl =    `${urls.API_URL}/cms/deleteSkill.json?model=${this.state.skillModel}&group=${this.state.skillGroup}&language=${this.state.skillLanguagelanguage}&skill=${this.state.skill_name}&access_token=${cookies.get('loggedIn')}`

 

Make an AJAX request to the built API URL and in case the request is successful we set the conditional for displaying the snackbar to true and set the snackbar message depending on whether the skill was deleted successfully or it failed.

$.ajax({
 url: deleteUrl,
 dataType: 'jsonp',
 jsonp: 'callback',
 crossDomain: true,
 success: function(data) {
        // redirect to the index page since the skill page won't be accessible
        this.handleDeleteToggle();
        this.setState({
         dataReceived: true,
        });
        this.props.history.push('/');
 }.bind(this),
 error: function(err) {
        console.log(err);
        this.handleReportToggle();
        this.setState({
         openSnack: true,
         snackMessage: 'Failed to delete the skill.',
        });
 }.bind(this),
});

 

This concludes the workflow of how the skill deletion feature was implemented on the CMS skill page for admins. I hope you found this useful.

Resources

Continue ReadingAdding a feature to delete skills from skill page for admins

Adding a feature to report skills in the CMS

A lot of interesting features were introduced in the SUSI.AI Skills CMS over the past few months but it lacked the functionality for users to be able to report skills which they find inappropriate or for any other reason. So an API was developed at the server to flag the skills as inappropriate and thus using this API endpoint an option was added at the skill page for each skill to mark the skill as inappropriate or report it. This data could be useful by admins to re-review the skills and see if something is wrong with it and it things seem out of place the skill can be removed or can be disabled.

About the API

An API is developed at the server so from the client we call this API to fetch data from the server and plug this data into the chart we wish to render.

Endpoint :

/cms/reportSkill.json?model={model}&group={group}&skill={skill}&feedback={feedback message}&access_token={access_token}

 

Parameters :

  • Model
  • Group
  • Skill
  • Language
  • Feedback
  • Access token (taken from the session of the logged in user)

Sample API call :

/cms/reportSkill.json?model=general&group=Knowledge&skill=Anime Suggestions&feedback=Not good&access_token=6O7cqoMbzlClxPwg1is31Tz5pjVwo3

Displaying option to report on skill page

The option to report skill should be available at the skill page for each skill so we add a field in the skill details section to the skill page component which will only be visible to the logged in users and on clicking over this field we display a dialog with a text field, the user can enter the message or the reason for reporting the skill and then clicking on the submit button when the user is done writing or click on the cancel button in case the user changes their mind to report the skill. Once the message is submitted we run a function by passing in the feedback message which in turn hits the corresponding endpoint and posts the data on the server.

Import some required components

import Dialog from 'material-ui/Dialog';
import FlatButton from 'material-ui/FlatButton';
import TextField from 'material-ui/TextField';

 

Adding some variables to the component state which will help us decide when the report dialog is to be shown and the feedback message as the user types and some other relevant data.

this.state = {
   ...
   skillTag: '',
   showReportDialog: false,
   feedbackMessage: ''
   ...
}

 

Display the report feature only when the user is logged in.

{
   cookies.get('loggedIn') ? (
           ...
   ): ''
}

 

Adding some jsx to the component’s render function which includes a div in the skill details section and the Dialog component for the report message and confirmation and the dialog contains a text field to take report message and some actions which in our case is the send report action and the cancel report action.

<tr>
 <td>Report: </td>
 <td>
        <div
         style={{ color: '#108ee9', cursor: 'pointer' }}
         onClick={this.handleReportToggle}
        >
         Flag as inappropriate
        </div>
 </td>
 <Dialog
        title="Flag as inappropriate"
        actions={reportDialogActions}
        modal={false}
        open={this.state.showReportDialog}
        onRequestClose={this.handleReportToggle}
 >
        <TextField
         hintText="Leave a feedback message"
         floatingLabelText="Feedback message"
         multiLine
         floatingLabelFocusStyle={{
           color: 'rgb(66, 133, 244)',
         }}
         underlineFocusStyle={{
           borderColor: 'rgb(66, 133, 244)',
         }}
         fullWidth
         onChange={(event, val) =>
           this.saveReportFeedback(val)
         }
        />
 </Dialog>
</tr>

 

Clicking the report as inappropriate will change the state variable which decides whether the dialog is to be shown or not.

handleReportToggle = () => {
        this.setState({
             showReportDialog: !this.state.showReportDialog,
        });
};

 

Adding submit and cancel actions for the dialog menu and send them to the dialog as a prop.

const reportDialogActions = [
 <FlatButton
        label="Cancel"
        key="cancel"
        style={{ color: 'rgb(66, 133, 244)' }}
        onClick={this.handleReportToggle}
 />,
 <FlatButton
        label="Submit"
        key="submit"
        style={{ color: 'rgb(66, 133, 244)' }}
        onClick={this.handleReportSubmit}
 />,
];

Hitting the endpoint for report submit.

Adding onClick event handlers for dialog actions, for the cancel button we simply toggle the view of the report dialog and for the submit button we take the feedback message entered by the user in the text field and hit the endpoint for skill reporting and then we display a snackbar message about the status of report submit if it succeeded or failed.
Once the skill is submitted button is clicked we hit the report endpoint and post the feedback data and call the API through AJAX with appropriate params and thus this concludes the skill reporting workflow.

Build the URL for the AJAX request.

let reportUrl = `${urls.API_URL}/cms/reportSkill.json?model=${
        this.state.skillModel
 }&group=${this.state.skillGroup}&language=${
        this.state.skillLanguage
 }&skill=${this.state.skillTag}&feedback=${
        this.state.feedbackMessage
 }&access_token=${cookies.get('loggedIn')}`;

 

Make an AJAX request to the built API URL and in case the request is successful we set the conditional for displaying the snackbar to true and set the snackbar message depending on whether the skill was reported successfully or it failed.

$.ajax({
 url: reportUrl,
 dataType: 'jsonp',
 jsonp: 'callback',
 crossDomain: true,
 success: function(data) {
        self.handleReportToggle();
        self.setState({
           openSnack: true,
           snackMessage: 'Skill has been reported successfully.',
        });
 },
 error: function(e) {
        self.handleReportToggle();
        self.setState({
           openSnack: true,
           snackMessage: 'Failed to report the skill.',
        });
 },
});

 

This concludes the workflow of how the skill reporting feature was implemented on the CMS. I hope you found this useful.

Resources

Continue ReadingAdding a feature to report skills in the CMS

Add Info on Skill Usage Distribution for all Skills by an Author in SUSI.AI

SUSI Skill CMS has a dashboard option available at the /dashboard route which displays several data for the logged in user as the skills created by the user and the ratings the user has provided to all the skills, since we have a skill usage section available on all skill pages which depicts the skill usage count for the past week in a line chart. Skill creators didn’t have a functionality to see the skill usage distribution on their skills which can provide some useful insight like how some of the skills they created are performing in comparison to the others so I developed a ‘My Analytics’ section in the dashboard page and displayed the skill usage distribution in the form of pie chart among the skills created by the logged in users.

About the API

An API is developed at the server so from the client we call this API to fetch data from the server and plug this data into the chart we wish to render.

Endpoint :

/cms/getSkillsByAuthor.json?author_email={email}

 

Parameters :

Email ID which is taken from the cookies since it is stored there once the user logs in.

Sample API call :

/cms/getSkillsByAuthor.json?author_email=anshu.av97@gmail.com

Fetching the data for the component

We first create a separate My Analytics component and require it in the dashboard and make an AJAX call to the appropriate endpoint inside a loadSkillsUsage function which is called inside the componentDidMount hook after which the server returns raw data in the form of JSON. We then pass the response into a saveUsageData function to parse the data for our use and save it to the application state.

loadSKillsUsage = () => {
 let url =
        urls.API_URL +
`/cms/getSkillsByAuthor.json?author_email=${cookies.get('emailId')}`;
 let self = this;
 $.ajax({
        url: url,
        dataType: 'jsonp',
        jsonp: 'callback',
        crossDomain: true,
        success: function(data) {
         self.saveUsageData(data.author_skills || []);
         ...
        },
        error: function(err) {
         ...
        },
 });
};

 

Set the application state with the received data which the pie chart component will use as it’s data source.

saveUsageData = data => {
 const skillUsage = data.map(skill => {
        let dataObject = {};
        dataObject.skill_name = skill.skill_name;
        dataObject.usage_count = skill.usage_count || 0;
        return dataObject;
 });
 this.setState({ skillUsage });
};

Implementing the UI

We create a separate ‘My Analytics’ component which is imported into the dashboard component to make the code cleaner and manageable. So inside the My analytics component, we fetch the data from the server as depicted above and after that, we render the pie chart component after importing from the recharts library.

Importing the pie chart components from the recharts library.

import { Legend, PieChart, Pie, Sector, Cell, ResponsiveContainer } from 'recharts';

 

Rendering the pie chart component while supplying appropriate props most important of which is the data prop which will be used in the chart and that data is available in the application state as saved earlier. We also have other styling props and a function which is triggered when hovering over cells of the pie chart to represent the data of the hovered cell. We also supply the appropriate nameKey and dataKey props as per the data format available in the state.

<ResponsiveContainer width={600} height={350}>
 <PieChart>
        <Pie
         activeIndex={this.state.activePieIndex}
         activeShape={renderActiveShape}
         data={this.state.skillUsage}
         cx={300}
         cy={175}
         innerRadius={80}
         nameKey="skill_name"
         dataKey="usage_count"
         outerRadius={120}
         fill="#8884d8"
         onMouseEnter={this.onPieEnter}
        >

       ...
         </Pie>
        <Legend wrapperStyle={{ position: 'relative' }} />
 </PieChart>
</ResponsiveContainer>

 

Configuring color for each Cell in the pie so it looks more interactive and we have distinguished colors for all devices.

{this.state.skillUsage.map((entry, index) => (
 <Cell
   key={index}
   fill={
     [
       '#0088FE',
       '#00C49F',
       '#FFBB28',
       '#FF8042',
       '#EA4335',
     ][index % 5]
   }
 />
))}

 

Rendering the Pie only when data is available in props so we don’t end up rendering a blank chart which obviously won’t look good.

{
 this.state.skillUsage !== [] ? (
   ...
 ): ''
}

Resources

Continue ReadingAdd Info on Skill Usage Distribution for all Skills by an Author in SUSI.AI

Implementing feature to filter skills by average customer review

SUSI Skill CMS showcases all the skills on the index page but lacks the functionality to refine skills according to average customer review which is a much-needed feature since some users may only want to try skills which have at least a minimum rating so they can know instantly which skills are performing well in comparison to others. Thus, we implement several star inputs on the sidebar to select skills which have ratings greater than or equal to the selected rating input.

Implementing the UI

Add a menu to the sidebar at the bottom of all categories and display ‘Refine by’ submenu text to denote the section.

<Menu desktop={true} disableAutoFocus={true}>
 <Subheader style={{ fontWeight: 'bold' }}>Refine by</Subheader>
 <h4 style={{ marginLeft: '12px', marginBottom: '4px' }}>
   Avg. Customer Review
 </h4>

...

 

Display rating options to the user by displaying a list of Ratings component imported from react-ratings-declarative, these are to be displayed for all ratings say four stars and above, three stars and above and so on, i.e.

<div
 style={styles.singleRating}
 onClick={() => this.handleRatingRefine(4)}
>
 <Ratings
        rating={4}
        widgetRatedColors="#ffbb28"
        widgetDimensions="20px"
        widgetSpacings="0px"
 >
        <Ratings.Widget />
        <Ratings.Widget />
        <Ratings.Widget />
        <Ratings.Widget />
        <Ratings.Widget />
 </Ratings>
 <div
        style={styles.ratingLabel}
        className={this.state.rating_refine === 4 ? 'bold' : ''}
 >
        & Up
 </div>
</div>

 

We add some styling and attach an onClick listener on each rating component which will handle the refining of skills according to the rating clicked, the idea behind this is to save the rating for the clicked option to the component state and re-render the skill cards

handleRatingRefine = rating => {
 this.setState(
        {
         rating_refine: rating,
        },
        this.loadCards(),
 );
};

 

When the component state is successfully set loadCards function as a callback is called which re-renders the cards by applying filter over the skills which match the average rating criteria which we just set.

if (self.state.rating_refine) {
 data.filteredData = data.filteredData.filter(
        skill =>
         skill.skill_rating.stars.avg_star >= self.state.rating_refine,
 );
}

Displaying a button to clear any refinements made

Once the skills are refined a button is needed to clear any refinements made. Initially when no refinements are made the rating_refine in the state is set to null which indicates that no refinements are made so whenever the value of that state is no null we render a button to clear the refinements or set the rating_refine state to null.

{this.state.rating_refine ? (
 <div
        className="clear-button"
        style={styles.clearButton}
        onClick={() => this.handleRatingRefine(null)}
 >
        Clear
 </div>
) : (
 ''
)}

Resources

Continue ReadingImplementing feature to filter skills by average customer review

Implementing My Rating Section on the SUSI.AI Skills Dashboard

SUSI Skill CMS provides the functionality to rate the skills, therefore users rate skills they use but there isn’t any place where they can see all the skills they rated, thus a ‘My Ratings’ section was implemented on the dashboard page to view these statistics. So to see what ratings they have given to skills they can just login to the cms and navigate to /dashboard and a my ratings components is visible there which lists all the ratings the user has provided in a nice tabular format.

About the API

An API endpoint is implemented on the server which fetches the skill data for skills the user has rated which includes the skill name, stars given and the timestamp.

/cms/getProfileDetails.json?access_token=

 

So we pass the access token of the authenticated user and a JSON response is received which contains all the details as depicted below, this data is then parsed on the frontend and filled in a tabular form on the MyRatings section.

{
 "rated_skills": [
   {"amazon_shopping": {
     "stars": "1",
     "timestamp": "2018-06-10 13:05:32.295"
   }},
   {"aboutsusi": {
     "stars": "2",
     "timestamp": "2018-06-10 13:26:26.222"
   }},
   {"anagrams": {
     "stars": "3",
     "timestamp": "2018-06-10 13:25:31.195"
   }}
 ],
 "session": {"identity": {
   "type": "email",
   "name": "anshu.av97@gmail.com",
   "anonymous": false
 }},
 "accepted": true,
 "message": "User ratings fetched."
}

Displaying the results on the web

Make a MyRatings component and render it on the dashboard component

Make an AJAX call to the API and save the returned data to the component state. First create a loadSkills function in componentDidMount which will be called just as the component is mounted to the DOM which will then fetch data from the server, extract the meaningful parts such as skill_name, skill_star and timestamp and push them to an array which in this case is ratingsData. While the data is being fetched we show a circular loader for better UX and once we receive the data we save it in the component state and turn loading to false which will replace the loading animation with the actual data.

loadSkills = () => {
 let url;
 url =
   urls.API_URL +
   '/cms/getProfileDetails.json?access_token=' +
   cookies.get('loggedIn');
 let self = this;
 let ratingsData = [];
 $.ajax({
   url: url,
   jsonpCallback: 'pxcd',
   dataType: 'jsonp',
   jsonp: 'callback',
   crossDomain: true,
   success: function(data) {
     if (data.rated_skills) {
       for (let i of data.rated_skills) {
         let skill_name = Object.keys(i)[0];
         ratingsData.push({
           skill_name: skill_name,
           skill_star: i[skill_name].stars,
           rating_timestamp: i[skill_name].timestamp,
         });
       }
       self.setState({
         ratingsData,
       });
     }
     self.setState({
       loading: false,
     });
   },
   error: function(err) {
     self.setState({
       loading: false,
       openSnackbar: true,
       msgSnackbar: "Error. Couldn't rating data.",
     });
   },
 });
};

 

Display a loading animation when the data is being fetched, we maintain a state in the component called loading which is initially true since we don’t have the data just as the component is rendered so after we receive the data we turn the loading state to false which will hide the circular loader and display the component with the data received.

{this.state.loading ? (
         <div className="center">
           <CircularProgress size={62} color="#4285f5" />
           <h4>Loading</h4>
         </div>
       ) : ( ... )
}

 

Add a table to structure the state data, since we’re using material-ui library throughout the project we use material table component from the library and populate the data received in it with 3 columns namely skill name, skill star, and timestamp.

<div className="table-wrap">
 <Table className="table-root" selectable={false}>
   <TableHeader displaySelectAll={false} adjustForCheckbox={false}>
     <TableRow>
       <TableHeaderColumn>Skill Name</TableHeaderColumn>
       <TableHeaderColumn>Rating</TableHeaderColumn>
       <TableHeaderColumn>Timestamp</TableHeaderColumn>
     </TableRow>
   </TableHeader>
   <TableBody displayRowCheckbox={false}>
     {ratingsData.map((skill, index) => {
       return (
         <TableRow key={index}>
           <TableRowColumn style={{ fontSize: '16px' }}>
             {(
               skill.skill_name.charAt(0).toUpperCase() +
               skill.skill_name.slice(1)
             ).replace(/[_-]/g, ' ')}
           </TableRowColumn>
           <TableRowColumn style={{ fontSize: '16px' }}>
             {skill.skill_star}
           </TableRowColumn>
           <TableRowColumn>
             {this.parseDate(skill.rating_timestamp)}
           </TableRowColumn>
         </TableRow>
       );
     })}
     <TableRow />
   </TableBody>
 </Table>
</div>

 

Display a message when there is no rating data available, let’s say the user has not rated any skills so it does not make sense to display an empty table and therefore we display a message instead for users to rate skills.

{ratingsData.length === 0 &&
!this.state.loading && (
 <div>
   <div className="center">
     <br />
     <h2>
       You have not rated any skill, go to{' '}
       <Link to="/">SUSI Skills Explorer</Link> and rate.
     </h2>
     <br />
   </div>
 </div>
)}

 

Import this component in the dashboard component and render it below My Skills

import MyRatings from './MyRatings';

 

Render the imported MyRatings component on a separate card from material-ui on the dashboard.

<Paper
 style={styles.paperStyle}
 className="botBuilder-page-card"
 zDepth={1}
>
 <h1 className="center">My Ratings</h1>
 <MyRatings />
</Paper>

Resources

Continue ReadingImplementing My Rating Section on the SUSI.AI Skills Dashboard

Upgrading Open Event to Use Sendgrid API v3

Sendgrid recently upgraded their web API to send emails, and support for previous versions was deprecated. As a result, Open Event Server’s mail sending tasks were rendered unsuccessful, because the requests they were sending to Sendgrid were not being processed. On top of that, it was also found out later that the existing Sendgrid API key on the development server was expired. This had to be fixed at the earliest because emails are a core part of Open Event functionality.

The existing way for emails to be sent via Sendgrid used to hit the endpoint “https://api.sendgrid.com/api/mail.send.json” to send emails. Also, the payload structure was as follows:

payload = {
    'to': to,
    'from': email_from,
    'subject': subject,
    'html': html
}

Also, a header  “Authorization”: “Bearer ” accompanied the above payload. However, Sendgrid changed the payload structure to be of the following format:

{

“personalizations”: [

{“to”: [

{“email”: “example@example.com“}

]

}

],

“from”: {

“email”: “example@example.com

},

“subject”: “Hello, World!”,

“content”: [

{

“type”: “text/plain”,

“value”: “Heya!”

}

]

}

Furthermore, the endpoint was changed to be “https://api.sendgrid.com/v3/mail/send”. To incorporate all these changes with the minimum number of modified lines in the codebase, it was required for that the structure change itself happens at a fairly low level. This was because there are lots of features in the server that perform a wide variety of email actions. Thus, it was clear that changing all of them will not be the most efficient thing to do. So the perfect place to implement the API changes was the function send_email() in mail.py, because all other higher-level email functions are built on top of this function. But this was not the only change, because this function itself used another function, called send_email_task() in tasks.py, specifically for sending email via Sendgrid. So, in conclusion, the header modifications were made in send_email() and payload structure as well as endpoint modifications were made within send_email_task(). This brought the server codebase back on track to send emails successfully. Finally, the key for development server was also renewed and added to its settings in the Heroku Postgres database.

Screenshots:

Screen Shot 2018-08-21 at 3.40.12 PM.png

Screen Shot 2018-08-21 at 3.40.32 PM.png

Resources

Continue ReadingUpgrading Open Event to Use Sendgrid API v3

Implementing Checkout Times for Attendees on Open Event Server

As of this writing, Open Event Server did not have the functionality to add, manipulate and delete checkout times of attendees. Event organizers should have access to log and update attendee checkout times. So it was decided to implement this functionality in the server. This boiled down to having an additional attribute checkout_times in the ticket holder model of the server.

So the first step was to add a string column named checkout_times in the ticket holder database model, since this was going to be a place for comma-separated values (CSV) of attendee checkout times. An additional boolean attribute named is_checked_out was also added to convey whether an attendee has checked out or not. After the addition of these attributes in the model, we saved the file and performed the required database migration:

To create the migration file for the above changes:

$ python manage.py db migrate

To upgrade the database instance:

$ python manage.py db upgrade

Once the migration was done, the API schema file was modified accordingly:

class AttendeeSchemaPublic(SoftDeletionSchema):
    """
    Api schema for Ticket Holder Model
    """
    
    checkout_times = fields.Str(allow_none=True)  # ←
    is_checked_out = fields.Boolean()  # ←
    

After the schema change, the attendees API file had to have code to incorporate these new fields. The way it works is that when we receive an update request on the server, we add the current time in the checkout times CSV to indicate a checkout time, so the checkout times field is essentially read-only:

from datetime import datetime
...
class AttendeeDetail(ResourceDetail):
    def before_update_object(self, obj, data, kwargs):
        
        if 'is_checked_out' in data and data['is_checked_out']:
        ...
        else:
            if obj.checkout_times and data['checkout_times'] not in \
obj.checkout_times.split(","):
                data['checkout_times'] = '{},{},{}'.format(
                    obj.checkout_times,
                    data['checkout_times'],
                    datetime.utcnow())

 

This completes the implementation of checkout times, so now organizers can process attendee checkouts on the server with ease.

Resources

Continue ReadingImplementing Checkout Times for Attendees on Open Event Server

Enforcing Constraints Throughout a Flask Back-End

Recently it was discovered that Open Event Server does not validate attendees’ tickets. Specifically, it was possible to create an arbitrary number of attendees who’d be attending an event on the same ticket! To fix this, a constraint had to be set up across different layers of Open Event Server, which is based on Flask and Postgres. This post will demonstrate how the constraint was added in the server, and these steps should apply in general to any Flask-based server with a relational back-end.

First of all, the immediate idea that comes after investigating such an issue, is to add a UNIQUE constraint to the database. For this specific case, the problem was in ticket_holders table of the Open Event database. There was originally no check imposed on the ticket_id and event_id columns.

As can be seen in the ticket_holders schema (using the \d+ ticket_holders command), there is no mention of uniqueness on either column. The initial guess was that the combination of ticket_id and event_id should be unique throughout the table to avoid multiple holders attending on the same ticket. However,imposing uniqueness on just the ticket_id column would’ve also worked. So, to be on the safer side, I moved ahead by adding uniqueness on both the columns.

To fix this, we need to make changes to the ticket_holder model. So, in the ticket_holder model file, we add a __table_args__ attribute to the TicketHolder class. This attribute represents the various constraints imposed on the ticket_holders table:

class TicketHolder(db.Model):
    __tablename__ = "ticket_holders"
    __table_args__ = (
db.UniqueConstraint('ticket_id', 'event_id', name='ticket_event'),
) # this is the constraint we add

    id = db.Column(db.Integer, primary_key=True)
    firstname = db.Column(db.String, nullable=False)
    lastname = db.Column(db.String, nullable=False)






The TicketHolder class has attributes named ticket_id and event_id, so to add a unique constraint over them, we pass their names to the UniqueConstraint constructor. Also, any suitable name can be given to the constraint, I chose ‘ticket_event’ to simply emphasize the relationship. Now that we’ve edited the database model file, we have to perform a database migration.

Before we command the migration, we have to remove the entries that potentially violate the constraint we just imposed. As a temporary fix, I connected to the database and deleted all non-unique rows via plain SQL. For a more consistent fix, I will implement this simple deletion code in the database migration file, if need be. So, once the non-unique rows are gone, we perform the database migration as follows:

$ python manage.py db migrate

And then,

$ python manage.py db upgrade

These commands may be different for different projects, but their purpose is the same – to update the database. The upgrade command generates a migration file which looks as follows:

from alembic import op
import sqlalchemy as sa
import sqlalchemy_utils


# revision identifiers, used by Alembic.
revision = '9d21de792967'
down_revision = '194a5a2a44ef'


def upgrade():
op.create_unique_constraint('ticket_event', 'ticket_holders', ['ticket_id', 'event_id'])

def downgrade():
op.drop_constraint('ticket_event', 'ticket_holders', type_='unique')

We can see that the upgrade() function has the command for adding our constraint. Once the database has been upgraded, we can revisit the schema of ticket_holders table (using the \d+ ticket_holders command again). Now we can see that our constraint is added very well in the table schema.

Now, if one tries to create multiple attendees that attend on the same ticket, s/he gets a 500 server error. Here are the related server logs:

2018-06-05 22:04:03.824 IST [46705] ERROR:  duplicate key value violates unique constraint "ticket_event"
2018-06-05 22:04:03.824 IST [46705] DETAIL:  Key (ticket_id, event_id)=(2, 6) already exists.
2018-06-05 22:04:03.824 IST [46705] STATEMENT:  UPDATE ticket_holders SET event_id=6 WHERE ticket_holders.id = 16
127.0.0.1 - - [05/Jun/2018 22:04:03] "POST /v1/attendees HTTP/1.1" 500 -
INFO:werkzeug:127.0.0.1 - - [05/Jun/2018 22:04:03] "POST /v1/attendees HTTP/1.1" 500 -

To get a more graceful error, we also need to make changes in the API schema. This will also allow to validate the data before it gets to the database. So, in the attendees.py file, we need to add a check. This check should extract the ticket and event ids from the data posted and see whether there is already an attendee in the database attending that event on the same ticket. If such an attendee is discovered, the check should raise an error and report it back to the API caller. The suitable place for this check is the before_post() method of the AttendeeListPost class. In any Flask app serving a REST API, such a method (perhaps of a different name) should exist in the API file corresponding to a model. Our check looks like the following within the before_post() method:

from flask_rest_jsonapi import ResourceList
from app.api.helpers.exceptions import ConflictException
from app.models import db
from app.models.ticket_holder import TicketHolder






class AttendeeListPost(ResourceList):
"""
List and create Attendees through direct URL
"""

def before_post(self, args, kwargs, data):
"""
Before post method to check for required relationship and proper permissions
:param args:
:param kwargs:
:param data:
:return:
"""
require_relationship(['ticket', 'event'], data)







if db.session.query(TicketHolder.id).filter_by(
ticket_id=int(data['ticket']), event_id=int(data['event'])
).scalar() is not None:
raise ConflictException(
{'pointer': '/data/attributes/ticket_id'},
"Attendee with this ticket already exists for the same event"
)

Once this check is implemented, we’re all good to go. Now, if an attendee is created that maps to a ticket belonging to an already existing attendee, the following error is sent back to the API caller:

{
"errors": [
{
"status": 409,
"source": {
"pointer": "/data/attributes/ticket_id"
},
"title": "Conflict",
"detail": "Attendee with this ticket already exists for the same event"
}
],
"jsonapi": {
"version": "1.0"
}
}

This completes our work of enforcing this constraint throughout our Flask server. This leads to a more consistent database and potentially avoids confusion at actual events!

Resources:

Continue ReadingEnforcing Constraints Throughout a Flask Back-End

Adding System Messages on Open Event Server

The Open Event Server enables organizers to manage events from concerts to conferences and meetups. It offers features for events with several tracks and venues. Event managers can create invitation forms for speakers and build schedules in a drag and drop interface. The event information is stored in a database. The system provides API endpoints to fetch the data, and to modify and update it.

The Open Event Server is based on JSON 1.0 Specification and hence build on top of Flask Rest Json API (for building Rest APIs) and Marshmallow (for Schema).

In this blog, we will talk about how to add API for accessing the System Messages on Open Event Server. The focus is on its Model updation and it’s Schema creation.

Model Updation

For the System Messages, we’ll make update model as follows

Now, let’s try to understand this Schema.

In this feature, we are providing Admin the rights to read email and notification formats used in Open Event application.

  1. First of all, there is the need to know that it has three columns notification_status, user_control_status and mail_status of type boolean.
  2. Next it has action attribute which is of type String.
  3. At last, we have hybrid properties email_message and notification_message which will return the format of email and notification respective to the action string.
  4. The hybrid properties depends on _email_message method and _notification_message method. These methods reads the MAILS and NOTIFS dictionaries and return there values corresponding to string of action key of corresponding record.

Schema Creation

For the System Messages, we’ll make our Schema as follows

Now, let’s try to understand this Schema.

In this feature, we are providing Admin the rights to read email and notification formats used in Open Event application.

  1. First of all, there is the need to know that it has three boolean properties notification_status, user_control_status and mail_status
  2. Next it has action attribute which is of type String and it’s value can be validated to have any one of the list provided in choices.
  3. At last, it has the String attributes email_message and notification_message which will return the action formats of email and notification concerning the action string provided.

So, we saw how System Messages Schema and Model is created / updated to allow Admin users to read it’s values.

Resources

Continue ReadingAdding System Messages on Open Event Server