How to implement JSONP to download XML resource

According to the same-origin security policy, web page can freely share images, stylesheets, scripts… but web fonts and AJAX requests are only accessible from the same domain.

Sometimes this is not desired behaviour, such as when getting a json / xml file from a different domain, using AJAX. In such cases, it all comes down to two solutions :

1- The source sets its header to allow specific domain cross-origin resource sharing (CORS). In PHP, this is done with header function :

<?php
header('Access-Control-Allow-Origin: *');

2- Use JSONP.

JSONP is short for JSON padding, a well-known “workaround” to solve CORS. It only works with GET.

To send AJAX requests using JSONP, the server must first knows how to respond to JSONP requests. It usually detects JSONP requests using callback parameter.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
<?php
// compute result
[...]

// output result
JsonpHelper::outputXML($result);

// json helper class - credits : @andibraeu
class JsonpHelper {
	private static function is_valid_callback($subject)
	{
	    $identifier_syntax
	      = '/^[$_\p{L}][$_\p{L}\p{Mn}\p{Mc}\p{Nd}\p{Pc}\x{200C}\x{200D}]*+$/u';
	    $reserved_words = array('break', 'do', 'instanceof', 'typeof', 'case',
	      'else', 'new', 'var', 'catch', 'finally', 'return', 'void', 'continue', 
	      'for', 'switch', 'while', 'debugger', 'function', 'this', 'with', 
	      'default', 'if', 'throw', 'delete', 'in', 'try', 'class', 'enum', 
	      'extends', 'super', 'const', 'export', 'import', 'implements', 'let', 
	      'private', 'public', 'yield', 'interface', 'package', 'protected', 
	      'static', 'null', 'true', 'false');
	    return preg_match($identifier_syntax, $subject)
	        && ! in_array(mb_strtolower($subject, 'UTF-8'), $reserved_words);
	}

	public static function outputXML($string) {
		# JSON if no callback
		if( ! isset($_GET['callback']) )
		    exit($string);
		$string = str_replace("\n", " ", str_replace("'", '"', $string));
		# JSONP if valid callback
		if(JsonpHelper::is_valid_callback($_GET['callback']))
		    exit("{$_GET['callback']}('$string')");
		# Otherwise, bad request
		header('status: 400 Bad Request', true, 400);
	}
}

Line 29 is dedicated to replace all single-quote characters to double-quote, and new lines to simple space. This is to be able to pass xml document as valid javasript string parameter.

Then, the client can use jQuery to get the document from a different domain (it works on same domain as well, which is convenient for testing & development) :

1
2
3
4
5
6
7
8
9
$.ajax({
	url : source_url,
	dataType : 'jsonp',  // if dataType is set to 'jsonp' 
			    // a 'callback' parameter will be appended to the request url
	success: function(data) {
		$data = $($.parseXML(data));
		items = $data.find('item');
	}
});

Something worth noticing on the client side is on line 5 : the data needs to be converted to xml first, with $.parseXMLbefore converting to jQuery object. Converting the result string directly to jQuery won’t work : $data = $(data);.

That’s about it ! To know more about JSONP, please refer to its wiki https://en.wikipedia.org/wiki/JSONP.

References :

API Integration

My FOSSASIA project is taking a big transitioning step from the starting plan. In milestone 3 and 4, rather than focusing on statistical retrieval and API documentation wepapp, my mentors – Mario (@mario) and Andreas (@andibraeu) are guiding me towards integrating fossasia api data in other services. The main goal remains the same though : getting more people to use and to know about the API.

Integrating our set of data into external services can be tricky, but this is also what makes the task so exciting. Currently, we’re already started the integration process into loklak, a distributed searching & harvesting server. You can find out more about loklak via their well detailed about page. But the plan doesn’t stop there. We’re targeting popular web platforms : WordPress, Drupal, Github Pages.. so that users can access our data with ease, and use our services / plugins in their websites within simple steps. That would be a big win for us, so I’m very impatient to get going with the code to brings these ideas to life.

References :

Get Facebook comments with large user pictures

On the /user edge, user pictures url are not returned by default, and when users ask for it, the response image is not in biggest format. This query should get the largest user pictures associated with post :

	/user{picture.type(large)}

There are other use cases where this syntax applies as well

  • get comments of a feed
	/feed?fields='from{picture.type(large)}'
  • get feeds of an user
	/userid/feed?fields={from{picture.type(large)}}

Examples tested with Facebook Graph API > v2.0, using Graph Explorer and PHP SDK.

A glance at FOSSASIA Calendar API

I am a frequent user of popular social platforms API : Facebook, Twitter, Instagram.. Among others, Facebook Graph API is the API I’m most familiarized with, and despite its lack of documentation, Graph API makes me from time to time admire the engineering mind of those who built it. It’s simple, has a clear design, and communicates a consistent philosophy. I inspired a lot from it to implement Calendar API, a simple web service at FOSSASIA.

Why naming it API, and not web service ? I don’t know, I was always confused about the two, and as far as I concerns, they are the same. The only difference might be the word ‘web’ that tights web service to HTTP, and the scenario involving two distant machines, while API is somewhat more general (an API call of an in-app library for example).

This is my first time building a web service, and so far there wasn’t any technical challenge : it’s just simple & straightforward PHP script with echo $results; in the end. Mainly because the hard part is still ahead : getting people to know about it, and maintaining it while there are (hopefully) enough clients who care to yell at API developers when they do something wrong. The API I built supposed to parse the merged ics file of all communities event feeds, and expose it to users, in various formats, with filters and options.

Don’t worry if the last sentence doesn’t make sense. We’ll dive in the details right away.

There are currently two possible formats : .ics and .json. Users set the format with format parameter, for e.g. format=icsics format is suited for developers who want to import events to their calendar app / wordpress plugin. json format is for front-end developers who want to build a web / mobile web without the hassle of parsing ics feed.

A few filter parameters are possible to refine search results : source to filter by community, fromto to filter by events time, limit to limit number of events returned, and fields to custom select event fields. Finally, sort parameter is to order the results by certain criteria. Currently only datetime ordering (ascending or descending) are available.

As I said, the Calendar API is inspired a lot from Facebook API. Those are the ideas I pickpocketed from their design :

  • multiple values parameter. With source and fields, multiple values can be provided, separated using comma.
  • fields parameter. The motivation for event field filter is because the event has lots of different fields, and different users are interested in different information.

There is also the “linked edge” characteristic of Graph API that I find very interesting : connection between related data make them more useful. It is done that way mainly because the API reflects Facebook own data model, but there is nothing thats prevents unconnected data to be presented in a connected way. So I’ll hope to implement something similar for FOSSASIA Calendar API in the futur.

(Right now the API serves events from Freifunk community, not FOSSASIA, mostly because there are not enough event feeds from FOSSASIA community to collect. So if you want to help us, you can add ics feed of your open-source community to FOSSASIA API. Here’s how).

References :

Calendario: Another JS calendar library

Calendario is a lightweight calendar library that I happen to find on the net. Their approach to calendaring solution is rather unique : it’s not a full-blown library like jQuery FullCalendar where everthing is ready to be used. With Calendario, calling init function simply generates plains html source code. The styling part is completely up to user.

Another plus is it’s lightweight (350 lines of code more or less). It’s easy to dive into the source and make advanced adaptations to specific use-case.

But what’s really remarkable is their examples. As I said the styling is out-sourcing to user, and they provide stunning examples that demonstrate how to do it. For me (a developer “designing” stuffs), readily beautiful examples are a crucial selling point.

But (there’s always a but..), Calendario is very poorly documented. A pointer to the same blog post all over again is not enough ! I was very confusing about how things work at first, especially about the data model. This blog contains what I found out (rather painfully) about Calendario library :

Initialization

<div id="calendar_id"></div>
<script>
	$('#calendar_id').calendario(options);
</script>

Available options

Option Purpose* Default
weeks weekday labels [‘Sunday’, ‘Monday’, ‘Tuesday’, ‘Wednesday’, ‘Thursday’, ‘Friday’, ‘Saturday’]
weekabbrs weekday short labels [‘Sun’, ‘Mon’, ‘Tue’, ‘Wed’, ‘Thu’, ‘Fri’, ‘Sat’]
months month label [‘January’, ‘February’, ‘March’, ‘April’, ‘May’, ‘June’, ‘July’, ‘August’, ‘September’, ‘October’, ‘November’, ‘December’]
monthabbrs monday short labels [‘Jan’, ‘Feb’, ‘Mar’, ‘Apr’, ‘May’, ‘Jun’, ‘Jul’, ‘Aug’, ‘Sep’, ‘Oct’, ‘Nov’, ‘Dec’]
displayWeekAbbr display week short labels false
displayMonthAbbr display month short labels false
startIn 1
fillEmpty true
zone ‘00:00’
events [[‘click’], [focus’]]
checkupdate true
weekdays ‘MON, TUE, WED, THU, FRI’
weekends ‘SAT, SUN’
feed http://calendario.t15.org/sync/
month
year
caldata calendar events data

Most of them you can safely ignore, I enlisted them all for the sake of completeness. Checkout the next section for interesting ones.

Option format

caldata

A list of json objects. The list’s keys and values contain following information :

  • key : the event date in MM-DD-YYYY format
  • value : an array of event values that occured on the given date. Each case of the array can have following fields :
Option Purpose Value type
content title of the event string
url Link to event page string
note added note for the event, will be included in the html as inside tag string
startday string, in MM-DD-YYYY format
endday string, in MM-DD-YYYY format
allDay true for all day event boolean

Examples

var events = {
	'06-06-2015' : [{content: 'Important Meeting at 4', startTime: '04:00', endTime: '06:00', note : 'Everyone please gather at Markketing Department'}],
	'06-07-2015' : [{content : 'First event of July 7', allDay : true}, 
	                {content : 'Second event of July 7', allDay : true}
	               ],
	'07-01-2015' : [{content: 'All day event', allDay: true, note : 'So loongg !'}]
}

Wrap-up : Pros and cons of Calendario

  • Pros :
    • lightweight
    • fully customizable design
    • beautiful examples
  • Cons
    • poor documentation
    • lots of work undone, not recommended for those who want a powerful library that just work

That’s about it ! Found something wrong, or just want to help completing this unofficial calendario documentation ? Send me a PR here

*NOTE : this documentation is based on Calendario v4.0.0 https://github.com/deviprsd21/Calendario

References :

Optimize React-Redux App

React is advertised as quite fast out of the box. In extreme cases when your app hits performance problem, you just have to optimize the re-rendering process by shallow comparing React props and state inside shouldComponentUpdatelifecycle function. The strategy for optimizing seems quite straight-forward.

However I found that this strategy is less clear in a React/Redux environment. Here are some advices that you should considering when dealing with your app performance:

Use Immutable data

This one should be a no-brainer with any React app. shouldComponentUpdate can only be useful when all the props can be shallow-compared. So as a rule of thumb, your components props can only be of 3 types:

  • Immutable
  • Primitive type (number, boolean, etc.)
  • Function type (e.g. callback event)

I didn’t mention how you should handle React state. That is because Redux uses a separate store as application state. But sometimes, you need to use both React’s state and Redux’s store. In that case, both props and state are shallow-compared so state elements should follow mentioned rules as well.

Do not use ownProps

When you create stateful components (containers) using connect function from react-redux library, shouldComponentUpdate is declared implicitly, so you normally don’t have to implement it yourself.

However the implementation of shouldComponentUpdate is such that if you uses ownProps in your mapStateToProps or mergeProps function, it will always return false, so the component will always be updated.

This means that if you want your component to be optimized, you can’t use information from passed props to get other information from the store. Why the implementation has to be this way is still unclear to me.

Avoid connect on frequently-called elements

connect function wraps the component inside another component with lots of decorated elements. These heavy overheads should be avoided on components that are called a lot in your application, for e.g. an element of a list.

Use Performance Tools

Finally, here are some excellent tools from the React team. Use them to identify the bottleneck in your React application:

FOSSASIA API calendar service

ICalendar is a file format residing under RFC 5445, to represent and exchange calendaring and scheduling information. The file extension is generally .ics, and .iCal for Apple software. The aim was to define a standard format that could be understood everywhere, independent of calendar vendors : Google Calendar, Apple iCal, Yahoo! Calendar, Mozilla Lightning, Microsoft Outlook… Nevertheless, true interoperability doesn’t seem to be the reality. My next task at FOSSASIA is to this circumvent this situation and create a calendar service / vizualisation from ICalendar sources provided by different open-source communities, hence probably from different vendors and format deviations.

To have an idea of the problem I’m dealing with, I gathered a bunch of ics files, mostly provided by freifunk communities, using these little node scripts, and try to spot a not-so-standard implementation. To have a less mechanical approach, I developed a visual tool to debug my ics. Here are some screenshots of the app:

Ics debugging tool screenshot 1Screenshot 1

Screenshot 2

Ics debugging tool screenshot 1

The tool is fairly simple : you copy the ics text into the text area (or choose between example files), press Feed me !, the system will get running and provide some outputs. There is a general info panel, a yearly heatmap, and a calendar. It is a great help for me since I can just switch between ics files, see all the events counts in the heatmap, and event detailed information in the calendar.

Source code & demo version coming soon ! Built with ics parser library https://github.com/johngrogg/ics-parser, CalHeatmap https://kamisama.github.io/cal-heatmap/ and jquery plugin FullCalendar.io.

Update :

Source : https://github.com/zyzo/ics-tools

Demo : http://fossasia-api.herokuapp.com/

References :

Scrum journey

Exception, perception, and knowledge are things that fade away. The truth is that your memory betrays you all the time by erasing (without asking you a word !) all your experience about things that you used to know. If you are old enough to forget how awesome your childhood was, and are frustrating because you do, you’ll understand what I mean.

The problem is, experience is often very valuable, even if it is about how you felt being ignorant. For instance, as a teacher, you want to know what it’s like to know nothing about a subject, so you can deliver it to your students, in a less boring or overwhelming way. As a framework designer, if you want to strive for easy integration and small learning curve, you should at first know what small learning curve and easy integration look like for newbies.

I’m not going off topic any further. The reason I’m saying all this is because I’m learning a whole new concept that I have absolutely zero idea about: scrum mehodology. This framework is applied in my gsoc project at FOSSASIA. I don’t have all the data yet, but my first impression is that this is a hell of a subject that would deeply impact me in my work life.

Therefore I’ll use this blog as a diary to log all my experience in my scrum’s discovery journey, so when I’m skillful enough, I can look back at this and remember what it’s like to be ignorant again. I’ll also post useful links and stuffs that help me, so I hope this blog will benifits others as well.

As consequence, this blog will likely be updated in the future. I don’t know when, but it will be. Unless I (am very unlikely to) become a scrum master with no more stuff to learn.

Image broken
Image credit : http://www.agilelearninglabs.com/2012/02/what-does-a-scrum-master-do/

Outline

Day 1 : What the heck is scrum, and scrum questions ?

Scrum is a version of agile programming, as opposed to “waterfall” programming where all specifications are planned in the beginning and all deliverables in the end. My first impression about scrum is, it doesn’t bring anything new to the table. Smaller release cycle ? Daily team communication ? Sprint-based/task-based development ? More client interaction ? These are useful tips/guidelines for sure, but for a developer, this sounds quite trivial and just like common sense. Maybe from a manager’s point of view, it might looks more like a fresh idea. But if that’s all agile programming is about, then it might not worth the hassle.

My mentors first asked me what my scrum questions are. I was like, questions about scrum ? Yes please, I have tons of them, because I know nothing yet ! But, what scrums questions really mean is simple questions that each member of the team has to answer in each daily standup meeting. It is intended to keep the meeting short (< 15 min) and concise. The most basic line of scrum questions is these three :

  1. What did you do yesterday?
  2. What will you do today?
  3. What obstacle did you find in your way?

The meeting is intended to help team members to know about others’ work, and to (eventually) help them . Most importantly, it’s about making commitments (“I intend to complete this task today, but if I don’t, please don’t look down on me!”). That way the next day others developers will certainly know if the developper did or did not achieve what they intended, and why.

Image broken

So far so good. The way I see it, the daily standup idea has good faith in itself : it encourages just enough amount of team communication. When I was at OnePoint, where waterfall methodology is used, and no team interaction protocol is enforced, I feel like this is exactly the thing we needed.

In my next update that I don’t know when, I’ll probably talk about scrum in my one-man-land situation, that is, a single developer in the development team.

References :

GSoC 2015 with FOSSASIA : the beginning

I was choosen as a student participant to Google Summer of Code 2015.

It’s a program organized by Google that offers students stipends to write code for open source projects. Being selected is a big deal for me since GSoC is a vey selective program, and a meaningful way to spend my summer, by doing what I very much enjoy : coding. What’s more, students get paid doing it. I’m not denying that the $5500 stipend per student is really sexy (at least for me). It’s way more than most software interns in France get for a three months period.

I just knew about GSoC two months ago thanks to a teacher. Before that, I have very little notion about open-source software, and none about GSoC. I cure my ignorance since then with a lot of interesting blogs and definitions. Based on the information I’ve been fed with, I could now say open-source is great. From the company’s point at view, it’s a great form of advertising. For the project, open-source often means better code and documentation. The developers knows the whole world has access to the source code, so they’ll probably take better care of it. Also, open-sourcing is a must if you want to create a vibrant community and ecosystem.

Open-source is great, getting involved, even better. With GSoC you have the opportunity to work on an open-source project with mentoring support from the core developers who are maintaining it themselves. Awesome ! To get into the program as a student, you have to choose betweens accepted mentoring organizations and propose a project idea suited to them. I was particularly attracted to FOSSASIA, an open-source event community in Asia, and their project idea Extend toolbox for FOSSASIA and Freifunk API. The Freifunk/FOSSASIA API is designed to solve the problem of collecting data from different communities in a decentralized way. It makes some interesting parts of the FOSSASIA community site possible, e.g. the Common Map.

Image broken FOSSASIA logo

I think the toolbox has a great potential to create a strong network between different communities at different locations, by providing attractive statistics visualization. With that spirit in mind, my project’s proposal aims to add new components and improve old ones. Here are some of them :

  • Improve the json generator page
  • Calendar service
  • Community news / events panel
  • Common Map filters :
    • by country
    • by technical area
  • Statistic service / charts :
    • communities counts
    • active period panel based on news/events date (like the contribution panel on github)
  • API Documentation web app
  • etc.

In my gsoc blog series, I’ll cherry-pick sutffs that I find valuable from my GSoC project to show to the world. So if you’re interested on what I’ll do, stick around for upcoming updates !

References :

Embed Leaflet map as coordinates picker

As a warm up task for my GSoC project, I want to embed a map to the FOSSASIA API generator form as a convenient way for users to provide their location coordinates. When the user clicks on a location on the map, two input fields Latitudeand Longitude will be automatically updated with fresh values. Here’s how.

For this task you need jquery and leaflet – an openstreetmap library. You also want to prepare an empty div for the embed map :

<html>
<head>
    <link rel="stylesheet" href="http://cdn.leafletjs.com/leaflet-0.7.3/leaflet.css"/>
</head>
<body>
    <label for="latInput">Latitude</label>
    <input id="latInput"/>
    <label for="lngInput">Longitude</label>
    <input id="lngInput"/>
    <div id="map" style="height : 200px"></div>
    <script src="http://cdn.leafletjs.com/leaflet-0.7.3/leaflet.js"></script>
    <script src="https://code.jquery.com/jquery-1.11.3.min.js"></script>
<body>
</html>

Now, let’s instanciate the div using the leaflet API. A tile layer must be provided, in this example I’m using mapbox but there are other tile providers as well :

var mapCenter = [22, 87];
var map = L.map('map', {center : mapCenter, zoom : 3});
L.tileLayer('https://{s}.tiles.mapbox.com/v3/{id}/{z}/{x}/{y}.png', {
    maxZoom: 18,
    attribution: 'Map data &copy; <a href="http://openstreetmap.org">OpenStreetMap</a> contributors, ' +
    '<a href="http://creativecommons.org/licenses/by-sa/2.0/">CC-BY-SA</a>, ' +
    'Imagery © <a href="http://mapbox.com">Mapbox</a>',
    id: 'examples.map-i875mjb7',
    noWrap : true
}).addTo(map);

When the user clicks on the map, I want to point a marker to their choice and display a popup with choosen coordinates. That’s what updateMarker function is good for :

var marker = L.marker(mapCenter).addTo(map);
var updateMarker = function(lat, lng) {
    marker
        .setLatLng([lat, lng])
        .bindPopup("Your location :  " + marker.getLatLng().toString())
        .openPopup();
    return false;
};

The last missing piece is, on map click event, updating the lat and lng inputs :

map.on('click', function(e) {
    $('#latInput').val(e.latlng.lat);
    $('#lngInput').val(e.latlng.lng);
    updateMarker(e.latlng.lat, e.latlng.lng);
});

Now if you need the other way around – enter manually the input fields to update the marker, it could be easily done as well, by capturing a useful event associated with HTML input tag – the… input event :

var updateMarkerByInputs = function() {
	return updateMarker( $('#latInput').val() , $('#lngInput').val());
}
$('#latInput').on('input', updateMarkerByInputs);
$('#lngInput').on('input', updateMarkerByInputs);

That’s about it ! Check out this tutorial on Gist and JSFiddle.

References :