Implementing Advanced Search Feature In Susper

Susper has been provided ‘Advanced Search’ feature which provides the user a great experience to search for desired results. Advanced search has been implemented in such a way it shows top authors, top providers, and distribution regarding protocols. Users can choose any of these options to get best results.

We receive data of each facet name from Yacy using yacy search endpoint. More about yacy search endpoint can be found here:  http://yacy.searchlab.eu/solr/select?query=india&fl=last_modified&start=0&rows=15&facet=true&facet.mincount=1&facet.field=host_s&facet.field=url_protocol_s&facet.field=author_sxt&facet.field=collection_sxt&wt=yjson

For implementing this feature, we created Actions and Reducers using concepts of Redux. The implemented actions can be found here: https://github.com/fossasia/susper.com/blob/master/src/app/actions/search.ts

Actions have been implemented because these actually represent some kind of event. For e.g. like the beginning of an API call here.

We also have created an interface for search action which can be found here under reducers as filename index.ts: https://github.com/fossasia/susper.com/blob/master/src/app/reducers/index.ts

Reducers are a pure type of function that takes the previous state and an action and returns the next state. We have used Redux to implement actions and reducers for the advanced search.

For advanced search, the reducer file can be found here: https://github.com/fossasia/susper.com/blob/master/src/app/reducers/search.ts

The main logic has been implemented under advancedsearch.component.ts:

export class AdvancedsearchComponent implements OnInit {
  querylook = {}; // array of urls
  navigation$: Observable<any>;
  selectedelements: Array<any> = []; // selected urls by user
changeurl
(modifier, element) {
// based on query urls are fetched
// if an url is selected by user, it is decoded
  this.querylook[‘query’] = this.querylook[‘query’] + ‘+’ + decodeURIComponent(modifier);
  this.selectedelements.push(element);
// according to selected urls
// results are loaded from yacy
  this.route.navigate([‘/search’], {queryParams: this.querylook});
}

// same method is implemented for removing an url
removeurl(modifier) {
  this.querylook[‘query’] = this.querylook[‘query’].replace(‘+’ + decodeURIComponent(modifier), );

  this.route.navigate([‘/search’], {queryParams: this.querylook});
}

 

The changeurl() function replaces the query with a query and selected URL and searches for the results only from the URL provider. The removeurl() function removes URL from the query and works as a normal search, searching for the results from all providers.

The source code for the implementation of advanced search feature can be found here: https://github.com/fossasia/susper.com/tree/master/src/app/advancedsearch

Resources

Designing A Remote Laboratory With PSLab: execution of function strings

In the previous blog post, we introduced the concept of a ‘remote laboratory’, which would enable users to access the various features of the PSLab via the internet. Many aspects of the project were worked upon, which also involved creation of a web-app using EmberJS that enables users to create accounts , sign in, and prepare Python programs to be sent to the server for execution. A backend APi server based on Python-flask was also developed to handle these tasks, and maintain a postgresql database using sqlalchemy .

The following screencast shows the basic look and feel of the proposed remote lab running in a web browser.

This blog post will deal with implementing a way for the remote user to submit a simple function string, such as get_voltage(‘CH1’), and retrieve the results from the server.

There are three parts to this:
  • Creating a dictionary of the functions available in the sciencelab instance. The user will only be allowed access to these functions remotely, and we may protect some functions as the initialization and destruction routines by blocking them from the remote user
  • Creating an API method to receive a form containing the function string, execute the corresponding function from the dictionary, and reply with JSON data
  • Testing the API using the postman chrome extension
Creating a dictionary of functions :

The function dictionary maps function names against references to the actual functions from an instance of PSL.sciencelab . A simple dictionary containing just the get_voltage function can be generated in the following way:

from PSL import sciencelab
I=sciencelab.connect()
functionList = {'get_voltage':I.get_voltage}

This dictionary is then used with the eval method in order to evaluate a function string:

result = eval('get_voltage('CH1')',functionList)
print (result)
0.0012

A more efficient way to create this list is to use the inspect module, and automatically extract all the available methods into a dictionary

functionList = {}
for a in dir(I):
	attr = getattr(I, a)
	if inspect.ismethod(attr) and a!='__init__':
		functionList[a] = attr

In the above, we have made a dictionary of all the methods except __init__

This approach can also be easily extrapolated to automatically generate a dictionary for inline documentation strings which can then be passed on to the web app.

Creating an API method to execute submitted function strings

We create an API method that accepts a form containing the function string and option that specifies if the returned value is to be formatted as a string or JSON data. A special case arises for numpy arrays which cannot be directly converted to JSON, and the toList function must first be used for them.

@app.route('/evalFunctionString',methods=['POST'])
def evalFunctionString():
    if session.get('user'):
        _stringify=False
        try:
            _user = session.get('user')[1]
            _fn = request.form['function']
            _stringify = request.form.get('stringify',False)
            res = eval(_fn,functionList)
        except Exception as e:
            res = str(e)
        #dump string if requested. Otherwise json array
        if _stringify:
            return json.dumps({'status':True,'result':str(res),'stringified':True})
        else:
            #Try to simply convert the results to json
            try:
                return json.dumps({'status':True,'result':res,'stringified':False})
            # If that didn't work, it's due to the result containing numpy arrays.
            except Exception as e:
                #try to convert the numpy arrays to json using the .toList() function
                try:
                    return json.dumps({'status':True,'result':np.array(res).tolist(),'stringified':False})
                #And if nothing works, return the string
                except Exception as e:
                    print( 'string return',str(e))
                    return json.dumps({'status':True,'result':str(res),'stringified':True})
    else:
        return json.dumps({'status':False,'result':'unauthorized access','message':'Unauthorized access'})
Testing the API using Postman

The postman chrome extension allows users to submit forms to API servers, and view the raw results. It supports various encodings, and is quite handy for testing purposes.Before executing these via the evalFunctionString method, user credentials must first be submitted to the validateLogin method for authentication purposes.

Here are screenshots of the test results from a ‘get_voltage(‘CH1’)’ and ‘capture1(‘CH1’,20,1)’ function executed remotely via postman.

 

Our next steps will be to implement the dialog box in the frontend that will allow users to quickly type in function strings, and fetch the resultant data

Resources: