Uploading Files via APIs in the Open Event Server

There are two file upload endpoints. One is endpoint for image upload and the other is for all other files being uploaded. The latter endpoint is to be used for uploading files such as slides, videos and other presentation materials for a session. So, in FOSSASIA’s Orga Server project, when we need to upload a file, we make an API request to this endpoint which is turn uploads the file to the server and returns back the url for the uploaded file. We then store this url for the uploaded file to the database with the corresponding row entry. Sending Data The endpoint /upload/file  accepts a POST request, containing a multipart/form-data payload. If there is a single file that is uploaded, then it is uploaded under the key “file” else an array of file is sent under the key “files”. A typical single file upload cURL request would look like this: curl -H “Authorization: JWT <key>” -F file=@file.pdf -x POST http://localhost:5000/v1/upload/file A typical multi-file upload cURL request would look something like this: curl -H “Authorization: JWT <key>” -F files=@file1.pdf -F files=@file2.pdf -x POST http://localhost:5000/v1/upload/file Thus, unlike other endpoints in open event orga server project, we don’t send a json encoded request. Instead it is a form data request. Saving Files We use different services such as S3, google cloud storage and so on for storing the files depending on the admin settings as decided by the admin of the project. One can even ask to save the files locally by passing a GET parameter force_local=true. So, in the backend we have 2 cases to tackle- Single File Upload and Multiple Files Upload. Single File Upload if 'file' in request.files: files = request.files['file'] file_uploaded = uploaded_file(files=files) if force_local == 'true': files_url = upload_local( file_uploaded, UPLOAD_PATHS['temp']['event'].format(uuid=uuid.uuid4()) ) else: files_url = upload( file_uploaded, UPLOAD_PATHS['temp']['event'].format(uuid=uuid.uuid4()) ) We get the file, that is to be uploaded using request.files[‘file’] with the key as ‘file’ which was used in the payload. Then we use the uploaded_file() helper function to convert the file data received as payload into a proper file and store it in a temporary storage. After this, if force_local is set as true, we use the upload_local helper function to upload it to the local storage, i.e. the server where the application is hosted, else we use whatever service is set by the admin in the admin settings. In uploaded_file() function of helpers module, we extract the filename and the extension of the file from the form-data payload. Then we check if the suitable directory already exists. If it doesn’t exist, we create a new directory and then save the file in the directory extension = files.filename.split('.')[1] filename = get_file_name() + '.' + extension filedir = current_app.config.get('BASE_DIR') + '/static/uploads/' if not os.path.isdir(filedir): os.makedirs(filedir) file_path = filedir + filename files.save(file_path) After that the upload function gets the settings key for either s3 or google storage and then uses the corresponding functions to upload this temporary file to the storage. Multiple File Upload elif 'files[]' in request.files:…

Continue ReadingUploading Files via APIs in the Open Event Server

Deploying Yacy with Docker on Different Cloud Platforms

To make deploying of yacy easier we are now supporting Docker based installation. Following the steps below one could successfully run Yacy on docker. You can pull the image of Yacy from https://hub.docker.com/r/nikhilrayaprolu/yacygridmcp/ or buid it on your own with the docker file present at https://github.com/yacy/yacy_grid_mcp/blob/master/docker/Dockerfile One could pull the docker image using command: docker pull nikhilrayaprolu/yacygridmcp   2) Once you have an image of yacygridmcp you can run it by typing docker run <image_name>   You can access the yacygridmcp endpoint at localhost:8100 Installation of Yacy on cloud servers: Right now installation yacy on cloud servers is documented at https://github.com/nikhilrayaprolu/yacy_grid_mcp/tree/documentation/docs/installation We have documentation provided for hosting yacy on Google Cloud, AWS, Bluemix and digital Ocean and Heroku. Installing Yacy and all microservices with just one command: One can also download,build and run Yacy and all its microservices (presently supported are yacy_grid_crawler, yacy_grid_loader, yacy_grid_ui, yacy_grid_parser, and yacy_grid_mcp ) To build all these microservices in one command, run this bash script productiondeployment.sh `bash productiondeployment.sh build` will install all required dependencies and build microservices by cloning them from github repositories. `bash productiondeployment.sh run` will run all services and starts them. Right now all repositories are cloned into ~/yacy and you can make customisations and your own changes to this code and build your own customised yacy. The related PRs of this work are https://github.com/yacy/yacy_grid_mcp/pull/21 and https://github.com/yacy/yacy_grid_mcp/pull/20 and https://github.com/yacy/yacy_grid_mcp/pull/13 Resources: Docker documentation: https://docs.docker.com/ Deployment to Google Cloud: https://engineering.hexacta.com/automatic-deployment-of-multiple-docker-containers-to-google-container-engine-using-travis-e5d9e191d5ad Writing bash script http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html

Continue ReadingDeploying Yacy with Docker on Different Cloud Platforms