DetachedInstanceError: Dealing with Celery, Flask’s app context and SQLAlchemy in the Open Event Server

In the open event server project, we had chosen to go with celery for async background tasks. From the official website,

What is celery?

Celery is an asynchronous task queue/job queue based on distributed message passing.

What are tasks?

The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing.

After the tasks had been set up, an error constantly came up whenever a task was called

The error was:

DetachedInstanceError: Instance <User at 0x7f358a4e9550> is not bound to a Session; attribute refresh operation cannot proceed

The above error usually occurs when you try to access the session object after it has been closed. It may have been closed by an explicit session.close() call or after committing the session with session.commit().

The celery tasks in question were performing some database operations. So the first thought was that maybe these operations might be causing the error. To test this theory, the celery task was changed to :

@celery.task(name='lorem.ipsum')
def lorem_ipsum():
    pass

But sadly, the error still remained. This proves that the celery task was just fine and the session was being closed whenever the celery task was called. The method in which the celery task was being called was of the following form:

def restore_session(session_id):
    session = DataGetter.get_session(session_id)
    session.deleted_at = None
    lorem_ipsum.delay()
    save_to_db(session, "Session restored from Trash")
    update_version(session.event_id, False, 'sessions_ver')


In our app, the app_context was not being passed whenever a celery task was initiated. Thus, the celery task, whenever called, closed the previous app_context eventually closing the session along with it. The solution to this error would be to follow the pattern as suggested on http://flask.pocoo.org/docs/0.12/patterns/celery/.

def make_celery(app):
    celery = Celery(app.import_name, broker=app.config['CELERY_BROKER_URL'])
    celery.conf.update(app.config)
    task_base = celery.Task

    class ContextTask(task_base):
        abstract = True

        def __call__(self, *args, **kwargs):
            if current_app.config['TESTING']:
                with app.test_request_context():
                    return task_base.__call__(self, *args, **kwargs)
            with app.app_context():
                return task_base.__call__(self, *args, **kwargs)

    celery.Task = ContextTask
    return celery

celery = make_celery(current_app)


The __call__ method ensures that celery task is provided with proper app context to work with.

 

Migrating FOSSASIA blog from Drupal to WordPress

Last week I migrated FOSSASIA’s blog from Drupal to WordPress and it was an amazing learning experience.

The steps one can use for migration are as follows:

Create a WordPress website:

In order to convert your drupal website to wordpress, you need to have a wordpress site where the data will be imported. By WordPress site, I mean a local installation where you can test whether the migration worked or not.

Truncate default posts/pages/comments:

Once you have your WP installation ready, truncate the default pages,comments etc from your wordpress database.

TRUNCATE TABLE wordpress.wp_comments;
TRUNCATE TABLE wordpress.wp_links;
TRUNCATE TABLE wordpress.wp_postmeta;
TRUNCATE TABLE wordpress.wp_posts;
TRUNCATE TABLE wordpress.wp_term_relationships;
TRUNCATE TABLE wordpress.wp_term_taxonomy;
TRUNCATE TABLE wordpress.wp_terms;
Selection_068
WordPress Database

Get hold of the Drupal mysql DB:

Import your Drupal DB to your local mysql installation where you have your WP database. Why? because you need to do a lot of “data transfer”!

Selection_069
Drupal Database

Execute a lot of scripts (Just kidding!):

There are some pretty useful online references which provide the required mysql scripts to migrate the data from Drupal to WordPress DB with proper formatting. Look here and here.

Depending on the kind of data you have you might need to do some modifications. e.g. depending on whether you have tags or categories/sub-categories in your data, you might have to modify the following command to suite your needs.

INSERT INTO wordpress.wp_terms (term_id, name, slug, 
term_group)
SELECT
	d.tid,
	d.name,
	REPLACE(LOWER(d.name), ' ', '-'),
	0
FROM drupal.taxonomy_term_data d
INNER JOIN drupal.taxonomy_term_hierarchy h USING(tid);

Recheck if entire data has been imported correctly:

Once you execute the scripts error free. Check if you imported the DB data (users/taxonomies/posts) correctly. Since WP and Drupal store passwords differently, you would have to ask your users/authors/admins to change their passwords on the migrated blog. We are almost there!! (not quite).

Transfer media files to WP and map them to Media:

You would have to transfer your media (pics, videos, attachments etc) to your WordPress installation from Drupal site. Selection_066

Put them under wp-content/uploads/old or any other suitable directory name under wp-content/uploads/.

Once you are done with it! In order to add the files to “Media” under Admin Panel, you can use plugins like Add from Server which map your files to folder “Media” expects your files to be in.

Change the permalinks (optional):

Depending on default permalinks of your Drupal blog, you might have to change the permalink format.

To do that, go to <Your_WP_Site>/wp-admin/options-permalink.php

You can change the permalink structure from one of the many options you are provided. Selection_067

Add themes as you may. Upload your WordPress site online. And we are done!!

The new face of blog.fossasia.org looks like this! Selection_070