This sqooping process is to be done at the beginning of every month, and it is done since the beginning of time every time. It contains localized namespaces for every wiki (see namespace mapping script below). In addition to that, another table is created in Hadoop: namespace_mapping. Finally some special case sqoop jobs are used to get the cu_changes and discussiontools_subscription tables from the production replicas. Tables imported from the production replicas are: actor, comment, watchlist. The MediaWiki tables imported from the public replicas are archive, category, categorylinks, change_tag, change_tag_def, content, content_models, externallinks, image, imagelinks, ipblocks, ipblocks_restrictions, iwlinks, langlinks, logging, page, pagelinks, page_props, page_restrictions, redirect, revision, slots, slot_roles, templatelinks, user, user_groups, user_properties, wbc_entity_usage.

The main script uses apache sqoop to import some MediaWiki tables from publicly available databases replica and production replicas into Analytics' Hadoop cluster.

After that, next steps in the pipeline will process that data within the cluster and generate the desired output. This is done using a couple of scripts stored in the analytics-refinery-source repository. This page describes the first step of the edit history reconstruction pipeline: the loading of MediaWiki data into Hadoop.

Mediawiki table generator