July 9, 2014
Site Migration? 32 Problems & Solutions To Keep You Out Of Trouble
There are several reasons you may be considering migrating a website including the following;
- Re-branding requiring new domain
- Expansion into foreign markets requiring new TLDs
- Upgrade/redesign leading to a new navigation or URL structure
- Re-purposing of content on new website
Restructuring or moving a website is one of the most dangerous things you can do from an SEO point of view if not handled correctly and we strongly recommend that you use specialists if you are not completely sure how to do it.
Why? What Could Go Wrong?
If you don’t implement redirects when you move a page or whole domain to a new URL the search engines may not index the new page/site quickly and you could lose traffic in the meantime. Additionally if the old page no longer exists, search engines will direct visitors to a 404 error page which could do harm to the user experience and your online reputation. Lastly, any authority (or “link juice”) going to the original page will also be lost if not properly rerouted to the new page..
To set the tone of the article we shall look at what redirects are (1) and how to implement them (2), followed by the necessary steps for carrying out a migration (3).
The real value however comes in part (4) where we discover all the things that could possibly go wrong and what to be aware of – all derived from real experiences! (Feel free to skip to section 4 if you wish – I won’t take it personally!)
- 1. What is a Redirect?
- 2. Types of Redirects
- 3. Site Migration Steps
- 4. 32 Things You May Not Have Considered
If you have ever moved house or your business to a new address you will have needed to redirect your mail so it doesn’t pile up at your old address. A URL redirect is similar in that it will automatically reroute traffic coming to your site to your new address and is at the heart of a migration project.
Depending on what type of server your site is on (Apache or Windows Microsoft Internet Information Services (IIS)), redirects take different forms which we will look at in section 2.
It is worth mentioning at this point that there are several types of redirect for different situations.
A 301 redirect is a permanent redirect which tells the search engines “this page/site has moved permanently to a new location – please de-index this page and reroute all traffic/link authority to the new page immediately”.
A 302 redirect is a temporary redirect (where the original page remains indexed) which should only be used if you are likely to resort back to the original page in the future. The drawbacks of using 302 redirects mistakenly is that the search engines might misunderstand the new page as duplicate content (and therefore not index it) or dilute link-juice across the two versions. Having said this, the top search engines are often smart enough to judge if a mistake has been made.
Moving Single URLs
You may need to move an existing page to a different area of your site (for example, moving a product to a new product category) which means it now sits in a new location. In the example below we are selling off the last remaining red tshirts and have moved them all to a sale category permanently until sold out.
www.your-website.com/tshirts/red-tshirt to www.your-website.com/sale/red-tshirt
In this case you need only create a page-level redirect which looks like this:
Apache Format (the most common, added to the .htaccess file):
Microsoft IIS Format (a little trickier, added to the URL Rewrite module):
There are no limits to the amount of URLs you can include. Remember to validate each redirect by trying to access each old page.
In the interest of keeping this this document streamlined we will explain other redirects from an Apache point of view as this is the most common type. The principals for Microsoft IIS redirects are similar and many tutorials can be found on the internet.
Rewritten URLs should be treated like moved pages. If you want to create a new URL for an existing page you must redirect the old URL to the new one as above. In the example below we have streamlined the URL of this product by removing the word “cotton”.
e.g. www.your-website.com/tshirts/red-cotton-tshirt to www.your-website.com/tshirts/red-tshirt
Deleted pages should be treated like moved pages as above. Always redirect the deleted page to the nearest equivalent page or a suitable alternative location. Sometimes there won’t be an equivalent page so you may need to redirect to the category or even the homepage.
In the example below, the red tshirt has sold out and we will no longer be stocking red tshirts. We could redirect this to a green or blue tshirt equivalent but in this case I chose to redirect to the tshirt category instead.
e.g. www.your-website.com/tshirts/red-tshirt to www.your-website.com/tshirts
Moving Whole Directories/Sub-Folders
If you are staying on the same domain and simply moving pages to new directories/sub-folders or restructuring you will need to redirect all of the URLs in one folder to the other.
e.g. www.your-website.com/clothes/tshirts to www.your-website.com/tshirts
There may be times where you need to move an entire domain, for example if you are rebranding or if you have bought a better domain name. In this case you are specifying that everything including and beyond the root directory (indicated by a forward slash) should be redirected
NOTE: Do not use a site-wide redirect if you are setting up a new site to escape a search engine penalty. This will simply transfer the penalty to the new domain.
My colleague Piyush wrote an extensive article on redirects which can be found here.
Now that we understand how the various redirection commands work lets have a look at the full migration process; This involves grabbing all existing URLs and mapping them to the new equivalents which tells the search engines about the move and how to find the new pages.
1. Take a full scan of the existing site and gather all existing URLs.
We recommend using a tool like Screaming Frog (Note: the free version limits you to 500 URLs so if you have a large site, consider purchasing an upgraded subscription).
2. Find additional URLs.
Sometimes Screaming Frog will miss out orphaned pages or pages from an old version of the site that may be indexed in Google. For this reason you should also scrape all the URLs in Google by using a [site:www.YOURWEBSITE.com] command (see screenshot below). Also take URLs from the XML sitemap and take this chance to clean up 404s in WMT. Combining these lists in Excel and removing duplicates will give you a full list to work with.
3. Map all existing URLs with the new equivalents.
This can be time-consuming if you have long list of URLs – unfortunately there is no way round this. However, if the migration simply concerns a change of domain with the URL structure staying the same, you can perform a one-off domain redirect which can save tonnes of time!
4. Upload the redirects and new sitemap.
Once the migration takes place you want to upload the redirects as quickly as possible so the file should be ready to upload prior to launch of the new site. At this point you will also want to upload the updated XML sitemap ready for when the search engine bots crawl the new site for the first time. If the URL of the sitemap has changed, make sure you notify Google and Bing in Webmaster Tools. You may also wish to force a crawl of the new site and structure if you want the site to be picked up quickly.
5. Monitor and react to any issues that arise.
Manually checking the redirects work upon upload and monitoring Webmaster Tools on an on-going basis for the next few weeks should quickly alert you to any issues with your redirect file. Issues with the migration could severely impact the success of your new website launch so this is an incredibly important stage. Thankfully, Webmaster Tools makes things easy for you.
6. Change directory and other links where possible.
Starting with your most important referral sources (see Google Analytics) and highest PA links, look to contact the webmasters to get links changed to the new locations. Although permanent (301) redirects pass much of the link equity to the new source, it is always a good idea to get as many updated as possible.
Further Reading: How to Completely Ruin (or Save) Your Website with Redirects – Moz
- When planning or going through a migration beware that new added pages could be lost or unaccounted for in the final list of URLs. For instance, if you have an active blog, new posts should be kept unpublished until the new site goes live.
- Keep original web files on server if possible. If for whatever reason a page is not included in the new site you may need to recover it!
- Make sure you have a useful 404 page in place just in case. This 404 page should include an apology, useful links to help the user get back on track (as well as the standard navigation menu template), an internal search bar and a request for the user to report the problem.
- Both Google and Bing index images and therefore if these are a big source of incoming traffic, these should be factored in to your redirect file.
- If a big migration is planned, you may want your blogging/communication/PR team to pre-emptively notify users of the imminent change – particularly if the navigation, branding or aesthetics are changing drastically and ESPECIALLY if the domain is changing.
- If you are going through a re-brand you may want to arouse curiosity and anticipation through a clever social media or email marketing drive. If done well this can attract lots of attention and even links!
- Don’t forget to sync your Google Analytics/Webmaster Tools accounts with the new site as soon as it goes live! Monitoring the success of your migration is highly dependant on data (drop in visitors, increase in missing pages etc.) for early warning signs – you don’t want to be missing data for those first few critical days.
- Make sure the meta data for new pages and optimised content is all pre-written ready for launch. You don’t want your rankings to drop because of an oversight in basic SEO elements.
- Your website may already be benefiting from caching (and if it isn’t, it really should be) – make sure any caching of resources is carried over to the new site. Any upgrade should be towards a faster website and forgetting to carry over speed enhancing elements can hinder the success of your site launch
- If your new site is bigger or more image-heavy you need to make sure your current hosting package can handle the increase in data. Otherwise, consider upgrading to a better platform or dedicated server.
- If your site is dramatically changing you may wish to consider using your first blog post to describe any new features and how to navigate through the new site
- Implement your migration one step at a time and do the necessary checks each time. This way you can isolate issues more easily. (e.g. start with the main website redirects, then consolidate any dupe URLs, then implement the blog, then any sub-domains etc)
- Research tools to help you with your migration such as scraping tools like Screaming Frog and XML sitemap generators if necessary
- While gathering your URLs to be redirected, this is the ideal time to find redundant, outdated or “thin” (offer no real value) pages. Shedding this excess weight or updating with useful content will make your new site far more streamlined and ensure every page has a purpose.
- If migrating to a new domain, spend some time carefully choosing a domain name and TLD extension as it is much harder to rectify once established. Domain names should focus around your brand name or if taken, a suitable substitute. Google is cracking down on keyword-stuffed URLs so don’t opt for something like www.used-cars-northampton.com for instance (unless of course this is your legitimate business name!). Additionally, carefully choose a domain extention (e.g. .com, .co.uk, .net or one of the newly released ones like .london, .shop or even .football!). You may also want to claim your brand social media profiles at this point too.
- No matter how clean your migration is, there will always be some loss of traffic initially as the search engines rediscover and re-index your site. This is unavoidable but the impact can be lessened with a fully-thought out migration strategy and swift implementation. Make sure that you superiors are aware of this during any talks of migration.
- Most people realise that redirecting web pages is critical. However, have you given a thought to your images and PDFs? These are indexed too, so if they make up a chunk of your incoming search traffic make sure these are accounted for in your redirect file.
- Aside from Analytics tracking you will want to ensure any additional tracking (such as on-click button tracking, UTM sourcing etc.) is carried over to the new site to avoid any loss of data. These are easily overlooked.
- If your content such as blog articles contain internal links to other pages, make sure these are updated to reflect the new URLs – don’t just rely on your redirects to do the hard work as this looks unprofessional.
- Testing is critical in the early days of a site migration. Ensure you have Webmaster Tools set up as well as using a scanning tool such as Xenu to check for broken links. Of course, most of the issues should have been caught through manual checks performed prior to the site going live!
- Both Bing and Google Webmaster Tools have a feature to “call” their bots to come and crawl your site rather than waiting for them to naturally crawl it. This should be done as soon as the site goes live so that the search engines are indexing your new content and site structure immediately.
- In the early days of discussing migrating to a new domain make sure you buy up (as well as relevant variations) as soon as a decision is made. Nothing is worse than kicking off branding and marketing developments and finding out the brand domain has been snapped up!
- Set up a useful (by “useful” I mean useful to potential customers that happen to stumble upon it) holding page so that it can start accruing age. Always buy a domain for the maximum time possible (usually 10 years) as long-term investments are a credibility signal for the search engines
- If your new site runs on a new CMS system or server this could have positive or negative implications on SEO. For instance if the server is strained/unreliable or the CMS is bulky/bloated this may affect the load speed of your new site which could affect rankings.
- Chances are, the new site will be developed on a staging site. It seems redundant to even say it but make sure this staging site is completely “noindexed” – I have known instances where this has been forgotten and the search engines began to index both live and staging versions of website pages!
- New site, new robots.txt file – make sure this is not blocking any critical areas of your new site and that relevant chunks
- If you are migrating your site because of a search engine penalty DO NOT 301 REDIRECT the site – you risk carrying the penalty over and you will have to start all over!
- If you are opting for a new domain, keep ownership of old domain for as long as you can afford. This stops competitors buying up the domain and building off the back of the authority you have built up. They could also remove/modify the redirect commands within the .htaccess file hosted on the original domain. Lastly, any existing customers will associate you with the old domain/brand until the update sinks in – don’t confuse them.
- Pointing Google Ads to URLs that automatically redirect is against Google’s guidelines. Make sure your ads are updated to reflect the changes as soon as they go live or risk having your Ads suspended. If you are moving to a completely new domain you may need to have your ads re-approved altogether which can take time. When deciding on a migration, make sure your PPC team is aware!
- Beware, when moving domains you will have to set up a brand new Analytics profile for the new site. Historic data cannot be transferred so make sure you hang on to this old profile for reference.
- Unlike link juice, social proof such as Likes, +1s, Tweets etc associated with URLs that get redirected will be lost. If social signals do indeed affect rankings (not confirmed) you may suffer a rankings dip as a result. Make sure re-promoting content to recoup some of the lost signals is part of your post-migration strategy.
- When gathering your URLs (particularly those of files) remember, some servers can be picky. URLs with capital letters are different from those that are all lower-case – make sure you redirect both versions if necessary. Likewise, underscores could be mistaken for spaces (in underlined links).
- Find Orphaned Pages From Your Sitemap.xml File with Excel and IIS Toolkit
- How To Reduce SEO Risk In Migrations, Domain Changes, etc.
- SEO for PDFs
- How to Optimize a PDF for Search – Hubspot
- Has Google made domain migration tougher? – Branded3