Example Sitecore Robots.txt

Just a note of example robots.txt file that I’m using in Sitecore:

User-agent: *
Disallow: /sitecore
Disallow: /Sitecore
Disallow: /sitecore_files/
Disallow: /sitecore modules/
Disallow: /App_Browsers/
Disallow: /App_config/
Disallow: /App_Data/
Disallow: /temp/
Disallow: /upload/
Disallow: /xsl/
Sitemap: http://www.example.com/sitemap.xml

Also, don’t forget to set up your Sitecore.Analytics.ExcludeRobots.config

Example Sitecore Robots.txt

Web Forms For Marketers blows up if Analytics disabled

This one was a real Sherlock Holmes case. The site I’ve been working on has Azure App Insights monitoring it, and on the live site we started to see lots of exceptions that hadn’t appeared in testing.

Exceptions

Digging into the exceptions, I noticed that:

  • The exception was System.Web.HttpUnhandledException with an inner System.NullReferenceException
  • The method throwing the exception was Sitecore.Form.Core.Ascx.Controls.SimpleForm.OnAddInitOnClient, which is part of the Web Forms for Marketers suite.
  • The user agent was always Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) – so it was Googlebot making the request. There’s no reason that should cause an exception though!

I realised after a while that Googlebot is a robot, and so is excluded from analytics, so this problem probably had the same cause as the exceptions our testing was starting to throw up. The OnAddInitOnClient method isn’t one of mine, though – it’s part of Sitecore’s Web forms for Marketers – so I opened up Reflector and down the rabbit hole I went… Continue reading “Web Forms For Marketers blows up if Analytics disabled”

Web Forms For Marketers blows up if Analytics disabled

Take care with Sitecore.Analytics.ExcludeRobots.config and fixed IP addresses

Right, so this is a lesson of “It’s obvious when you think about it”.

The new company that I work for have had a policy of adding the IP addresses of their gateways to the Sitecore.Analytics.ExcludeRobots.config file. The idea behind this is that we don’t want to pollute customer’s analytics data with traffic from ourselves, particularly if we are running health monitoring service that occasionally polls their website. That all seems sensible enough.

Here’s the problem – our tester was trying to test some of the pages we’d built on a live site. There was a problem with that – more in a later post – but the upshot was that it appeared that on two identical devices (iPhones) one would work, and the other would receive the yellow screen of death. A straw poll of iPhones in the office showed 3 working, and 3 failures. Curious… Continue reading “Take care with Sitecore.Analytics.ExcludeRobots.config and fixed IP addresses”

Take care with Sitecore.Analytics.ExcludeRobots.config and fixed IP addresses

Configure Sitecore to update Sitemaps.xml on ALL servers

In a robust web-based system one would expect to have multiple servers serving content. In Sitecore, this is typically that you have a content management server, and then one or more content delivery servers.

Sitecore also offers a ‘Sitemap XML module’, which is pretty neat. When you publish content it will generate an updated Sitemap.xml file – basically, a listing of pages that a web-crawler like Googlebot can use to crawl and index a site. It can also ping the major search engines, telling them that the sitemap has been updated, and that they should recrawl it at their leisure.

SiteMap

However, things get trickier where you have content delivery servers. Continue reading “Configure Sitecore to update Sitemaps.xml on ALL servers”

Configure Sitecore to update Sitemaps.xml on ALL servers

Clearing Sitecore Analytics data

So I’ve been moving a Sitecore website to a live server, and one of the problems I had was that the analytics database already had data in it. The MongoDB instance was new, so that was empty – but how do I clear our test/development data out of the Analytics database?

Well it turns out that there is a stored procedure – __DeleteAllReportingData – in the Analytics database for just that:

capture21

Run that, and all your data will be gone. Obviously, use with care – but useful.

It seems to work by removing all foreign keys, truncating each table, and adding the keys back, so the user running this will need permission to do all these things.

Also, you can exclude tables, but I just ran it with the input of an empty string ( ” ) to clear all.

Clearing Sitecore Analytics data

Using custom placeholders in Bucket Search views

Sitecore lets you design custom views for searching item buckets – that is, when you’ve got so many items in a part of the tree that search is the only reliable option for finding the items.

However, while Sitecore comes with a number of ‘placeholders’ for the text to use in those views, it’s not obvious how to add more.

Well, I found this mentioned in this excellent post: Add Search Views to Sitecore 7, and here’s my take on it… Continue reading “Using custom placeholders in Bucket Search views”

Using custom placeholders in Bucket Search views

Problems setting a default Bucket search view in Sitecore

I’ve been doing a bit of customisation to a Sitecore 7.5 instance that is going to have to store a lot of nodes in one particular area. Naturally, thought of using buckets, which would be ideal. I’ve a few things to cover in this, but I’m going to start at the end – and the hardest problem.

I wanted to set a default search to run when someone comes to the bucket in the content editor. Continue reading “Problems setting a default Bucket search view in Sitecore”

Problems setting a default Bucket search view in Sitecore