However, I wonder how many people actually complete the responses? How much of the data they provide is useful? Do the companies actually take not of the responses? Or is it all for show?
I suspect that many users will simply opt-out of these surveys as they can't be bothered with them. The process of asking the user if they wish to take part in a survey as they open the website means that the user has not had a chance to experience the site before they are asked if they want to comment on it. I suspect that many users will simply opt-out because they don't think they will have anything useful to say.
But what happens if the user has opted out, and after using the site realises they have issues that they wish to report? Well, there are few (if any) mechanisms to let the user change their mind once they have opted out of the original survey request.
It would be nice if these sites had a permanent link for giving feedback of your experiences when visiting the site, with the occasional reminder to visitors tthat they can give their opinions by clicking on this or that link.
Another issue with this method of gathering data is that the demographic that provides feedback will not necessarily be a proportionate representation of the site's readership. The snapshot of information that is provided is unlikely to be representative of the site's wider audience. If the developer's of the site were to take the feedback into consideration and, perhaps, do a complete redesign based on the results they could end up alienating their main readership. All responses from these types of survey must be taken within the context they are provided, and with consideration of the main user base.
Finally, if the user does decide to take the survey, they will probably find that the main focus is not on the website itself. Often, the surveys are more focused on the companies products/services that the website's usability. For example, I recently completed one of these surveys for a mobile phone selling site. The pop-up said "Let us know what you think of our website" but actually, the survey was much more focused on the company as a whole. Only 1 of 10 questions was actually about the website!
As someone that has good knowledge of HCI, I have developed a strategy for dealing with these surveys. If a site asks me for feedback and I do reasonably have time to give it, then I will do so. But I give a maximum of 10 minutes to each survey and I focus mainly on the HCI aspects of the site. Any longer thatn 10 minutes and I feel that I am doing consultancy and to be frank, as an almost-expert in the field I deserve payment for this! But, as someone who believes that improving HCI really does matter, a mere 10 minutes of my time can be given for free.
I do a lot of work with local government websites, and there's a huge range of commercial products to collect user responses (Socitm Insight springs to mind — if you've ever seen a survey pop up on a council website, it'll be this http://www.socitm.net/insight).
ReplyDeleteI've found that there's a massive range in the type and quality of feedback, depending on when and how the survey form is presented to users.
Exit-surveys on the web are nigh-on impossible, and entry-surveys, as you say, don't allow the user enough context of using the site before forming an opinion. On one site we've found that the best feedback results from embedding the feedback form directly into the content itself. So instead of a generic overlay that appears over any page, the survey form is shown in the context of the content to which it is relevant.
After running some split tests between form types, we found that a single "Was this useful" (y/n) and a "Comments" free text field, yielded the poorest results. Users took this to be a contact form for requests, complaints, follow-ups, callback requests... almost everything *but* feedback about the website.
Changing the form to providing a rating of the usefulness (1-5 rather than y/n) means that the rating can now be attributed to that piece of content — a report can be generated of content that performs consistently high or low, and you can see how a single piece of content performs over time (did the last edit improve user satisfaction?).
We didn't remove the open free text comments field, but prepended a category selection to it (Content, Suggestion, Compliment, Technical Issue, Other). This little bit of context seems to make users write more specifically. Plus it means the feedback becomes more actionable — you can assign Technical Issues to the IT/web department, Content comments to the web editor responsible for that page etc.
Anyway, yes, a little bit of thought into asking the right questions means you get vastly more useful answers :-)