E-mail Testing For Web Applications

E-mails help your business!

User Experience

 Does my e-mail look beautiful everywhere?

 How do emails appear on any email client software(s); such as Microsoft Outlook, Mozilla Thunderbird?

 Am I having sufficient information about the unsubscribing and providing information as, “Do not reply to this e - mail” for “no-reply” e-mail accounts?

 Are we good with email etiquette with respect to developing email templates?

 Is the email system sending junk or bulk emails to the users, which could be annoying to them?

 What type of email template must be used

  HTML

  Plain

 Is it important to run a spell and grammar check and also check for a content formatting?

 Should there be an email signature that must be used for automated emails?

 Does the email system provide information via application after submitting Forgot Password or Sign up or any other form? Example: “Please make sure that you check bulk / junk / spam folder if you get the email in yourinbox within 30 minutes or “X” minutes.”

 Are there hyperlinks in the e-mail that navigate the customers to the website? This could possibly be an encourage browsing agent.

 Is there an optional text area or a drop-down box (HTML element) for customers who unsubscribe can enter the reason for unsubscribing?

 How does the HTML email elements render in the email viewer?

 What font style is used in emails and do the operating systems have them and support them?

 Do we use a font style which exists in widely used operating systems or the ones which our application supports?

 Are there any creative concepts for email templates design?

 Example: If your web application is developed for target audience which includes old age people of 60’s or 70’s or before that. They might enjoy the design being a notebook paper with lines or no lines however, giving look of a letter. Most or some loved the way they used to write letters on paper and send them or read them. You might give them that feeling.

 Do emails render and look as expected on all electronic gadgets that support email? Are they compatible on all the platforms and versions of platforms?

Security

 If you are using multiple server and you are using some other server name in the from e-mail address then it would be delivery failure as it is kind of impersonation of some other sender. This helps in finding the threats during design phase of your e-mail system or SMTP architecture for your application.

Functional

 Can email addresses be case-sensitive?

 What is the maximum length e-mail text field that has to be set on the client-side, server-side and database?

 What happens when an SMTP server is down?

 What could be the consequences or threats of using HTML templates when not developed properly?

 Is there a size limit for cached content or delivery failure notifications or any kind of data onthe SMTP server?

 What are the consequences when the content reaches the limit? (E-mail size)

 Does the email system send emails with attachments (Auto-generated)? If yes:

  Does the email system send emails with attachments (Auto-generated)? If yes:

  What is the permissible file size of the attachment?

  Is it a zip file or raw file format?

  If it is a zip file, does the user need to download an external application to unzip it? Or the default uncompressing application of the operating system can be used?

Errors & exceptions

 What happens when an email is bounced for any unknown reasons?

 What are the exceptions that can be handled when an SMTP server is down?

 Is there a plan to maintain the queue of bounced e-mails and re-sending them?

 When the SMTP server is down and there are features that invoke the email system; forexample, Forgot Password Tokenized URL, a message must be provided, “We are facing some technical challenges, we will be back soon and you can use this feature”.

Reliability / Backup

  Can I have multiple SMTP servers as back up or alternative, where emails can be routed if bulk emails are sent via the server?

Stakeholders / Product Owner

 Is that email important or can I afford to ignore it?

 Should the emails be sent to all or only selected people who have opted for it?

 Should I consider Single and double opt-in?

 Do I use DKIM (Domain Key Identified Mail) to avoid phishing attempts, or is there any kind of counter-measures I have?

Domain Knowledge

 Do I know how an email system works?

 Do I know SMTP, POP3, IMAP and PST? Is it important for me to understand this to do a good and an effective testing?

 What are the different characters that email service providers allow and do I have consistency in our email address field sanitization? (Black-list and white-list for characters).

 Do I understand “SPAM” and “SPAM FILTERING”?

 Do I measure CTR (Click Through Rate) and Conversion Rate of e-mail campaigns?

 If there is no better CTR, how can I improve it?

 Do I understand “Opt-in Marketing”?

 Am I aware of Phishing Attempts via email?

 Am I aware of “Backscatter spam”?

Test Coverage

 Have I tested on widely used free email service providers and also on different domain names; such as top level and country level (Sampling)?

 The HTML formatting must be compliant with e-mail service provider anti-spam rules.

 Have we tested enough if e-mails land in inbox, bulk, junk or spam folder with widely used free e-mail service providers; such as Gmail, Yahoo, Hotmail, Rediffmail

 Have I tested on webmail clients; such as GoDaddy, Arvixe, Siteground and other widely used hosting providers?

Business Testing

 Is there a plan for getting back the customers who have abandoned the website? For example, sending emails or alerts about discounts, or offers to encourage revisiting the website.

 What is best time send or trigger specific emails? Example: Let us say, you do a business which is for people who are techies and work from 0900 hours to 1800 hours. It is possible that they get lots of e-mails. Do you want your e-mail to be positioned in the first when they open their inbox? Then, deciding on when to send e-mail campaign or specific e-mails is one of the ways.

Claims Testing

 Have I read emailing policy or privacy policy of the customer?

  If there is not, check with the stakeholders or product owner

  Different countries can have different policies or compliances of e-mail spam. Do you know them?

Reasons to abandon e-mail

 Not engaging

 Not pleasing

 Not appealing content

 Usage of colors

 The way the email is started

 The way email is addressed to the recipient

 The frequency in which emails are sent

 Same information can be got from different sources

 Preference to seek information instead of applications pushing the information

 Lack of poor targeting o Different countries can have different policies or compliances of e-mail spam. Do you know them?

References / Books

 http://www.uxmatters.com/

 http://www.nngroup.com/

 http://www.webstandards.org/2007/11/28/the-email-standards-project/

Check Automation

 Rendering aspects across cross browser, multi clients and public e-mail service providers

 Code analysis

 Link and image validation

 Spell & grammar check

 Character conversion