Multivariate Testing

Quote from: phil&steph at Jan 24, 2013, 07:18 AM

yeah, I've just set up an experiment and (hopefully) I'll be able to see the results if I've set the goals correctly

In your case with the button the goals could simply be the destination URL of the button to test which version converts best

Of general interest: How did you setup the two versions? With two resources or with some dynamic change with a snippet/chunk combo?

Yes, you could use duplicate pages as the result page, then use one of the page hit counter packages to keep track of which one gets more hits.

Originally I wanted to use the snippet/combo method but it turned out that marketing had unique links for each button anyway so I duplicated the resource for the experiment. I've also added analytics to each landing page.

I'd still like to find a nice tool for testing though, a few folk have mentioned http://visualwebsiteoptimizer.com/ but don't know enough to know if it's worth paying for...

Quote from: phil&steph at Jan 24, 2013, 07:50 AM

Originally I wanted to use the snippet/combo method

It is interesting. Considering to make a split testing snippet that changes a specific chunk based on a parameter in the URL. Could be done with a test on something like if someURL.com/page#verA then chunkA or if someURL.com/page#verB then chunkB.
Another method could be to set the template based on a parameter in the URL, but that would be overkill if all you wanted to test was a button...

I know that was the issue I was having, I was trying to avoid doing a full on A/B test when the only element I was testing was the button. But in the end I had to concede.

New to coding, new to marketing and new to Modx - it's been one heck of a learning curve

Easy A/B testing is a requirement of a project I'm starting. I'm not sure how I'm going to do it yet, but if I get something I'm happy with I'll try to release it as an extra.

There was some discussion of an extra here: https://forums.modx.com/thread/72691/does-anybody-need-an-a-b-test-add-on?page=2, but it appears that nothing came of it.

I would think that you'd want to check for a $_SESSION variable for each user. If it's not set, you'd randomly select what to show (or alternate), set a placeholder with selected content, and then set the result in the $_SESSION variable so that that user would see the same selection throughout that visit.

Yeah, Bob, that makes sense. Some other considerations:

• Should it swap out different chunks? Resources? Templates?
• How to track goal conversion and analyze the results - probably needs a CMP and a custom table/tables to record test settings and results.
• How to record a goal conversion - is it just when someone visits a particular URL, or do you use JS to allow tracking of any browser event, or...?
• How much detail to track about conversion events - for example, if a user is logged in, do you want the conversion event connected to their user profile?
• How to integrate with testing tools such as Google Experiments
• How to take advantage of caching as much as possible to maintain good performance - getCache might be useful for that

I think the answer for most of those depends on what you're trying to track, what the "goal" is, and how critical the information is. In some cases, you're goal might be simply clicking on something. In other cases, it might be a product purchase or a completed registration.

In a lot of simple A/B tests, you just want to know the click totals (or percentages) for each option.

You'd have to tailor the method to the specific task.

Here are some other considerations: http://searchengineland.com/googles-seo-guide-on-ab-multivariate-testing-130093

Ooo, that link is a good one - yeah, that's very important, you don't want to mess up your SEO.

As for the requirements, my thinking was to try to make it as generally applicable as possible, so maybe it should have options for testing various thing such as different chunks, different templates etc, and for tracking things such as clicks and page views.