Monolingual vs. Multilingual pages / root name mode and SEO

56 Posts

kongo09 Reply #1, 14 years, 3 months ago

Setup

YAMS runs in root name mode.
Two languages are defined, say "en" and "de".
"en" is default and root name for "en" is empty
In monolingual settings, server name is left empty, defaulting to the site URL for monolingual pages (which with an empty root name for "en" is the same as the site URL for "en" pages.
URL redirection mode is set to "current else browser"
Browser language is set to "sv"
The start page at www.mysite.com is multilingual in "en" and "de".
The linked sub-page www.mysite.com/next/ is monolingual (so both pages have their own template)

Observation

Now, when I surf to www.mysite.com, with all cookies cleared, the English "en" version of the page is displayed, which is correct, since there is no Swedish "sv" version and English is the default.

When I go there with browser set to "de", still the English version is displayed and the cookie is set to "en". That’s wrong. I would expect it to switch to German instead. Interesting enough, when the configuration is set to use a root name of "en" for English as well, then www.mysite.com correctly redirects to www.mysite.com/de/ and sets the corresponding cookie yams_lang to "de".

Going back to the original setting without root name for "en", I click on a link to the sub-page www.mysite/next/ (which is monolingual). The page correctly displays in its language (in that case also English, but that doesn’t matter). The cookie remains set to "de". On the page is a link back to the start page, created via (yams_site). It goes back to www.mysite.com and not to www.mysite.com/de/ although I’m in "current else browser" mode. That’s also wrong. Clicking it, the cookie doesn’t seem to be evaluated and I stay on www.mysite.com without a redirect to www.mysite.com/de/ and the cookie is changed to "en". Not good!

The same test with "en" set as root name for English still doesn’t rewrite the link of (yams_site) to the German www.mysite.com/de/ as expected, but at least it redirects from www.mysite.com to www.mysite.com/de/ which gives the right result.

Expectation

If the system doesn’t have a current language set (via cookie or via URL) then the browser language should be used - or if not available the default language
If the URL defines a language, that one should be used, unless the cookie indicates that the user before has already chosen another current language. Browser language is not relevant then.
In effect, the rule would be "cookie, else URL, else browser" - which I understand should be covered by "current, else browser"

Conclusion

I feel the automatic URL rewriting doesn’t work correctly. The "current, else browser" rule seems to be more a "URL, else current, else browser" rule. Without proper rewriting of the URL on monolingual pages, this will fails.

I also believe that the cookie is not correctly evaluated when moving from a monolingual page to a multilingual page. This seems to be associated with an empty root name for the default language - coinciding with the server name for monolingual pages.

Any thoughts, suggestions, solutions, analysis?
Happy new year!

56 Posts

kongo09 Reply #2, 14 years, 3 months ago

Digging a bit deeper into the [tt]DetermineCurrentLangId[/tt] function in the [tt]yams.class.inc.php[/tt] file, it indeed seems that before the cookie is actually looked at (and regardless of the [tt]itsRedirectionMode[/tt]) the URL is evaluated. So if one language coincides with the monolingual URL, I’m not sure how this is supposed to work...

805 Posts

PaulSuckling Reply #3, 14 years, 3 months ago

Quote from: kongo09 at Dec 31, 2009, 02:20 PM

Conclusion

I feel the automatic URL rewriting doesn’t work correctly. The "current, else browser" rule seems to be more a "URL, else current, else browser" rule. Without proper rewriting of the URL on monolingual pages, this will fails.

I also believe that the cookie is not correctly evaluated when moving from a monolingual page to a multilingual page. This seems to be associated with an empty root name for the default language - coinciding with the server name for monolingual pages.

Any thoughts, suggestions, solutions, analysis?
Happy new year!

Happy New Year to you too kongo09!

Thanks for your report. I’ve read through it carefully. I don’t agree with you that YAMS is functioning incorrectly. I think that there is just a misconception about how YAMS works. YAMS is designed to work in a search engine friendly way whereby each URL is associated with the same unique and specific content on each request. I think the misunderstanding could be related to poor or incorrect documentation and to the fact that this is a tricky topic. If you think the documentation is confusing or misleading anywhere then please let me know which text so that I can improve it.

Here’s my analysis. Lets take your scenario point by point:
Quote from: kongo09 at Dec 31, 2009, 02:20 PM

When I go there [edit:to www.mysite.com] with browser set to "de", still the English version is displayed and the cookie is set to "en". That’s wrong. I would expect it to switch to German instead.

Actually, YAMS only does redirection when it’s not possible to uniquely determine the required document and language from the URL. This could occur, for example, if a standard MODx URL is supplied that doesn’t happen to coincide with a multilingual URL, or if root names are used for all languages but the site root http://www.mysite.com/ is visited. In your example above the root name for English is blank, so http://www.mysite.com/ is a valid URL for the English site start document. Therefore there is no need to do redirection. Displaying English is correct. This is a good thing. It means search engines know exactly what to index. If it were the case that sometimes when you visit an URL it provides English content and other times it redirects to a different page to display content in a different language then search engines would get confused about which URL and content to index. You really don’t want this happening for your site start document.

Quote from: kongo09 at Dec 31, 2009, 02:20 PM

Interesting enough, when the configuration is set to use a root name of "en" for English as well, then www.mysite.com correctly redirects to www.mysite.com/de/ and sets the corresponding cookie yams_lang to "de".

In this case (root names for all languages) http://www.mysite.com/ is not a valid (multilingual) URL. YAMS identifies that this is a valid standard MODx URL for the site start document. It doesn’t know which language was intended, so it uses the user specified logic to determine it, which in your case is ’current else browser’. Since no cookie has been set, the browser is used to determine the language - which in this case selects German. So, it redirects to the German variant of the site start document, which is http://www.mysite.com/de/

Quote from: kongo09 at Dec 31, 2009, 02:20 PM

Going back to the original setting without root name for "en", I click on a link to the sub-page www.mysite/next/ (which is monolingual). The page correctly displays in its language (in that case also English, but that doesn’t matter). The cookie remains set to "de". On the page is a link back to the start page, created via (yams_site). It goes back to www.mysite.com and not to www.mysite.com/de/ although I’m in "current else browser" mode. That’s also wrong.

No. It’s not. According to the definition of [tt](yams_site)[/tt], when on a monolingual document and server mode is off the output is the same as [tt][(site_url)][/tt] - which is http://www.mysite.com/ - which is what you got. In general, if you want to create a link to the site start document, then you should use [tt](yams_doc:[(site_start)])[/tt].

In this case however, even if you did use [tt](yams_doc:[(site_start)])[/tt], then a link to the English version of the site start page would be created - which again would be http://www.mysite.com/. This is because monolingual documents are always parsed as if they are in the default language (English in your case).

I think you were expecting that the link from the monolingual document to a multilingual document would take you to the language of the last visited multilingual document (which in this case would be German). However this would lead to the situation whereby the document would contain different links dependent on the last visited page - which again would be very search engine unfriendly.

What all this is pointing to is the fact that that you should be using a multilingual document in this situation - not a monolingual document. If you use a multilingual document then the English version would contain links to English resources, German to German etc. (even if the actual text used for each language was the same and written in either English or German).

Monolingual documents are normally used to list content in multiple languages (using the [tt][[YAMS? &get=`repeat` ... ]][/tt] snippet call) rather than displaying content in a single (the default) language. For example; a landing page that allows you to select which language you want, or a multilingual site map that lists all documents in all languages. In that sense, monolingual is probably not a good name for such documents!

Quote from: kongo09 at Dec 31, 2009, 02:20 PM

Clicking it, the cookie doesn’t seem to be evaluated and I stay on www.mysite.com without a redirect to www.mysite.com/de/ and the cookie is changed to "en". Not good!

See above. This is a good thing. If you browse to a valid multilingual URL you stay on it and this is used to set the current language.

Quote from: kongo09 at Dec 31, 2009, 02:20 PM

The same test with "en" set as root name for English still doesn’t rewrite the link of (yams_site) to the German www.mysite.com/de/ as expected, but at least it redirects from www.mysite.com to www.mysite.com/de/ which gives the right result.

As described above, [tt](yams_site)[/tt] evaluates to http://www.mysite.com/ in this case - which is not a valid (multilingual) URL, so with your setup ’current else browser’ redirection takes place - which takes you to the German page: http://www.mysite.com/de/

Quote from: kongo09 at Dec 31, 2009, 02:20 PM

Expectation

If the system doesn’t have a current language set (via cookie or via URL) then the browser language should be used - or if not available the default language

I claim that YAMS does this when in ’current else browser’ mode - but only when it is not clear from the URL which language was intended.

Quote from: kongo09 at Dec 31, 2009, 02:20 PM

If the URL defines a language, that one should be used, unless the cookie indicates that the user before has already chosen another current language. Browser language is not relevant then.

In effect, the rule would be "cookie, else URL, else browser" - which I understand should be covered by "current, else browser"

I disagree. YAMS currently does "URL, else user specified redirection mode", or, in other words: "If the URL defines a language then the document should be displayed in that language. Otherwise use the specified redirection mode to select a language variant of the document. This could be: default, current (else default), current else browser, browser etc."

If your scheme was used, then a user could browse to an URL for an English document and then decide that they actually wanted the French version - but when they try to browse to the URL for the French document they would keep being redirected back to the English version - which would be very annoying. As mentioned earlier, it would also be search engine unfriendly.

I hope the explanation above will help you achieve a better understanding of how YAMS works. Let me know if there is a specific set-up or behaviour that you are trying to achieve. I might be able to help you achieve it.

YAMS: Yet Another Multilingual Solution for MODx
YAMS Forums | Latest: YAMS 1.1.9 | YAMS Documentation
Please consider donating if you appreciate the time and effort spent developing and supporting YAMS.

56 Posts

kongo09 Reply #4, 14 years, 3 months ago

Happy new year!

wow, thanks for the perfect explanation of what is going on. Indeed, a complicated topic, but I’m sure people reading this thread here on the forum have now a way to better follow the mechanics (and logics behind the mechanics) of YAMS. I understand the main rationale driving the logic is search engine optimization.

The bottom line of what I wanted to achieve is a consistent user experience. Once I’m in a certain language environment, I’d like to stay in that environment. No matter, if that involves a click-path changing between monolingual and multilingual pages. However, I get the point that coming back from monolingual takes out back to default language for SEO reasons.

Therefore, I have two suggestions:

The current monolingual mode is rather a "default language" mode. This would make it explicit what’s going on. In Manager, the ManagerManager plugin should be used to modify the input tabs by presenting an interface similar to the multilingual pages, but just show the tab for the default language. This way, it is very clear, that the content entered is not just monolingual but also a specific language, i.e. the default language.
A true monolingual mode could be achieved by presenting the monolingual content in a consistent multilingual path. That means, the Manager fields exist only for one language but not necessarily the default language. The language tabs would be shown but the normal input fields are used as it is right now for monolingual pages. However, in the background, this same content would be used on (to stay with the example) www.mysite.com/en/ as well as www.mysite.com/de/ and as a consequence, linking back would also be consistent. The remaining question would be if search engines penalize you if you have the same content under two different URLs on the same site. If yes, then we’re a bit stuck as it means that there cannot be a "true" monolingual page on a SEO site and everything should be multilingual.

Thanks for the great discussion!

805 Posts

PaulSuckling Reply #5, 14 years, 3 months ago

Hi again kongo09.

What you appear to be requesting is a mechanism that would allow users to follow a click path that would take them from a document in their language, to a document in the default language and back to a document in their own language. I would not regard this switching of language as providing a ’consistent user experience’. I think that to provide a consistent user experience all content must be translated into all languages provided. This means that all documents need to be multilingual (one URL variant for each language), with the possible exception of some monolingual documents (accessible via only one URL) which contain content that is meaningful in all languages.

So, I’d be interested to know why you want or need to use a ’monolingual’ document and what kind of content you intend using it for. Perhaps you are using a monolingual document because the content for that document has not yet been translated from the default language into the other languages? If that’s the case I think that a multilingual document is actually required in this case. For example, if the default language was English, then in the content fields of the other languages you could use

<div lang="en" xml:lang="en">[*content_en*]</div>

to incorporate the English content. The output for the other languages would be the same as for the English version of the document - except that the content would be parsed by YAMS in the correct language. Therefore any links within the content would point to the correct language versions of those documents.

With regards to SEO, Google’s quality guidelines warn against duplicate content. The scheme that you suggested in your second point whereby the same page would be accessible via different URLs could cause Google to penalise the site - unless, as suggested by the guidelines, the rel="canonical" link element was used to inform Google about the duplicate content.

With regards to monolingual documents, I agree that they are an odd entity:

They are actually just standard MODx documents in the sense that: [list]
[li]they are accessible via a single URL,
content is specified via the standard document variables and
the document could contain content in any language.

[/li]
[li]They are not standard MODx documents in the sense that they may contain YAMS placeholders and snippet calls etc. which will be evaluated as if the document is written in the default language. That means that links to multilingual documents will point to the default language variants of those documents.[/li]
[li]They enable users who have a non-YAMS (monolingual) site written in a single language to convert it to be multilingual by:

altering their (monolingual) document templates to include YAMS markup,
changing their templates from monolingual to multilingual,
translating the content into other languages and finally
including a language select control on the site in order to publicise the existence of other languages variants.

[/li]
[li]They can be used for documents accessible via a single URL that presents content in multiple languages (like a sitemap listing all language variants of all documents, or a language select landing page) or that is meaningful in all languages (like a graphic or image).[/li]
[/list]

Your idea of moving the standard document variables on monolingual documents to a default language tab is growing on me. I agree that it would probably make it clearer that YAMS will parse the content as if it were in the default language... although it might make it less clear that the content is being stored in the standard document variables rather than template variables.

For multilingual documents the standard document variable pagetitle is normally displayed on the first (non-language) tab. This brings up the question of what to do with the pagetitle for monolingual documents. Should it be left there or should it be moved to the default language tab? I think that moving it to the default language tab would be best, but I’ll listen to arguments for not moving it.

YAMS: Yet Another Multilingual Solution for MODx
YAMS Forums | Latest: YAMS 1.1.9 | YAMS Documentation
Please consider donating if you appreciate the time and effort spent developing and supporting YAMS.

56 Posts

kongo09 Reply #6, 14 years, 3 months ago

Having read a bit further on SEO topics, I’m still not 100% clear about the best approach to handle multiple languages and how to direct/redirect the user.

One mantra everyone tells me: Don’t think about the search engine, think about the user. And another thing is: Search engine robots don’t know language preferences.

My feeling is, a good SEO approach and user friendly way of managing the multilingual site would be:

Have the default language start at root, i.e. at www.mysite.com
Put other languages into subdirectories, i.e. at www.mysite.com/de/
For a visitor to the root (or any other document not in a language tree) check if a YAMS cookie indicates that the user has already chosen a language. Search engines don’t carry cookies so there won’t be a choice.
If there is no choice, serve the default language (i.e. give that to the search engine as well)
If there is a previous choice, redirect 302 to the chosen language (current, in YAMS terms)
Provide language links on every page to the corresponding other pages (also helps the search engine to find them)
Provide a sitemap.xml with all language variants

In effect, this would be user friendly (automatic language choice which can be overridden anytime by the user) but also SEO friendly (one page one URL, no redirect for the main entry door, all language pages findable).

To top it off (but that’s not YAMS at work) you could have country domains that 301 redirect to the right language tree (www.mysite.de -> www.mysite.com/de/)

With the current YAMS implementation, I feel this doesn’t work. Maybe we can discuss a bit if a change makes sense, or if I still lack some (SEO/MODx) understanding.

56 Posts

kongo09 Reply #7, 14 years, 3 months ago

Going back to your quote:

This is a good thing. It means search engines know exactly what to index. If it were the case that sometimes when you visit an URL it provides English content and other times it redirects to a different page to display content in a different language then search engines would get confused about which URL and content to index. You really don’t want this happening for your site start document.

I did some more tests with webmaster tools from Google. With root name mode ON, there is always a temporary redirect from the home page to the default language page. That’s not good for the search engine. It is rather desirable to have this temporary redirect work only on clients which send a language preference or carry a yams_lang cookie, i.e. on normal browsers and not on Googlebot.

805 Posts

PaulSuckling Reply #8, 14 years, 3 months ago

Quote from: kongo09 at Jan 04, 2010, 04:32 PM

One mantra everyone tells me: Don’t think about the search engine, think about the user. And another thing is: Search engine robots don’t know language preferences.

Quote from: kongo09 at Jan 04, 2010, 04:32 PM

My feeling is, a good SEO approach and user friendly way of managing the multilingual site would be:
[list]

Have the default language start at root, i.e. at www.mysite.com

Put other languages into subdirectories, i.e. at www.mysite.com/de/

Up to here I agree and this is possible with YAMS.

Quote from: kongo09 at Jan 04, 2010, 04:32 PM

For a visitor to the root (or any other document not in a language tree) check if a YAMS cookie indicates that the user has already chosen a language. Search engines don’t carry cookies so there won’t be a choice.

If there is no choice, serve the default language (i.e. give that to the search engine as well)

If there is a previous choice, redirect 302 to the chosen language (current, in YAMS terms)

Provide language links on every page to the corresponding other pages (also helps the search engine to find them)

Provide a sitemap.xml with all language variants

In effect, this would be user friendly (automatic language choice which can be overridden anytime by the user) but also SEO friendly (one page one URL, no redirect for the main entry door, all language pages findable).

Here, personally, I disagree about the automatic redirection being a good idea or user friendly... So, YAMS doesn’t do this by default. However, if you wanted to do it you could quite simply write your own cookie and redirection script to do it for you. Here are some things to think about:

Would you only do this redirection from the default language page to a different language page on the home page, or would you do it on any default language page?
If the user lands on a non-default language page, would you redirect them back to the default language page if a cookie had been set?
Wouldn’t it be annoying if you sent someone a link to some content, but when they look at it they see something different because they either get or don’t get redirected?
Search engines could misinterpret such behaviour as an attempt at cloaking.
Serving different content to search engines than to users.

In general, I think that automatically redirecting people based on what you think they might want is not a good idea. YAMS gives you other options. You could have a home page that was monolingual and which provided the user with a list of languages from which to choose which language they want. Once they have chosen a language - they stay in that language and any page they visit will always be there and would be bookmarkable (and search indexable by search engines). This solution involves no automatic redirection and so is search engine friendly too. If you really wanted however, you could use cookies to implement a a ’remember this preference’ option on the home page, which if set, would do automatic redirection to the language of their choice on subsequent visits. YAMS doesn’t prevent you from implementing this if you wanted to.

Quote from: kongo09 at Jan 04, 2010, 04:32 PM

To top it off (but that’s not YAMS at work) you could have country domains that 301 redirect to the right language tree (www.mysite.de -> www.mysite.com/de/)

If you’ve got the luxury of being able to use country specific domains, then why bother with root names and redirection? Just use the country specific domains. YAMS allows you to do this and Google even recommends it. Here is a quote from Google’s webmaster guidelines:

Use top-level domains: To help us serve the most appropriate version of a document, use top-level domains whenever possible to handle country-specific content. We’re more likely to know that http://www.example.de contains Germany-focused content, for instance, than http://www.example.com/de or http://de.example.com.

[/list]

YAMS: Yet Another Multilingual Solution for MODx
YAMS Forums | Latest: YAMS 1.1.9 | YAMS Documentation
Please consider donating if you appreciate the time and effort spent developing and supporting YAMS.

56 Posts

kongo09 Reply #9, 14 years, 3 months ago

Thanks for the swift reply (as usual!)

Country level domains are good for countries - not necessarily languages. What language would you use with Switzerland or Belgium?

Automatic redirection would only happen if YAMS is set that way: e.g. unique URL else current else browser else default language. It would happen on any default language page, i.e. where the root name is empty.

I’ll try to put a new mode into YAMS to achieve this wink

and will post my results here.

805 Posts

PaulSuckling Reply #10, 14 years, 3 months ago

Quote from: kongo09 at Jan 05, 2010, 08:46 AM

Going back to your quote:

This is a good thing. It means search engines know exactly what to index. If it were the case that sometimes when you visit an URL it provides English content and other times it redirects to a different page to display content in a different language then search engines would get confused about which URL and content to index. You really don’t want this happening for your site start document.
I did some more tests with webmaster tools from Google. With root name mode ON, there is always a temporary redirect from the home page to the default language page. That’s not good for the search engine. It is rather desirable to have this temporary redirect work only on clients which send a language preference or carry a yams_lang cookie, i.e. on normal browsers and not on Googlebot.

If you choose to use root name mode and to specify root names for all languages, so that there is no content at the site root - then yes, YAMS will automatically do redirection to one of the defined languages. (This isn’t the best way to structure a site so YAMS lets you do it differently - as discussed above). As well as the language redirection mode being customisable (also as discussed previously) the redirection status codes for redirection to the default and non-default languages are customisable. See Modules>YAMS>Other Params>URL Redirection Settings. You could use "See Other (303)" or whatever you feel is most appropriate.

YAMS doesn’t treat users and the Googlebot differently. That would be against the rules and could be considered cloaking as mentioned previously.

YAMS: Yet Another Multilingual Solution for MODx
YAMS Forums | Latest: YAMS 1.1.9 | YAMS Documentation
Please consider donating if you appreciate the time and effort spent developing and supporting YAMS.