Search Engines Unite On Unified Sitemaps System

In alphabetical order, Google, Microsoft and Yahoo have agreed to all support a unified system of submitting web pages through feeds to their crawlers. Called Sitemaps, taking its name from the precursor system that Google launched last year, all three search engines will now support the method.

More about Sitemaps is to be provided through the new Sitemaps.org site. As part of the announcement, the existing sitemaps protocol from Google gets a version upgrade to Sitemaps 0.9. However, no actual changes to the system have taken place. The new version number was simply done to reflect the protocol moving from an exclusive Google system to one that all three search engines now support.

Anyone already using Google Sitemaps needn't do anything different. The only change is now those sitemaps will be read by Microsoft and Yahoo, as well. More information will either be posted at the Sitemaps.org site or see these sections from each of the search engines, which I expect to be updated soon:

• Google Webmaster Central or sitemaps info.
• Microsoft Live Search submit page or help info (choose the Live Search Site Owner option from the drop-down box. Sorry I can't link to the exact relevant pages. Microsoft has this seemingly impossible to bookmark help system)
• Yahoo Site Explorer or submit information.

Other search engines are also invited to use the system -- it has specifically been placed as open property through Creative Commons so that others can make use of it. FYI, Ask isn't part of this announcement because it wasn't invited by the other three to take part, which I find unfortunate. Then again, among all four, Ask is the only one that doesn't already accept submissions in some way.

How can others contribute to its development? That remains to be worked out. So far, there's a working committee involving the three major search engines named. They say they are open to participation from other search engines, as well as content owners, to see the system grow and develop. I expect we'll find more structure to this emerging soon. At the moment, the key work has been in getting all three to agree to support the existing standard.

How about unification around other search standards, such as improving the robots.txt system of blocking pages. Again, this is something the search engines (specifically Google and Yahoo when I spoke to them), say they're interested in. So fingers crossed, we'll see more of this down the line.

Overall, I'm thrilled. It took nearly a decade for the search engines to go from unifying around standards for blocking spidering and making page description to agreeing on the nofollow attribute for links in January 2005. A wait of nearly two years for the next unified move is a long time, but far less than 10 and progress that's very welcomed. I applaud the three search engines for all coming together and look forward to more to come.

(Postscript: Announcements are up now from Yahoo, Microsoft and Google)

Below is more from the press release. Sorry I can't do a longer post about the system, but I'm also busy attending the PubCon conference, where the announcement has happened.

Las Vegas, November 16, 2006 - In the first joint and open initiative to improve the Web crawl process for search engines, Google, Yahoo! and Microsoft today announced support for Sitemaps 0.90 (www.sitemaps.org), a free and easy way for webmasters to notify search engines about their websites and be indexed more comprehensively and efficiently, resulting in better representation in search indices. For users, Sitemaps enables higher quality, fresher search results. An initiative initially driven by Yahoo! and Google, Sitemaps builds upon the pioneering Sitemaps 0.84, released by Google in June of 2005, which is now being adopted by Yahoo! and Microsoft to offer a single protocol to enhance Web crawling efforts.

Together, the sponsoring companies will continue to collaborate on the Sitemaps protocol and publish enhancements on a jointly maintained website www.sitemaps.org, which provides all of the details about the Sitemaps protocol.

How Sitemaps Work

A Sitemap is an XML file that can be made available on a website and acts as a marker for search engines to crawl certain pages. It is an easy way for webmasters to make their sites more search engine friendly. It does this by conveniently allowing webmasters to list all of their URLs along with optional metadata, such as the last time the page changed, to improve how search engines crawl and index their websites.

Sitemaps enhance the current model of Web crawling by allowing webmasters to list all their Web pages to improve comprehensiveness, notify search engines of changes or new pages to help freshness, and identify unchanged pages to prevent unnecessary crawling and save bandwidth. Webmasters can now universally submit their content in a uniform manner. Any webmaster can submit their Sitemap to any search engine which has adopted the protocol.

The Sitemaps protocol used by Google has been widely adopted by many Web properties, including sites from the Wikimedia Foundation and the New York Times Company. Any company that manages dynamic content and a lot of web pages can benefit from Sitemaps. For example, if a company that utilizes a content management system (CMS) to deliver custom web content – (i.e., pricing, availability and promotional offers) - to thousands of URLs places a Sitemap file on its web servers, search engine crawlers will be able discover what pages are present and which have recently changed and to crawl them accordingly. By using Sitemaps, new links can reach search engine users more rapidly by informing search engine “spiders” and helping them to crawl more pages and discover new content faster. This can also drive online traffic and make search engine marketing more effective by delivering better results to users.

For companies looking to improve user experience while keeping costs low, Sitemaps also helps make more efficient use of bandwidth. Sitemaps can help search engines find a company’s newest content more efficiently and avoid the need to revisit unchanged pages. Sitemaps can list what is new on a site and quickly guide crawlers to that new content.

“At industry conferences, webmasters have asked for open standards just like this,” said Danny Sullivan, editor-in-chief of Search Engine Watch. “This is a great development for the whole community and addresses a real need of webmasters in a very convenient fashion. I believe it will lead to greater collaboration in the industry for common standards, including those based around robots.txt, a file that gives Web crawlers direction when they visit a website.”

"Announcing industry supported Sitemaps is an important milestone for all of us because it will help webmasters and search engines get the most relevant information to users faster. Sitemaps address the challenges of a growing and dynamic Web by letting webmasters and search engines talk to each other, enabling a better web crawl and better results," said Narayanan Shivakumar, Distinguished Entrepreneur with Google. "Our initial efforts have provided webmasters with useful information about their sites, and the information we've received in turn has improved the quality of Google's search.”

“The launch of Sitemaps is significant because it allows for a single, easy way for websites to provide content and metadata to search engines," said Tim Mayer, senior director of product management, Yahoo Search. "Sitemaps helps webmasters surface content that is typically difficult for crawlers to discover, leading to a more comprehensive search experience for users.”

“The quality of your index is predicated by the quality of your sources and Windows Live Search is happy to be working with Google and Yahoo! on Sitemaps to not only help webmasters, but also help consumers by delivering more relevant search results so they can find what they’re looking for faster,” said Ken Moss, General Manager of Windows Live Search at Microsoft.

The protocol will be available at sitemaps.org, and the companies plan to have Yahoo Small Business host the site. Any site owner can create and upload an XML Sitemap and submit the URL of the file to participating search engines.

Posted by Danny Sullivan on Nov. 16, 2006

Trademarked Keywords

NOVEMBER 17, 2006

A legal gray area.

By Ben Macklin - Senior Analyst

Over the last few years, there have been a variety of cases brought before the courts both in the US and overseas relating to search engines selling trademarked words as keywords. For example, is it legal for Coca-Cola to buy the keyword "Pepsi," so that every time a user types in "Pepsi" she receives the sponsored advertising listings of Coke?

The answer is that the issue is still a little gray. A recent ruling in a case brought by GEICO, a US insurance company, is illustrative. This decision allowed Google to sell the search term "GEICO" to the highest bidder, as long as the "sponsored link" that was served up with any search results did not contain the term "GEICO". For background on relevant cases, see this blog.

Google's trademark complaint procedure for US and Canadian companies includes the following statement (available here):

"When we receive a complaint from a trademark owner, we only investigate the use of the trademark in ad text. If the advertiser is using the trademark in ad text, we will require the advertiser to remove the trademark and prevent them from using it in ad text in the future. Please note that we will not disable keywords in response to a trademark complaint."

This would seem to confirm the GEICO ruling: that the text within the sponsored link is more important in terms of trademark infringement than the keyword that triggers delivery of the sponsored link.

However, this does differ from Google's statement with regard to companies outside the US and Canada:

"When we receive a complaint from a trademark owner, our review is limited to ensuring that the advertisements at issue are not using a term corresponding to the trademarked term in the ad text or as a keyword trigger."

Survey data from MarketingSherpa certainly seem to indicate that companies are becoming more aware and particular about how their trademarks are used in search marketing.

Companies, of course, have a great incentive to protect their trademarks and their overall brand. In search marketing, branded keywords are more likely to be clicked by Internet users than non-brand keywords, according to 360I.

eMarketer explains where best to reach the mobile customer-- whether SMS, voice or MIM.

It is logical but not quite right to deduce that Americans simply love the sound of their own voices. A closer look at most U.S. voice rates in comparison to other countries suggests that talk is cheap in the United States vis-à-vis the rest of the world. Data published by the Telecommunications Management Group (TMG) in January 2006 showed that average American mobile voice usage is four times higher than it is in Europe.

However, U.S. operators are noting a significant uptick in messaging-related traffic over the last year. According to CTIA -- The Wireless Association -- U.S. operators recorded a 71 percent increase in SMS traffic in June 2006 (12.5 billion messages) compared to June 2005 (7.3 billion messages). On a six-month basis, the U.S. figures are even more impressive, with 98.8 percent growth in the first half of 2006 (64.8 billion messages) compared to the same period of 2005 (32.6 billion). CTIA also reported that by June 30, 2006, U.S. operators had recorded $6.5 billion in mobile data revenues. Using a 70 percent rule of thumb for the proportion of messaging revenues to total mobile data revenues according to previous estimates by Morgan Stanley, the U.S. messaging market might reach about $9.1 billion by the end of 2006.

When M:Metrics surveyed U.S. mobile users about the mobile data services they access over a rolling three-month period, text messaging came out on top at 38 percent, which is well under European, let alone Asian, norms. However, use of MIM was higher at 7.4 percent, probably due to the fact that more Americans are IM users as a matter of course.

On the handset side, U.S. mobile customers are placing greater value on the ability to send and receive messages. J.D. Power and Associates reported in May 2006 that sending/receiving SMS messages (22 percent) ranked just under use of the speakerphone (26 percent) as the handset feature American users turned to the most after voice calling.

However, all is not smooth sailing in the U.S. messaging growth story. Granted, the last 18 months has seen a substantial shift in messaging traffic that suggests the market is maturing into mainstream adoption, but a Harris Interactive study in June 2006 revealed that many Americans still do not understand why much of the world is worked into a lather over messaging. Over 70 percent of those surveyed didn't see a need for using their mobile handset for messaging.

The plot thickens, so to speak, when messaging is tied to another great American pastime, watching television. "American Idol" on Fox Television planted SMS voting into the conscious of mainstream America, even though in its first few seasons it was available only to AT&T Wireless, and later Cingular, customers.

However, those early days have spawned a new cottage industry of mobile-led referendums on everything ranging from whether a given Idol singer deserves a shot at the next round or a sports player will or will not score, as well as local issues mediated largely by radio. According to a Mobile Data Association/M:Metrics survey conducted in January 2006, nearly a quarter of the highly-sought-after 18 to 34 demographic participated in some form of SMS voting, either through television or radio. The same study revealed a 60 to 40 preference by men for SMS voting, a data point sure to throw gasoline on the fire of marketers' interest in reaching this population.

John Gauntt is a senior analyst at eMarketer. This article was drawn from his report Mobile Message Marketing: Cash Not Flash.

Report: Mobile ad spending to double to $2.9B by 2011

Spending on mobile messaging and display ads is forecast to double to $2.9 billion by 2011, from $1.4 billion this year, according to JupiterResearch. Right now, 22% of online marketers, including wireless carriers, media organizations and automotive and financial service companies, also are advertising via mobile devices.

Hyatt invests in new online booking technology

October 18, 2006

OpenJaw Technologies has signed a contract with Hyatt Hotels and Resorts to provide its Internet booking engine technology for travel distribution, xRez.

The solution from the Dublin-based company will enable Hyatt’s global customer base to make Hyatt Hotel Reservations online in up to 17 languages including Chinese, Japanese and multiple European languages. Japanese property websites are currently in production.

The booking engine technology is utilizing XML (eXtensible Mark up Language) standards from the Open Travel Alliance to connect to the Hyatt Central Reservations System (CRS).

Uniquely, the solution also detects content changes in the Hyatt CRS and performs an automatic language update at regular intervals by connecting to Hyatt’s translations partners, Translations.com, Global Link service, according to an official release.

The OpenJaw solution is a fully managed service hosted out of Dublin. OpenJaw contributes to the ongoing development of the OTA XML specifications, an open standard messaging system that facilitates improved connectivity between travel companies and their distribution partners, better packaging of travel products, and lower distribution costs.

Reliance on search is scary prospect: Liberty Media exec

October 13th, 2006

NEW YORK – Gregory Maffei, president/CEO of Liberty Media Corp., shared his forecasts on the future of e-commerce during the Oct. 12 keynote called “Differentiating in a Converging World” at Shop.org’s annual summit at the New York Hilton.

Mr. Maffei based his examples largely on what the company has done with home-shopping brand QVC, which Liberty Media purchased in 2003. An avenue for worldwide retail, the outlet ties its entertainment broadcast to purchase opportunities across various platforms including the Internet and mobile, with a focus on brand and viewer community.

“I think it’s inevitable,” Mr. Maffei said. “You’ll see more and more commerce embedded in community.”

Opportunities in digital community growth need not be restricted to the younger generation, he said, referring to sites MySpace and similar sites. In fact, he said that the number of teens on MySpace had dropped to only 12 percent and more than half of the site’s visitors were older than 35.

One forecast he said online retailers should pay attention to was the aging population. Rather than targeting “millennials,” the population segment 18-24, QVC and other Liberty Media properties are watching older consumers who are more stable and more affluent.

As international markets open up, use of digital devices increase and more consumers demand online video, Mr. Maffei said a serious obstacle to online business could be the rising cost of clicks on major search engines.

The growth of vertical search engines, which provide a focused place for searchers and the potential for better leads for marketers, may be one solution.

“If you want to make and control your own destiny I think reliance on search to drive your business is a very scary prospect,” Mr. Maffei said.

Walking TV Nets Nivea 6,600 Potential New Customers

A couple years ago, we told you about a technology that mounted TV's on people so they could walk around and sell stuff. Now, everyone's doing it including Nivea who contracted with AdWalkers, trained street walking marketers who wear TV's and hand out stuff, to promote the company's "Nivea Touches New York" Exhibit.

Nivea deployed eight Adwalkers in its first week of operation and four during its second week. The Adwalkers fanned out around Chelsea, Union Square, Gramercy Park, and Herald Square on a Wednesday through Saturday basis. Of the people exposed to the AdWalkers, a total of 6,600 took a virtual tour of the Nivea exhibit and got a printout reminder/invitation to visit the West 19th Street installation.

Src: AdRants.com