Tuesday, December 14, 2004

Google Suggest Dissected...

People have been contributing their two cents to how this works, but I have un-compressed (ie. re-written) Google's compressed javascript, so that the average web developer should be able to get a detailed understanding of how this works.... My final rewrite is available from my website here.

I saw the coolest thing I've seen since realizing that Mozilla was embedding a wsdl-enabled SOAP client into this browser... Google Suggest returns suggested results as you type... This is technically amazing on about at least two different levels:


  1. How fast this is... I type pretty fast, and it updates with every single keypress...
  2. The cool web interface... I used to be pro-server side web updates, and avoiding javascript, but I'm really turning around on this with the impressive interfaces I've seen with gmail, and now google suggest (among others...)


So everyone is impressed by this... My shock and awe goes further in terms of how nice this interface works:


  1. That the suggestion list lines up perfectly with the query input field...
  2. The high-lighting of the additionally suggested text (I type "fa", it suggests "fast bugtrack" and highlights the "st bugtrack" so that the next character I type wipes out it's suggestion... beautiful...)
  3. The great handling of keypresses (cursors up and down...)
    And After going through googles code:
  4. How the javascript cache's the dynamic results so that if you backspace, it doesn't have to go back to google...
  5. How the code dynamically adjusts it's main (time/alarm) driven loop based on how quickly you're getting results back from google...


So I wanted to understand the web interface and it's dynamic interface... Just a note that the good and brilliant folks at google wrote all of the code we'll be looking at here this evening... I didn't write any of it, but I will be stepping through it with you, and hopefully helping to improve everyone's understanding of this great dynamic web interface...

A couple of tips for how I went about reversing the logic here:

  1. I saved the html and javascript locally... I managed to get a local copy running, and placed some alerts into the code to observe behaviour as well as using the javascript console to catch places I made mistakes renaming variables and functions...
  2. The google code uses an XMLHttp object to make calls back to google, and executing the results... to fully understand the code, I need to see what google is sending back... BUT when I tried the url directly, I didn't get anything but a 404 back from google (it turns out I had mis-typed the generated url...)... I tried to have my browser go through a local proxy server, but it appears that the XMLHttp object doesn't use the browsers proxy when communicating (which means that this might not work if you're behind a proxy server... Can people confirm this??) ... I would have fallen back on a packet sniffer to capture the data, but caught my mistake in the URL before reaching this point...


Looking at the main page source, just go to google and view source... At the bottom of this file, we can see a reference to javascript which drives the dynamic interaface (available directly from google here...)

The good folks at google compress their code as they should, so in order to understand it, I first re-indented it as can be seen here... Then I began the fun process of figuring out what the global variables are for, and what the various functions do, and renaming them to meaningful names... I made it pretty far as can be seen in my final re-write of Google's suggest javascript codehere

Things I didn't know before this exercise that I learned going through this...
1) You can turn the browsers autocomplete off by adding autocomplete="off" attribute to an input field... How did I not know this before...
2) The XMLHTTP / XMLHttpRequest object to communicate back with a server and get new info / instructions without refreshing the page ... the new black of web development... go read everything you can about this...
3) How powerful the keypress handling can be with javascript... (capturing keyup/keydown and events and changing state for cursor key events, etc...)
4) You can highlight text in an input field using javascript...


Stepping through it:

The html page calls InstallAC()...
This set's up the system... An interesting line:
var Jb="zh-CN|zh-TW|ja|ko|vi|";
So while they say they support English only, there is definitely code that looks for locales in Japan, Korea, and China and handle requests appropriately...

The installAC function calls another function (I called installACPart2)... This function checks that our browser supports XMLHttp, creates what I call the "_completeDiv" ... the DIV inwhich google suggestions will be populated when we get data back from google... It uses absolute positioning to line it up with the input text field, and is intially hidden...
The installACPart2 function also sets up some keydown and resize event handlers...It also begins the creation of the url for which we will be making our dynamic requests to google...

The function I called mainLoop sets itself up to be called repeatedly using the javascript setTimeout function... It's interesting to note that the designers decided to use this timeout based mechanism rather than the keydown mechanism... This would handle fast typers on slow connections (so if I typed 3 characters between timeouts, a single request would go out to google...) The mainLoop checks if the state of the input field has changed and if so, takes action - looking first in the result cache, then making a call out to google... The google suggestion code also handles older browsers that don't have an XMLHttp object by using cookies and frame reloading (I haven't tried this yet...)

The callGoogle routine is fairly straight-forward... I makes calls of the format (if I am in an English locale, and have typed "fast bug"):
http://www.google.com/complete/search?hl=en&js=true&qu=fast%20bug
It sets up a callback _xmlHttp.onchange event function, that will simply evaluate the (what ends up being a javascript funciton) that gets returned from google...
What gets sent back looks like this:

sendRPCDone(frameElement, "fast bug", new Array("fast bug track", "fast bugs", "fast bug", "fast bugtrack"), new Array("793,000 results", "2,040,000 results", "6,000,000 results", "7,910 results"), new Array(""));

The sendRPCDone function is defined in the ac.js file... It adjusts timing in the mainloop, caches the results received, sets up the _completeDiv DIV with the result arrays, and ultimately ends up displaying this DIV....

The function displaySuggestedList takes the results and dynamically creates a series of DIV and SPAN data structures (using the DOM model) that ultimately form the suggestion list that gets displayed... For each element in our list, our data structure looks something like: (where (x) is the variable in the code)

<DIV (u) - mousedown/mouseover/mouseout class="aAutoComplete">
<SPAN (ka) class="lAutoComplete">
<SPAN (ua) class="cAutoComplete">
bug tracking
</SPAN (ua)>
<SPAN (ea) class="dAutoComplete">
500,000 results
</SPAN (ea)>
</SPAN>
</DIV (u)>


The Pa() function [I never came up with a satifactory name] gets called when results are received and whenever a key is pressed (and perhaps on some of the mouse events as well(?)... It does the high-lighting of text that we didn't type...

You'll want to look over and step through the code yourself to truly understand it... Let me know if you have any questions or comments... There is a good chance that I made a typo or two as I renamed things...

138 Comments:

Anonymous Anonymous said...

Thanks for your work!

December 14, 2004 2:56 AM  
Anonymous Anonymous said...

I know, what you have done is useful to many people and will help improve the usability of many rich webapps, but are you sure about legality of re-publishing Google copyrighted code?

Few months ago, I did a similar exercise with gmail JavaScript code and was tempted to publish the un-compressed code, but then better sense prevailed.

December 14, 2004 12:12 PM  
Blogger Sri Sankaran said...

Great sleuthing! BTW, it works behind a proxy server.

December 15, 2004 12:55 PM  
Anonymous Anonymous said...

Wonder what they did on the server side to make the response time so quick. That is an AMAZINGLY fast reponse time for data over the web....not to mention the fact that it comes out of a database of some sort.

December 15, 2004 2:13 PM  
Anonymous Anonymous said...

Well Done Chris, thanks for making me know this service and its internals.
Tamer Salama
http://www.jroller.com/page/tamer

December 15, 2004 2:24 PM  
Anonymous Anonymous said...

You just got slashdotted.

December 18, 2004 4:13 AM  
Anonymous Anonymous said...

When do you start work for Google? ;-)

December 18, 2004 4:15 AM  
Anonymous Anonymous said...

It works behind a proxy.

December 18, 2004 4:36 AM  
Blogger mdonatas said...

"The google suggestion code also handles older browsers that don't have an XMLHttp object by using cookies and frame reloading"

Man, I just love how Opera 7.6 falls into this category ;) I don't get these folks, they add so much bullshit crap to opera but can't add something which is becomming so usefull these days.

December 18, 2004 4:38 AM  
Anonymous Anonymous said...

Dunstan Orchard has had this on his site for a while: see the LiveSearch box.

December 18, 2004 4:40 AM  
Anonymous Anonymous said...

Great job. Even greater that shared your findings.

December 18, 2004 4:47 AM  
Anonymous Anonymous said...

I can't confirm your "It looks like it won't work behind a proxy".

It works behind a proxy.

I even made some tcpdump and I can reply it with telnet.

See: http://www.megabot.nl/public/googledump.txt

December 18, 2004 4:48 AM  
Anonymous Anonymous said...

Now is only someone can disect the code used at amazons www.a9.com to dynamically resize table columns.

December 18, 2004 4:54 AM  
Anonymous Anonymous said...

Great job
.. and don't worry about the legal side of things: JS and html code is just not meant to be private!

December 18, 2004 5:19 AM  
Anonymous Anonymous said...

Nice stuff. Thanks for sharing ^^

December 18, 2004 6:05 AM  
Anonymous Anonymous said...

Chris... Great job. What a pain it would have been to rename all those functions, but you did an A+ job and I will be going over the code tomorrow.

Thanks!

Ryan
Toronto, Ontario, Canada

December 18, 2004 6:18 AM  
Anonymous Anonymous said...

If you'd used the excellent LiveHTTPHeaders extension to Firefox you'd see all calls from your browser to Google's serverside script and you could easily see what google sent back to your browser.

December 18, 2004 6:52 AM  
Anonymous Anonymous said...

>are you sure about legality of re-publishing Google copyrighted code?

It is a sad state when the media has so demonised internet piracy that people are afraid to do anything. The fundamental basis behind copyright is that it protects *EXPRESSION*, not *IDEAS*. Thus you can attempt stuff like Wind Done Gone but not reprint Gone With the Wind. There are also other criteria such as creativity etc but I won't go into the boring specifics. Also different jurisdictions have different exemptions, for example US Fair Use or Commonwealth Fair Dealings.

In short, a technical commentary which addresses a completely different market from intended, with reverse engineering to create similar functionality but expressed differently is probably legit [IANAL] depending on the precise jurisdiction. There may be mutters about theft of service or other forms of economic torts in abusing the google web server but that's a different issue
--
LegalEaglet

December 18, 2004 7:17 AM  
Anonymous Anonymous said...

Informative... thanks for sharing.

Azmeen

December 18, 2004 7:25 AM  
Anonymous Anonymous said...

pretty good stuff..
btw here the url for the google autocomplete dropdown

http://www.google.com/complete/search?hl=en&js=true&qu=java

The last one is the actual term.

check it out.
thanks
Murali Varadarajan

December 18, 2004 7:34 AM  
Anonymous Anonymous said...

I just love JavaScript! :)

/ MackanZoor

December 18, 2004 7:44 AM  
Blogger Richard Soderberg said...

"Pa" could be "ParseRPCAnswer()".

December 18, 2004 7:54 AM  
Anonymous Anonymous said...

This comment has been removed by a blog administrator.

December 18, 2004 8:32 AM  
Anonymous Anonymous said...

> There is a good chance that I made a typo or two as I
> renamed things...

With all due respect, your posting would be even more
impressive and credible if you would bear in mind that
words like "its" and "caches" do not have apostrophes
in them.

December 18, 2004 8:34 AM  
Anonymous Anonymous said...

Does Google Suggest prove that Google is using alien technology from the future?

December 18, 2004 9:15 AM  
Anonymous Anonymous said...

Not to get to far off topic, but "its" does in fact sometimes use an apostrophe -- "It's" is the contraction for "It is."

Great work! I've certainly found this to be most interesting!

December 18, 2004 9:16 AM  
Anonymous Anonymous said...

I...think you put dots...where they don't really belong...they add nothing but show how poorly...you write.

-nb

December 18, 2004 9:56 AM  
Anonymous Anonymous said...

Fiddler (http://www.fiddlertool.com) would save you a lot of time as you were trying to determine what you were sending up and Google was sending back.

December 18, 2004 10:12 AM  
Anonymous Anonymous said...

Guys, chill on the grammar. He *is* Canadian.

:)

December 18, 2004 10:13 AM  
Anonymous Anonymous said...

For your next task - adapt the Javascript to a Firefox search bar.

December 18, 2004 10:22 AM  
Blogger Nadeeshyama said...

Google has definitely shown the power of Javascript with its Gmail. And here they go again with Google Suggest!

You have done a nice job explaining what goes 'behind the screen'.

December 18, 2004 10:43 AM  
Anonymous Anonymous said...

Well done, very useful - it appears that Google is using some kind of Trie data structure at the server side, which is what's giving them such fast response times - for more info see here:
http://c2.com/cgi/wiki?StringTrie

December 18, 2004 11:03 AM  
Anonymous Anonymous said...

mmmmm I would really love this for firefox google box.

December 18, 2004 11:07 AM  
Anonymous Anonymous said...

I haven't checked, but it wouldn't surprise me if they were using a compressed content attribute on the XML HTTP stream; that makes the rendering of web downloads quite impressive.

December 18, 2004 11:27 AM  
Blogger kordless said...

Yeah, that XMLHttpRequest stuff is good stuff. We use it on our site at Zoto to send back tag updates on photos without reloading the page.

At the time we implemented it, Opera didn't support the request call, so our web dev guy hacked up a function that emulated it for Opera using image requests.

Nice analysis of the script BTW. Kind of makes you wonder if it's worth obfuscating the code....

December 18, 2004 11:37 AM  
Anonymous Anonymous said...

What we need now is for someone to build (and sell us?) a simple toolkit that provides this functionality for those of use who don't want to use this in our apps.

Whoever does this please provide implementation for the following: Coldfusion, JSP, JSF, ASP, PHP.

I'd buy such a component right now (for say, $30 bucks?).

December 18, 2004 11:48 AM  
Anonymous Anonymous said...

Learn how to use apostrophes correctly!

http://kubed.org/archives/2004/12/07/apostrophe-abuse/trackback/

December 18, 2004 11:55 AM  
Anonymous Anonymous said...

FYI: Google Suggest works from behind the proxy server / firewall in my company.

December 18, 2004 12:08 PM  
Anonymous Anonymous said...

Great Job,

I'm glad you see the value in getting new data without a new page request. This just highlights how incorrect normal web developer assumptions can be. That the heavy lifing is reserved for the server side while JavaScript handles only form validation and saying "Good Morning".

The new black indeed!

Ken

December 18, 2004 12:15 PM  
Anonymous Anonymous said...

Please proofread this, or have it proofread. I'm probably more peevish about this than most people, but I couldn't finish it with all the inappropriate uses of apostrophes. Thanks.

December 18, 2004 12:49 PM  
Anonymous Anonymous said...

> Nice analysis of the script BTW. Kind of makes you wonder if it's worth obfuscating the code....

i'm guessing google uses funny short variable names not to obfuscate, but to compress the data so it gets to your computer faster.

December 18, 2004 1:21 PM  
Blogger bloodnok said...

the possessive form of "it" is "its". "it's" is a contraction of "it is".

December 18, 2004 2:07 PM  
Anonymous Anonymous said...

thank you very much for the work. this same idea for dynamic shrinking of the search term space was in a pre-internet CDROM product called Computer Select. it was a collection of all the articles from PCWeek, PCMagazine, etc. etc. and it was really nice to be prompted for product model names, company names, etc. as they were typed into the search interface.

i've been dying to see it show up on the web.

December 18, 2004 2:16 PM  
Anonymous Anonymous said...

Great work!
Well done...
Joe.

December 18, 2004 3:13 PM  
Anonymous Anonymous said...

Yeah, but check out this: http://aminaked.com/poker
And another thing, Google Suggest is cool and also completely useless.

December 18, 2004 3:43 PM  
Anonymous Anonymous said...

Thanks for sharing!

December 18, 2004 4:16 PM  
Anonymous Anonymous said...

var Jb="zh-CN|zh-TW|ja|ko|vi|";

The line above is more than for just Chinese, Japanese and Korean. It covers Simplified Chinese (used in mainland China), Traditional Chinese (used in Taiwan and Hong Kong), Japanese, Korean and Vietnamese.

December 18, 2004 4:26 PM  
Anonymous Anonymous said...

grammar nazis back off. this is a blog. that means anyone can write any which way they please. it's not formal. wait! its not formal (j/k :P).

this is not a white paper on the inner workings of google. this is his analysis posted to his blog so technical people who want know how google works can have a peek. this is not for english professors. can you understand the explanation? if yes, then shut up. if not, then might i suggest a bunch of "x for dummies" books? buy javascript for dummies and internet for dummies and http for dummies and even xmlhttp for dummies (although msdn does a nice enough job of it for their version of this).

let the man speak. he's not here to win an elite english writing prize. sheesh, if you don't understand the technical stuff stay away. if you do understand the technical stuff, well that is the purpose of this blog entry.

* most punctuation intentionally omitted for your temperament.

December 18, 2004 5:09 PM  
Anonymous Anonymous said...

The XMLHttpRequest object is one of the best discoveries I have made in Javascript this year. I've been using it like this for a few months, now, and it does wonders for making 'application like' web sites.

Being able to reload small portions of a web page is great for saving bandwidth and server usage too...

December 18, 2004 5:35 PM  
Anonymous Anonymous said...

Nice to see people discovering the XMLHTTP component. I've been using this technique for over three years (via the XMLHTTP component and other methods).

December 18, 2004 8:41 PM  
Anonymous Anonymous said...

Interesting article, but PLEASE learn to write without using the ellipsis. It's a crutch; and at the rate you use them, it's distracting.

December 18, 2004 10:01 PM  
Anonymous Anonymous said...

>Guys, chill on the grammar. He *is* Canadian.

heh, and I guess you're from the USA, a place well known for its strict use of the English language... oh hang on, that's right you guys just made up your own bastardised version. BTW I'm from Australia, where we is all real good wiff riten n stuff :)

December 18, 2004 10:41 PM  
Anonymous Anonymous said...

Pretty nice stuff, I wanted to see if i can make it into behavior and it works out pretty nicely. anyone who wants the .htc file can email me at junk.huskies@gmail.com

December 18, 2004 11:06 PM  
Anonymous Anonymous said...

Grat to see people using the XmlHttp component. I personally invented it 18 years ago (and three months or so)

December 19, 2004 12:32 AM  
Anonymous Anonymous said...

I personally invented it 20 years ago, on an Alto, in assembler, over BITNET.

December 19, 2004 12:37 AM  
Anonymous Anonymous said...

For those curious about the language support -- I posted a much higher-level analysis of this feature when it was first discussed on Slashdot, and got dozens of responses confirming my guess about suprisingly wide language support (even though Google's FAQ apparently says something about adding support for other languages later).

Details here.

December 19, 2004 3:05 AM  
Anonymous Anonymous said...

xmlhttprequest object is first developed by Microsoft, not mozilla. Basically it is due to Microsoft's efforts to extend basic web technologies, we have google and other companies implementing such technologies. If it was left to Microsoft bashers, we clearly wouldn't have these services today. Just a quick observation who is really behind the web.

December 19, 2004 6:17 AM  
Anonymous Anonymous said...

don't forget to take out the alert on line 686.

other than that, its pretty slick when it gets working. very fast, and very clever.

but i wish there was an easy way to identify a primary key with the dynamic menu selection - but google uses v.value for copying, instead of indicies which could be extracted via dom.. bit of a bummer until you get to work around it..

December 19, 2004 7:37 AM  
Blogger Jessta said...

it's interesting that nobody will have the bandwidth that google has to make this work as nice.

December 19, 2004 5:09 PM  
Blogger Adfero Affero said...

Can't follow the technical stuff, but have an interest in this because I think it is termed a PAL [predictive adaptive lexicon], which they tried to use with dyslexics using word-processors. Probably dozens of PALprogs out there.

A cursory Google [ordinary :) ] gives this article with some detail

http://www2.edc.org/NCIP/library/wp/Newell.htm


I explain in a little bit more detail what I think in regard to Google Search use by dyslexics at

www.weblogworld.blogspot.com

December 20, 2004 5:38 AM  
Anonymous Anonymous said...

Ss co-developer of PostOfficeBox, i have a parallel project located at:
http://mangoduck.org/postofficebox/jsqsocket/demo/1/

Source Forge progect page:
http://sourceforge.net/projects/postofficebox/

When I have time, hopefully soon, I intend to write an autocomplete demo nearly identical in function to google suggest, but using JsqSocket. The one hangup with this method is it works in all "modern" browsers except Safari (as of version 1.2.3), but can be implemented to degrade into a normally posting interface when needed.

December 20, 2004 11:23 AM  
Blogger black said...

Above was me. Forgive my typos!

December 20, 2004 11:26 AM  
Anonymous Anonymous said...

Great work!
I can't believe the anal grammar people that are commenting, it gives on the feeling that they couldn't parse the content so picked at the grammar. Especially those that chose to repeats grammar comments that had already been made.

Anyhoo Great Work, and keep it up!

December 20, 2004 12:32 PM  
Anonymous Anonymous said...

Javascript still sucks. :) As soon as Google decides to make a change in their JS file, it's going to take a good week for that change to propagate out across the web, what with browsers caching that JS file, AOL caching that JS file, etc. And in the meantime, lots of people will be seeing a broken Google Suggest page. Not to mention the fact that you've got to do double- (or triple) work to support older browsers that don't support the XML object. Sure, it's a neat feature, but I'm glad I don't have to maintain it. ;)

December 20, 2004 1:25 PM  
Anonymous Anonymous said...

about the grammar: i read the ellipses quite naturally as pauses in normal conversations. it makes the reading very easy to follow, avoiding language constructs in favor of technical content. it gets straight to the point on what his impressions are and what the facts are.

my credentials: i was reading lotr before the fifth grade. i had a reading diet of 2 stephen king novels per week in grade 7. my poor myopic eyes spend roughly 3 hours a day reading. i finished war and peace last month. i volunteer my time to tutor adults in literacy.

and i have no problem with the way this guy writes.

December 20, 2004 1:28 PM  
Anonymous Anonymous said...

Thanks for the analysis. I had tried XmlHttp previously but switched to background requests in hidden frames, because it works with more browsers. Hopefully XmlHttp becomes better supported in the near future.

As for the grammar Nazis, be sure your grammar is correct before complaining ;-)

And for those who don't think grammar or punctuation matter-- bad grammar and punctuation does tend to reflect on the person, and can be distracting even if the meaning is discernible.

Just use a browser with built-in spell checking (like Mozilla with extensions) :-)

December 20, 2004 1:49 PM  
Anonymous Anonymous said...

Just wonder, how much time did you take to re-write this stuff ?

December 20, 2004 10:36 PM  
Blogger Chris said...

Just a note that I've added a short follow-up to this article...

http://serversideguy.blogspot.com/2004/12/google-suggest-dissected-follow-up.html

It took a total of 8 hours over 2 evenings to transform the code from it's initial state to the end result I got it in. I could have gone further, but felt I had reached a pretty good point, and didn't want to eat up any more time. :)

December 20, 2004 11:08 PM  
Anonymous Anonymous said...

I like this javascript server call-back method, having the server return Javascript is a relatively elegant method to do pretty much anything. I found a method to do this some time back, using invisible IFRAMES the XMLRequest method is a good addition to this method. Check out the following http://developer.apple.com/internet/webcontent/iframe.html. Maybe one day the entire internet will be like one big app.

December 21, 2004 5:42 AM  
Anonymous Anonymous said...

Cmon people, quick whining about his spelling or grammer, he did you a FREE SERVICE!

Looks awesome by the way!

December 21, 2004 2:20 PM  
Anonymous Anonymous said...

As Ralph Wiggum said: "Me fail English ? That's unpossible"

Great work Chris... ;)

December 21, 2004 2:57 PM  
Anonymous Anonymous said...

I'm just chiming in with my 2 c. Grammar Nazis - BACK OFF! Nobody forced you to read this blog. If you don't like it, just take your anal selves and move it along.

There are many of us who appreciate the effort put in and can learn from the post without being distracted by the relatively minor punctuation slip-ups.

Great job with the reverse engineering!

December 21, 2004 5:17 PM  
Blogger black said...

"As soon as Google decides to make a change in their JS file, it's going to take a good week for that change to propagate out across the web, what with browsers caching that JS file, AOL caching that JS file, etc. And in the meantime, lots of people will be seeing a broken Google Suggest page."

Following that same logic, you can't change ANYTHING on the web -- scripts, images, markup, shockwave and flash, css -- without fear of it being cached somewhere. It's not realistic to hold back a great feature or critical change because some user agents cache too aggressively.

Also, if it does break, what do you get? A working search page like we have now. Is that so bad? Much of the elegance of this feature is that it can never hinder any other functionality.

"Not to mention the fact that you've got to do double- (or triple) work to support older browsers that don't support the XML object."

Not even double. For agents lacking xmlhttprequest, Google Suggest uses an iframe stashed ten thousand pixels offscreen as a target for the lookup-as-you-type requests and copies the resulting innerhtml from it, evaluating that text as javascript code. The innerhtml is accessible to the script because the iframe src is within the same domain. The only additions to the main script are a flag signifying the presence/absence of xmlhttprequest, and functions to pipe the data through the iframe instead of running the request. There is only one set of functions to capture events and display the list.

The only advantage I see to xmlhttprequest is it runs at a lower level than JsqSocket/PostOfficeBox would (dynamic script objects) which makes it very responsive, though only as responsive as any http request can be.

The turnoffs of this method for me are limited backward compatibility, and the fact that the object was a product of MS and related to activex to start with. Only more recently have other vendors picked it up, presumably because they thought it couldn't hurt to do so.

December 21, 2004 7:44 PM  
Anonymous Anonymous said...

Great Work and its very good you share ur finding.....Hats Off to u

December 22, 2004 6:04 AM  
Anonymous Anonymous said...

Chris> I tried to have my browser go through a local proxy
Chris> server, but it appears that the XMLHttp object doesn't
Chris> use the browsers proxy when communicating (which means
Chris> that this might not work if you're behind a proxy
server... Can people confirm this??) ...

No, I can't confirm this. The opposite happens:
I'm behind a corporate firewall and all works ok

Anton Tagunov

December 22, 2004 6:17 AM  
Anonymous Anonymous said...

You guys have clearly never used convea.

We used the XMLHTTP object some years ago to create a web based conferencing system that does not require a page refresh to send or retrieve messages.

The technology is nothing new; the implementation however is sweet!

December 22, 2004 10:48 AM  
Anonymous Anonymous said...

This is Durrani from India.

It does work behind the proxy also...

Cool google...

December 23, 2004 3:13 AM  
Anonymous Anonymous said...

This comment has been removed by a blog administrator.

December 23, 2004 4:04 AM  
Anonymous Anonymous said...

This comment has been removed by a blog administrator.

December 23, 2004 5:55 AM  
Anonymous Anonymous said...

Yeah, great work man! It's amazing! =]]]

Hum... have you ever seen that: http://gmail.google.com/gmail?view=page&name=js&ver=f81f5040341823b5

I've looked fast, but i think that has something nice there... =]

bohzzu@gmail.com

December 23, 2004 2:59 PM  
Anonymous Anonymous said...

There is nothing new about what google is doing. There has been an xml-rpc lib for js for a while now.

http://www.vcdn.org/Public/XMLRPC/

December 24, 2004 10:51 AM  
Anonymous Anonymous said...

Woah!! Great work! Thanks for sharing your findings...

December 24, 2004 11:29 AM  
Blogger Alan said...

Wonderful dissection. I propose you to dissect this other trivial (or at least to me when I first look at it) image protection that Google have used for their "Google Print" pages.

For instance, a search on google for "books on mastering digital photography" brings you several "Book results".

Click any of them, and now try to obtain the image of the page. (Getting it from your browser cache would be cheating.) If you check the page source, you'll see they use some very confusing techniques to protect the images on the "Google Print" pages. I'd love to learn how and what they are doing exactly.

Cheers,
-- Alan

December 24, 2004 1:11 PM  
Blogger gavi said...

I just finished implementing Google suggest for a dictionary database.

http://www.objectgraph.com/dictionaryThe code is clean and you could see it by using "View Source" or there is a link "how does it work?" where you will see the server code aswell.

The dictionary database is on an SQL server (total of 18000+ words) with an index on the word column.

December 25, 2004 3:29 PM  
Anonymous Anonymous said...

Alan -

The page images are implemented as background images of an area. I've seen this before, so I only got far enough into it to confirm my suspicion that this is what is going on with Google Print. The most common way to do it is to have a table cell with height and width set to the size of the image, and the background picture be the image URL. Then, use a clear GIF of the same size as the actual picture as the contents of the table cell. If/when the user right clicks on the "image" they see, they'll be clicking on the clear spacer GIF, and the background image will be slightly more protected. Google uses the same idea, it appears, with CSS instead of tables.

December 27, 2004 12:35 PM  
Anonymous Anonymous said...

I found the following error in the script:

This piece of code:

if(event&&event.keyCode==8){
if(X&&(_inputField.createTextRange&&(event.srcElementa&&(bb(_inputField)==0&&lb(_inputField)==0)))){



Should be:


if(event&&event.keyCode==8){
if(X&&(_inputField.createTextRange&&(event.srcElement==_inputField&&(bb(_inputField)==0&&lb(_inputField)==0)))){

December 28, 2004 4:44 PM  
Blogger Jonathan said...

I took the decompressed script and created a working demo. The only trouble I'm having is related to the css. In IE6 it looks like the right alignment for the # of search results only works in quirksmode. Has anyone came up with a workaround?

December 28, 2004 6:39 PM  
Anonymous Anonymous said...

I found a bug with the Google suggest feature:

Enter a name to search. Example: apple

Tab all the way back to the search field using the tab key

The value of the search field disappears ! (verified with Internet Explorer 6 and Firefox 1)

December 28, 2004 6:42 PM  
Anonymous Anonymous said...

Here is the code to modify to prevent the tabbing issue mentionned above (ascii value of tab is 9):

1/[snip]
if(X&&_eventKeycode!=0&&_eventKeycode!=9){
if(N>0&&v!=-1) {
V=V.substring(0,v);
}
[snip]

2/
[snip]
}else if(eventCode==9){
return false
}
[snip]

December 28, 2004 7:22 PM  
Anonymous Anonymous said...

gmail is also using XMLHttp to post the data back to the server. Infact we had developed a commercial application entirely on XMLHttp three years back. But performance was miserable. Guess its time to learn from google

December 29, 2004 5:42 AM  
Anonymous Anonymous said...

It is interesting that you think to not the compressed (or even obfuscated) JavaScript in the case of Google. I have noted there are a bunch of JavaScript 'crunching' programs out there that do white space, variable substitution, object remap, etc.. I have been using w3compiler.com mostly but it seems there are quite a number of folks getting into this game now. I bet someday we'll see some sort of JavaScript bytecode! how weird will that be!

December 31, 2004 12:26 AM  
Anonymous Anonymous said...

Thanks for this great work. Have anyone tried this version without asp? I would really prefer to run it on linuxbased Server with MySQL.

Greetings from Munich

January 04, 2005 11:00 AM  
Anonymous Anonymous said...

So, I took google's exact compressed code, changed the form action to an asp.net page (and E=my url), and made a .NET service which the asp.net page utilizes to return exactly the same thing google does. (except I renamed sendRPCDone to returnedData). Below is what my return looks like. Everything works great in IE6, NS 7.2, and FF 1.0, however, if I try to run this in NS 6.1 or 6.2, the alignment of the items in the DIV is completely messed up. Yet, if I got to Google Suggest in NS 6.1 or 6.2 their site works. Same code, apparently same return, different results? Maybe their sendRPCDone is returning something different if it is NS 6.1 or 6.2. Anyone else experience this? Any ideas?



This is what I am returning:
returnedData(frameElement, "foo", new Array("food", "food saver", "foodsaver", "foot warmer", "food processor", "foot massager", "football", "food processors", "foot massage", "foot warmers"), new Array("1184 results", "903 results", "624 results", "456 results", "418 results", "330 results", "200 results", "165 results", "161 results", "157 results"), new Array(""));

January 05, 2005 1:27 PM  
Anonymous Anonymous said...

So, never mind about my NS 6.x comment before. If the input field size is too small then the results get all messed up in the div. All I had to do was increase the input field size.

On a side note, if you spend about 16 hours reading through this code it becomes clear that there is alot of functionality google codes in, but doesn't use, or is meant for future use. For example, onload there are only 5 variables passed, but 8 in the function. sm is null so if(!sm)sm="query";w=sm; can be completely deleted and at the top var w can be set to var w="query" Further down, w is checked to equal "url". If you delete what I just recommended then this check is worthless.

Also, ufn, I think has something to do with frame names if you want to target the search into a new frame. I gather that from Tb="window.frames['"+mb+"'].location = \""+R+'"; where mb = ufn.

January 05, 2005 4:21 PM  
Anonymous Anonymous said...

This comment has been removed by a blog administrator.

January 10, 2005 1:56 PM  
Anonymous Anonymous said...

This comment has been removed by a blog administrator.

January 10, 2005 3:40 PM  
Anonymous Anonymous said...

good work. open a new door for web development.

http://www.jispot.com

January 11, 2005 2:31 PM  
Anonymous Anonymous said...

Objectgraph has another version. check it out:

http://www.objectgraph.com/dictionary/how.html

http://www.jispot.com

January 11, 2005 2:33 PM  
Anonymous Anonymous said...

Awesome job. In regards to XMLHTTP, around the year 2000, Microsoft had a product called Site Server 3.0 Commerce. One of the add-ons Microsoft created for it was something called Site Server Auction. It used XMLHTTP to post and retrieve bids to/from an auction without refreshing the page. I wrote a simple JavaScript timer that showed exactly how many seconds were left in the auction in real time.

Warning to everyone interested in creating a competitor to eBay - if the auctions won't be too popular, it will work like a charm; but if the auctions start getting heavy traffic, EVERYONE, and I mean EVERYONE will place bids in the last 5 seconds, so your database skills better be good.

January 13, 2005 7:18 PM  
Anonymous Anonymous said...

This comment has been removed by a blog administrator.

January 14, 2005 7:35 AM  
Blogger fmaguin said...

In InstallAC :

...SNIP...

if(Jb.indexOf(_enString+"|")==-1){
// We won't pass through here...
X=true;
Y=false;
Ba=false
}else{

...SNIP...

You will pass through because rl is set to "en". Not that important anyway ;)

January 18, 2005 12:29 PM  
Anonymous Anonymous said...

This comment has been removed by a blog administrator.

February 03, 2005 7:33 PM  
Anonymous Anonymous said...

This comment has been removed by a blog administrator.

February 09, 2005 8:40 AM  
Anonymous Anonymous said...

The XmlHttpRequest object is implemented in the latest beta of Opera (8), which can be downloaded now and works great for me.

February 10, 2005 10:07 AM  
Anonymous Anonymous said...

****

Try entering in:

zzz google select rocks my world

Or click here:

http://www.google.com/search?complete=1&hl=en&q=zzz+google+select+rocks+my+world&btnG=Google+Search

:-)

February 13, 2005 1:57 AM  
Anonymous Anonymous said...

Very nice decomposition and analysis. Thanks!

What's especially interesting to me is that Google's not using anything new in terms of client-side tech to make this happen. JavaScript and XMLHTTP have both been around for quite awhile. Google's just thinking in different ways about how to combine existing tools to make new and interesting toys... which is very cool. ;-)

February 23, 2005 12:35 PM  
Anonymous Anonymous said...

your analysis is as briliant as the initial code! I'm freezed !
hope theses guys at google are working on a browser, it must be spectacular too !
I dream of a 'suggested surfing' with an active map (in a frame/slide beside the page i'm visiting), a map of most popular pathes of sites followed on the net from the page I'm on, comments & stats on each hyperlink with a mouseover on it.....

February 23, 2005 4:59 PM  
Blogger Ruturaj Vartak said...

ah haaa... !!!
that was pretty neat.
I always scan websites fro XMLHTTPRequest features. good one

February 26, 2005 12:00 PM  
Anonymous Anonymous said...

Great work .... any clue on how to update a client browswer's page without refreshing on a server event..I am sure the day is not far...Let me know if you come across anything.

February 27, 2005 3:02 AM  
Anonymous Anonymous said...

J*O*H*N* said:
Why do the peeps at Google use such obscure variable and function naming scheme (i.e., Ja, Ur, etc)? It seems like a debugging nightmare.
I'd imagine it's to make the code lighter. I noticed they are using various techniques to use only the minimum characters to have syntactically correct code (no semicolons at the end of blocks etc).

March 02, 2005 11:07 AM  
Anonymous Anonymous said...

Thanks Chris. I managed to make a Census and Zipcode lookup with it.

http://www.bigkahunaburger.com/dvd/

http://www.bigkahunaburger.com/suggest2/

March 03, 2005 11:15 PM  
Anonymous Anonymous said...

Hey Chris -

Nice work! Rather above my head, but fascinating nonetheless.

Do you know whether the terms Google uses for the suggestions are based more on available web pages or frequently searched phrases?

BTW - I can't believe that some people have nothing better to do with their time than to post comments complaining about your grammar. Sad, really, that after you put in all this work on your own free time, and then are gracious enough to share it with the web community, and people have to be so meanspirited as 'scold' you for your use of apostrophes and elipses. I hope you don't take their negativity to heart! Keep posting - you've got a great site here.

March 04, 2005 4:23 PM  
Anonymous Anonymous said...

You might appreciate my "Zuggest" tool.

It uses "Ajax" concepts and works similarly to Google Suggest but searches against the Amazon Product database as you're typing.

Check it out: http://www.FrancisShanahan.com/zuggest.aspx

It's built with Javascript, Amazon Web Services, SOAP, XMLHttp, XML, C# and ASP.NET and SQL Server.

Would love to get some feedback on it.

-fs

March 04, 2005 10:20 PM  
Anonymous Anonymous said...

I've been trying to get this to work but didn't have any luck.

Can anyone show me an example which works like google suggest but doesnt reside on the official google server?

I'd love to get this working, but the
JavaScript just doesnt work for me...

please - help me! :-)

March 06, 2005 1:22 PM  
Blogger cyrano423 said...

Very nice work. I started doing this myself, but in the middle of doing so ... out of sheer curiosity ... I searched the web for the term "InstallAC" and came upon this.

I noticed that the code in which you traced, was different (they added more features). So I continued my work but used your function/variable names, as a point of reference.

Through my combing/stepping I saw that you missed a variable (I was being anal and was defining the purpose for every variable). I'm not sure if you even care, but I wanted to notify you of it since your code did indeed help me. It does not produce an error, because the search type was always a 'query'. Thus, the code was never executed. However if the search is a URL search, then the error is triggered.

Line 435 ...

427:function xb(eb){

435: if (_highlightedSuggestionIndex!=-1&&h){
436: R=valueOfCAutoComplete(_highlightedSuggestionDiv);
437: }


should be ...

435: if (_highlightedSuggestionIndex!=-1&&_highlightedSuggestionDiv){
436: R=valueOfCAutoComplete(_highlightedSuggestionDiv);
437: }

and as someone mentioned previously, you also forgot to take out the alert on line 677.

677: alert("ab: " + ab);

I didn't truly comb through your JS because of the difference between the code you used, and what I was using. So I'm not sure if there are other potentially error producing bits. However, of the new code ... I played with the "unused" features, like the URL search. By switching from 'query' mode to 'url' mode (the ability also exists in your version), things such as the setInputFieldSize (as you called it) are triggered.

Just thought I'd let you know that, cause it may potentially give you something new to play with. =)

If you'd like a completed version of my copy, just let me know.

- Prasand J.

March 10, 2005 4:50 PM  
Blogger Mayuresh Kadu said...

Good work. Kudos! Any chance of seeing something similiar on google mail scripts ?

March 21, 2005 1:17 AM  
Blogger kemiko said...

To the guy who wanted to know about www.a9.com's dynamically resizing table columns. Check out http://www.activewidgets.com/

March 21, 2005 10:04 PM  
Anonymous Anonymous said...

I am trying to get it working locally, both with Chris' and Google's ac.js and have not been able to get it to work. I downloaded Google's page and changed their script to be:

InstallAC(
document.f,
document.f.q,
document.f.btnG,
"http://www.google.com/complete/search",
"en");

Is there something else that needs to be done? Thanks for the help.

March 26, 2005 5:06 PM  
Anonymous Anonymous said...

Thanks for this analysis. I stumbled onto your page from who knows what I was researching at the time, and your shared info has encouraged me to start using javascript and DHTML heavily in new web projects, as I'm now convinced the standards are maturely implemented enough by IE and Mozilla to make it worth the effort.

Truly amazing what Google has done with these existing technologies.

-Dude in Denver

March 29, 2005 10:08 AM  
Anonymous Anonymous said...

How do you get more than one on a single page?

April 03, 2005 12:04 AM  
Anonymous Anonymous said...

Great work Chris. Someone asked about getting it to work with multiple inputs on a single page. I have a post about it for anyone who is interested in getting it more than one suggest going.

April 15, 2005 11:24 AM  
Anonymous Anonymous said...

Hi Chris, good work! I did some sleuthing myself and found out the following about the network/server side:

1. They use compression on HTTP for data transfer
2. Instead of making multiple connections, they just have one connection to Google server that is kept alive till you move out of that page

3. Their server caches results by keyword combination - so if you typed "ca" and somebody else did that within a certain amount of time, the results are picked from Cache and not from DB

April 18, 2005 12:05 AM  
Anonymous Anonymous said...

Hi,
Thanks for the dissection :)

For those newbies like me that don't really understand how this stuff works but would like to use more then one on their web page, here are my modifications to allow more then one suggest box on the web site.

<input autocomplete="off" maxLength=256 size=55 name="q1" value="" onfocus="InstallAC(document.f,document.f.q1,document.f.btnG,'search','en');">
<input autocomplete="off" maxLength=256 size=55 name="q2" value="" onfocus="InstallAC(document.f,document.f.q2,document.f.btnG,'search','en');">

So basically, just use the onfocus like suggested by someone previously

The problem with that is that the suggest box is always shown under the first input field where the focus was set... This is because the div name is static.

I modified these line:
(around line 220)
_completeDiv.id="completeDiv;
became
_completeDiv.id="completeDiv"+_inputField.name;

(around line 356)
document.getElementById("completeDiv).style.visibility="hidden"
became
document.getElementById("completeDiv"+_inputField.name).style.visibility="hidden"

(around line 360)
document.getElementById("completeDiv).style.visibility="visible";
became
document.getElementById("completeDiv"+_inputField.name).style.visibility="visible";

Basically all I did is make sure the div names were all unique. I don't really know if something else should be done or if there is another solution, but with the very little understanding I have of this, its a simple hack that seems to work fine.

Hope this helps... If it doesnt, then delete it :)

April 23, 2005 8:39 PM  
Blogger Eyal said...

If you have multiple input fields, you don't need to have multiple unique divs. 1 div is just fine. You just gotta make sure to not reinitialize it on every focus. So have a global variable called "firstFocus" or something and just do the initializations 1 time.

If anyone is interested I have a clean version of this code: http://eyalamir.blogspot.com/

April 25, 2005 5:57 PM  
Anonymous Anonymous said...

Excellent! However, in the function KeyDownHandler, you have: if(X&&(_inputField.createTextRange&&(event.srcElementa&&(bb(_inputField)==0&&lb(_inputField)==0))))

Should actually be:

if(X&&(_inputField.createTextRange&&(event.srcElement==_inputField&&(bb(_inputField)==0&&lb(_inputField)==0))))

You forgot to change the variable "a" to your renamed "_inputField".

You saved me SOOOO much time! Thanks!

May 03, 2005 1:07 PM  
Anonymous Anonymous said...

I tried for 2 days but was not able to run google suggest on my local machine. Was anyone able to do so. Please let me know. Updating the javascript was of great help.

May 11, 2005 12:39 PM  
Anonymous Anonymous said...

I noticed that google suggest does not show same number of results as google.com does:

http://www.nirendra.net/cms/google/suggest
-Nirendra Awasthi

May 11, 2005 3:00 PM  
Blogger Dave Bacher said...

There are a lot of theories on why google is as fast as it is.

Most likely, it's a straight pre-computed n-way tree.

/a/p/p/l/e/apple

This could be a pre-computed document, maintained via triggers from the database when a search term starting with "apple" is added to the database.

If you were doing this in your own application, you might do something like this:
http://localhost/complete.cgi/a/p/p/l/e/apple

That would let proxies and browsers cache the result, preventing a server round trip for completion data until the data expired.

May 16, 2005 11:01 AM  
Anonymous Anonymous said...

First let me tell you again what others have been saying, 'Its a pretty neat job'. Your worked saved me so much time. I would have been lost in the a's and b's in the javascript. I have it working for my own website..basically a customer search..which display customer and the address...all works fine...but displays blank customer name when there are duplicates...any idea why would this be happening..i been trying to find out where exactly is the code to check for the array...my gusss is the ..function Pa (localCompleteDiv,ib){

could any one help me with this ...

May 20, 2005 6:20 AM  
Blogger brip said...

Nice Article

http://brip.blogspot.com

May 25, 2005 12:58 AM  
Anonymous Anonymous said...

Googlesuggest is working from behind proxy. Thanks

October 30, 2006 3:12 AM  
Anonymous Anonymous said...

It's really a great.
Thank's for you effort.

November 04, 2006 7:48 AM  
Anonymous Anonymous said...

Thanks for the great work! I'm mostly interested in the communication part and reduced the example to a bare bones 20 lines example. Have a look if you're interested: http://doctorajax.blogspot.com/2006/11/calling-google-suggest.html

Anyone know why this is allowed and not considered a "cross domain" request? Doing this with XMLHttpRequest would not have worked without proxying google on the host where the script came from.

November 14, 2006 6:12 AM  
Anonymous Anonymous said...

This comment has been removed by a blog administrator.

November 20, 2006 3:13 PM  
Anonymous Anonymous said...

Great Work. Now the next frontier is to crack the backend. This is certainly not a Regex Query. Let me drop a Hint. Heard of Suffix Trees!!

November 29, 2006 4:02 PM  
Anonymous Skyrunner said...

Great work! Thx~

April 24, 2009 2:09 AM  
Anonymous control valves said...

I believe construction of such projects requires knowledge of engineering and management principles and business procedures, economics, and human behavior.

May 28, 2009 11:56 PM  
Anonymous SmartGuy said...

Thanks for sharing these info. Those are very useful.

June 18, 2009 5:47 AM  

Post a Comment

<< Home