Dylan Hunter

Was an excellent post on O’Reilly Radar by Nat Torkington reminding us in the course of designing social suggestion-based sites / software to not forget one of the most critical reasons for in the first place - serendipity / new discovery. Sometimes this is best accomplished by “more like this,”‘es and/or “people who viewed this purchased..” kinds of things, (i.e. narrowing / refinement goal-oriented) and sometimes best accomplished by purposely broadening / adjusting the scope to introduce some new variance and seeing where things progress from there.

Many years ago there was a little search startup called Direct Hit here in MA, (who earned $500M on a cheaper version of what I wanted to do, but that’s another story) who really was the first entity to hang their hat on social suggestion-based search results, knowing full well that most folks only look at results 1-3, and almost no one beyond 10, so that those results that it initially presented as 1-3 would almost always tend to stay there, whether they were the best results or not - i.e. for anyone who’s ever used Excel, an infinite circular error.

As we all took this kind of click-through analysis / adjustment functions in to become one of the ranking criteria we used, (and / or at least used to internally evaluate how good a job we were doing in delivering relevant results - with all of the appropriate mechanisms for negating bounce - i.e. clicking through on a link, saying “this isn’t what I wanted” and hopping back to the search page, etc) the import of regularly introducing different results into the mix to make sure we were actually doing a much better job of delighting folks became more and more clear then, and still holds true today.

But enough of me babbling, Nat does a superb job of describing, so go read it from him!

I actually worked on something similar a bunch of years ago, at the time more oriented toward personalization of search results, but, if anything, the threshold of how good you have to be on to make it worthwhile should definitely be lower in an ad context, so should be a good feature when they launch, (presuming they actually do it well, which, unfortunately, MSN Search does not raise great faith in - perhaps their folks in Beijing will be better at! ).

Is hardly iron-hard, but you definitely can predict several demographic components by how people express the same / similar ideas, at least good enough to try using as ad segmentation technique, and then as long as you can keep pushing the acquisition needle further over, you keep investing!

“Microsoft Corp may soon be able to tell whether an Internet search query comes from a man or a woman.

“…’There’s a confidence interval around one’s gender,’ [Jed] Nahum said in a telephone interview. ‘Advertisers can start to tailor their message based on those estimates. Using the new technology, Microsoft will be able to tell, for example, that someone searching for the term `Dodge Caravan’ is more likely to be female than someone searching for `Dodge’ alone.’”

More (warning: coming from Taipei, so it’ll take a sec to load)

An amusing mini-romp through some of those legal jewels embedded deep within the standard EULA. I liked Claria’s - to paraphrase: “I’m a scumbag, and am going to send back all kinds of nasty stuff, and you have no right to know what kind of stuff I’m going to send back, nor do you have the right to do anything about it…”

Funny stuff, especially for an app that no one, but no one, actually would _choose_ to install on their own, unbundled, un-”quietly mixed into the fine print of an installer,” un-ActiveX installed, etc., etc.

Too bad, too, from what I know, they’ve developed some pretty neat user profiling technology that could be used to make lots of folks very happy if applied to Search, eComm, etc., (i.e. similar technology is used to determine which result an SE should actually display to you, as opposed to your neighbor, to try to figure out what you mean when you pop in those 2 nigh-unto random words) - instead, it’s used to piss people off.

A search engine based on was the topic of my big Business Plan while at Babson, and still have not just great faith, but believe in the import of, as Search continues to mature - i.e. there’s not nearly enough being done to modify non-ad-related content to pay attention to what _I_ actually want - and yes, Local does get a bit further toward this, so it’ll be what I, and several million other people in MA might find relevant, but, guess what, for many items, I’m going to be a lot closer to folks in NJ, where I grew up, on others, closer to folks in NH or ME, where I vacation, on others closer to folks in Alaska, where someday I still hope to go, or folks who travel to the Himalayas, etc., etc.

I know _very_ well how tough it is to make these kinds of associations, as well as to update, determine when to re-assess which component groups we belong in, etc., but that doesn’t mean that it’s not an extremely important problem to work on, not just for the business opportunity, (though there is plenty of that) but for the good of us all. And yes, we’ll have to work through / around some the FUD that’s been inspired by the Scumbags, but we’ll get there - especially in societies that pride themselves on the value of the individual, to have Search reaching for anything less will always leave the prize unclaimed, and the opportunity to be surpassed, self-evident.

Ok, so what’s actually managed to get me out of a 10-month hiatus to actually put up another blog post? Really, podcasting, (which I’ve been falling in love with over the last 1.5-2 mths) but that’s fodder for another post, (or 2 or 12). But at this precise moment, it’s an article in the Channel Register that’s talking about Yahoo re-assembling some of the old IBM Clever guys again, with speculation toward reviving at least some portion of IBM’s old Clever project.

Why’s this so inspiring? Several reasons. I first actually read about Clever in Scientific American back in ‘98 or so. At the time, even though officially I was (and am) a business guy, I was (and am) a total tech geek, and the beauty of the system that they described was so amazingly apparent in comparison to what was out there at the time, (yes, even our friend Google, who I remember thinking was a “cheap copy” of Clever at the time - ok, “more pragmatic” copy would probably be more appropriate - Clever could take 11 mins at query time, which was, of course, ludicrous, but from deep geekdom, it was like seeing the Mona Lisa and then thinking about a poster of some other girl - the latter was much more practical, but nowhere near as breathtaking - see, told you - was and will always be, first and foremost a math geek, so I very much to this day see beauty in elegance rather than bludgeon) that I literally fell as close to in love with a search technology as one could.

Actually, not too long before the time that that article came out, I had been working on a business plan for a new kind of search engine for the business plan competition as part of my evening MBA with Babson, (back then, winning Babson’s business plan contest was an excellent way of getting real funding and being able to actually make a real company, so we all worked mighty hard on pulling it off). In essence, it was a better version of Direct Hit, before Direct Hit launched, using user clicks as the prime mechanism to improve relevance sorts, but with the addition of a strong user profiling component, so that it only would use the clicks of folks “like you” to define your sort for a given query, and of those people like someone else when they issued the same query.

Knew the marketing side of the business would be much easier, (to Direct Hit’s absolute credit, to this day there is no company’s tagline that I love better than what theirs was - “One Site, Millions of Minds” - perfection - simple, totally got the concept across - wondrous! ) so spent 6 months figuring out the tech side, i.e.:

- How much could you know about someone the first time they came to you - what things would be the best to use and to what degree - IP resolution, (removing all entries from Vienna, VA, of course; all the AOL folks) associating the person with the user group for which his search term was most populous, etc., etc.

- How, and to what level, could you refine the user’s profile given his query and click patterns, frequencies, etc., other data that you might be able to infer from offline sources, based on the data you could pull, (most notably, via IP, allowing for the error that the IP is actually where your provider connects, not you, which can radically affect the usefulness of this variable).

- How would you know when the user’s behavior suggested a subclass of whatever class you had him currently a part of vs. when you should migrate him to another segment tree, etc., etc., (hey, I told you I was a geek at heart - the business side is how I try to make things actually come about! ).

Anyway, enough of the “Dylan’s Direct Hit before Direct Hit” story for now, (though I still find it interesting that I presented the concept in Wellesley, MA, and shortly thereafter a new company was founded in Wellesley, MA, but that’s something else…. I know, I know, and Kennedy was really killed by a hellish combination of Girl Scouts and escapees from Area 51… ).

The second part of my interest (very short version) is that I was PM for Internet Search for Fast Search & Transfer from nigh unto the beginning in ‘99, (left in ‘01) which was bought out by Overture in ‘02, (I think) and thence by Yahoo, (4 mths thereafter) so if Yahoo is indeed working on bringing at least portions of Clever out of mothballs, and reunifying it with my Old Girl, that makes me happy on so many levels, (we actually did do a deal with IBM, and licensing some of Clever was discussed, but IBM decided they were going to figure out how to use by themselves, or nobody was). I truly miss the old discussions we used to have as to what defines relevance, to whom, what technologies / areas of R&D might improve, etc., etc. Awesome fun stuff to work on, and am working right now on figuring out how best to apply, (with no cash, of course - I have little-to-none, nor do I see a whole slew of VC’s dying to give me some! ).

And this time, when I want to do PFI or text P4P, (as I did in ‘00) or, much more importantly, the next thought on monetization, (since back then, I was in a good place to pull off - now, I’d have to be an idiot to go head-to-head vs. Google & Yahoo, but am sure there are plenty of holes around them) I won’t have to worry about speculation from others that doing so would isolate our B2B clients, (even after Google had done the same thing to Yahoo) even given that they had already pushed me from $4.00 per thousand queries down to $1.75 16 months later, (soon fully inverted to “I don’t pay you - you pay me $5.00 per thousand”).

$100M for the technology when Overture bought my Old Girl out, $1.5Bil when Yahoo bought Overture plus my technology 4 mths later, $23Bil+ Google IPO…. oh man, I could have been…. Stop… Not productive, except in once again reminding me that speculation belongs only as a way of educating tests in the actual open market, but it is the market, and not random speculation, that should determine what a company tries to do. And yes, I had Alltheweb.com, which we’d grown to 5.5M queries per day at that point, half of a Lycos, (when Lycos was actually impressive) so rather than costing me $1M a year in infrastructure costs to support, could have been…. Stop…. Again, would’ve, could’ve, should’ve - who cares - it’s done.

Ignore the still-sore spot, (obviously! ) read the Clever article. Hope that the guys can figure out how to use some of the better concepts from Clever such that it can respond in less than 11 mins, (i.e. beauty is certainly beauty, but it’s still better to actually make something a reality along a path and get better from there - if something only exists in a lab, it doesn’t exist) unify that with my Old Girl, (now also combined with AV & Inktomi) and bring something extraordinary about.

For me, gives me great hope - not just for Yahoo, (though, as you can tell, I still would love to see my Old Girl glow again - it’s been a bit, certainly now @ ATW.com, she’s barely even a shadow) but more importantly, in adding to the assertion that there indeed is still life in doing Search well. I have no cash, but I don’t care - at this moment, I am more pleased than I’ve been in a long time!

Now to figure out the hardest part of the solution that I’ve been trying to figure out for months, if not years - how the heck to work on, after work, while still putting the right amount of time into my wife and daughters, who are much more critical - if you find yourself lucky enough to be married to someone who was with you when you were (at least theoretically) worth a small fortune, and she stays with you when you’re once again worth nothing, without a moment of hesitation or recrimination, (other than the “don’t even talk to me about equity again!” ) hold onto her for dear life - better than one in a million. And if you further are greeted every day when you come home, regardless of whatever happened, with smiles, loving hugs and excited squeals of “Daddy!!!” you’d be a flaming idiot, (and yes, this is preemptive for an elder me, just in case) to endanger that - for any reason - ever.