Most code search engines index only the identifiers, comments and such words and hence fail to return impressive results. Stanford researchers show that instead of indexing keywords, indexing subtrees lead to an effective search.
AST’s of production grade software systems end up being too large. They claim that the average number of AST nodes, even in simple university programming assignments is close to 150! While the size is scary, the structure fits the purpose. AST construction tools are easily available. That becomes an added advantage.
Grammar drives programming language syntax. Parsers leverage grammar to detect errors, extract parts of source code, do type checking, etc. Since typical programming language grammars are context-free, their derivations are inherently trees. Thus, abstract syntax trees came into existence.
While the original purpose of AST was not much beyond parsing and direct applications of parsing, researchers inspect how semantic analyses can be built over ASTs. Interesting ideas evolved such as replacing subtrees for program repair, duplicate subtrees for clone detection, and semantics preserving AST transformation for synthesizing source code.
Code search has been lagging behind on these lines to leverage the richness of AST information. The gory details of source code can now be abstracted out and meaningful subtrees can be indexed. Hope these ideas make it to production-quality webscale search engines soon.