Omni-Opticon, the concept -- trends versus flukes

• BabyNameWizard.com
• The extinction of Mildred and the Miley phenomenon --
• Baby names starting with vowels

Lexicography

• Not AI, but fast and feasible
• Word frequency analysis (each document is a vector over the space of  |V| dimensions where V is the vocabulary of the language(s) used.
• Concordance (key word in context)
• Cooccurence -- University of Texas 1978.
• Resources available
• Huge word list -- British National Corpus
• Online Plain Text English Dictionary and Hyper-Hyphen (2002)
• Thesaurus:  Moby Project at U Sheffield and Semantic Soup (2002) (queried once a minute by Googlebots since mid 2010)
• Census data (US last names and first names, US place names, world name data)
• Internet itself

Word relations 

• intra-document cooccurrence -- "classical proximity research"
• census of cooccurrence  (topic relations via search query results)
  

  Google “results” Oct.2010 (x1000)
  Co/occurrence	svg	dojo	html	ajax	jquery	javascript	camel
  svg	42800	139	12300	752	761	1170	210
  dojo	138	7090	1420	792	626	880	42
  html	12900	1450	5310000	132000	8360	132000	4630
  ajax	747	792	39300	73300	5080	1780	264
  jquery	758	632	8330	5090	15000	11200	40
  javascript	1170	892	123000	2070	7970	357000	692
  camel	210	48.5	4560	258	40.5	684	21600

  Proximity matrix (using a simple average of the 4 relevant ratios related to two search terms)

  	svg	dojo	html	ajax	jquery	javascript	camel
  svg		0.0455	0.5935	0.05547	0.13676	0.061228	0.029258
  dojo			0.4053	0.24502	0.2613	0.254893	0.016954
  html				2.36923	1.11581	0.762308	0.427194
  ajax					0.81674	0.913482	1.015589
  jquery						1.331697	0.009094
  javascript							0.067558
  camel

  
  * Need for an "objective" web-crawler providing quantitiative data of this sort for web researchers. Meta-mining of this "expert" data is worth $G.
  
• string similarity (and graph similarity) [REF 2]
• take union of set of all substructures of an algebraic structure and let those be the axes of a vector space; each structure is measured by the number of substructures of each type it contains, and is hence, is a point in that vector space.   A Minkowski distance in that space determines a distance between any two structures; over set of all structures within a theory, we have a metric space for that theory -- works for graphs, strings, groups, metric spaces. Accordingly the set of all theories is a metric space with each theory being a point in it.
• large strings: good; 
• small strings: false positives

  gun	be gun
  anise	reorg anise
  herb	mot herb oard
  barium	her barium

• semantic similarity -- reductionism can be hard [REF3]
• dictionary (helpful for connection of subsubstrings : grow-ing versus gro-wing)
• thesaurus -- 
• Modern ones are proprietary (Sonny Bono Dark Ages),
•  public domain: Roget's problems 
  

  $ f=moby/mthes/thesaurus ./magic [^[:alpha:]]ground[^[:alpha:]] $f|sort|uniq|wc -l 12016

  ./magic [^[:alpha:]]ground[^[:alpha:]] $f|sort|uniq|tail zincograph zing zip zipper zodiac zombie zone zoo zoom zygote

  
  
  
• A solution: Semantic soup and quantifiying closeness (another temporary link)
  
  compare positions of k words in k-dimensional space -- how high is "ground" on the "flat" and "flight" dimensions?:

• ./magic [^[:alpha:]]ground[^[:alpha:]] $f|sort|uniq -c|head -4325|tail -20       2 flank       7 flap       3 flash      39 flat      23 flatten       2 flaunt       9 flavor       4 flaw       2 flax       1 flay      12 fleck       1 flection       6 fleece       5 fleet       6 flesh       1 fleshy       1 flexibility       1 flexure       6 flick       7 flight	./Webmagic ground 20 Content-type: text/html     381 ground     125 floor     104 place      96 terrain      82 seat      82 basis      80 set      79 field      74 level      71 bed      70 stage      70 milieu      69 arena      67 theater      66 base      65 territory      65 fix      64 canvas      63 principle      62 range      61 course
  Cooccurence frequencies of "ground" and selected "synomyms"	Sorted by frequency.

Earlier work