- assisted text scanning – Create databases that scan text patterns and automatically generate data summaries. Have a human verify the results.
- development strategy – Isolate tiny, near insignificant examples of common discourse, and automatically detect them in web pages as they are traversed.
- promotion study example – There are only so many ways to say, in English, that so-and-so was promoted to the position of such-and-such. Make a program that searches for lines in the present web page for anything that matches a very loose pattern. Unobtrusively automatically suggest an understanding, whenever something is found that might be it.
- google search – I did a google search for: “promoted to the position of”, and found a bunch of sensible stuff.
- sample result – “Scott Garson, who for the last two seasons has been UCLA men’s basketball video coordinator and before coming to Westwood served five seasons on Rick Majerus’staff at Utah, has been promoted to the position of Bruin assistant basketball coach, UCLA head coach Ben Howland announced Monday.” [1]
- identify likely name – “Scott Garson” “Westwood” “Rick Majerus”
- identify likely position – “Bruin assistant basketball coach” (after the “of,” before the punctuation.)
- identify likely dates – …and inferences the likely meanings of days like “Monday” or “last Saturday”
- suggest an interpretation – “SCOTT GARSON” was promoted to “BRUIN ASSISTANT BASKETBALL COACH” on “YYYY-MM-DD.”
- user corrects – The user can then correct the interpretation, by clicking on the other possibilities, or just manually editing it – “no, it was Rick Majerus,” – “no, it was on X date,” – “no, the date is unknown” – “make that just assistant basketball coach, - we will omit Bruin”
- ContentRouting for text snippets – The next thing to do, is to make it easy to collect the interpreted data in a central database, so that many people can aggregate what they have picked up from the web. Record the source article (URL,) record the source of each inference, and record the corrections applied.
- correct / massage interpreter rules – As data comes in, identify common patterns in input. Make the system smarter at interpreting English. Apply automated tests to existing samples to make sure that it doesn’t break much of what’s worked automatically in the past. Automatically distribute new heuristics to the browser plugins, to make it easier on people entering the data.
- database assist – As the database of knowledge accumulates, start making use of it to perform better analysis.
- promotion study example – Check to see if anybody else has already logged the particular promotion. If so, ask the user: “Is this the same person P as in X? Is this the exact same promotion, or a different promotion?” If it’s the exact same promotion, you don’t need to discard all the data: Rather, aggregate it together. You now have two URLs talking about the same promotion.
If this procedure works for one small niche (say, “promotions,”) then you should be able to apply it to other niches.
If you get a sufficient number of niches, you might even be able to start playing them with one another: Reinforce the interpretation of one text with data that’s outside of it’s particular field.
I was originally thinking about this in the context of the Futures wiki: I’d like to be able to select lines of text that have to do with predictions for the future, and then send that line, along with the URL attribution, to the Futures wiki / DataBall. That way, we can collectively build a timeline for the future, and keep all the attributions in place and so on. The problems are largely ContentRouting: It just takes way too long to perform the manual labor involved.
See Also
rough notes text analysis automatic automated inference
- PlainLink – automatically noting similarities between similar texts, and then linking them to one another
- RobotWisdom – “People First”
- SemanticWeb – the web of data