Viss kas jadara, akal jaiet google un jameklē ->
"robots.txt" "disallow:" filetype:txt
sajos linkos ko atrada, tiek apkopotas vietas ko google nemeklē,
bet tās pats var mainīt,piemēram, meklējot
"robots.txt" "disallow:" filetype:txt
Viens no atrastajiem linkiem bija -> http://www.whitehouse.gov/robots.txt
Un šeit ir apkopotas vietas kuras google nemeklē...
Piemēram :
User-agent: *
Disallow: /cgi-bin
Disallow: /search
Disallow: /query.html
Disallow: /help
Disallow: /360pics/text
Disallow: /911/911day/text
Disallow: /911/heroes/text
Disallow: /911/messages/text
Disallow: /911/patriotism/text
Disallow: /911/patriotism2/text
Disallow: /911/progress/text
Disallow: /911/remembrance/text
Disallow: /911/response/text
Disallow: /911/sept112002/text
Disallow: /911/text
Disallow: /ConferenceAmericas/text
Disallow: /GOVERNMENT/text
Disallow: /QA-test/text
Disallow: /aci/text
Disallow: /afac/text
Disallow: /africanamerican/text
Disallow: /africanamericanhistory/text
Disallow: /agencycontact/text
Disallow: /americancompetitiveness/text
Disallow: /apec/2003/text
Disallow: /apec/2004-summit/text
Disallow: /apec/2004/text
Disallow: /apec/2005/text
Disallow: /apec/2006/photoessay/text
Disal
...
Lasīt Vairāk »
