If a spider (of the Web-crawling variety) encountered the RDF snippet I generated from the Jenabean project URL namespace, it wouldn't be able to make anything of it.

如果一个spider (Web爬行变体)遇到我从jenabean项目URL名称空间中生成的rdf片段，则无法利用这个片段。

youdao

This example demonstrates the crawling phase of a Web spider.

这个例子展示了Web spider爬行的阶段。

youdao

But all the major and legitimate Web crawling engines obey the requests in robots.txt.

不过，所有主要的合法Web爬虫引擎都会遵从robots . txt内的要求。

youdao

It will be a free site that uses algorithms to build profiles based on users' preexisting Web-crawling information.

FORBES: Social Slingers

Webaroo's computer scientists in Mumbai and New Delhi cranked out dozens of Web-crawling algorithms that decipher the necessary and truly informative stuff.

FORBES: Narrow Your Horizons

The search engine's Web-crawling spider now keeps an eye out for viruses and malware as it combs the Web, and shows users a warning page before allowing them to access malicious sites.

FORBES: Made For Hacking