crawlerbase.de / trace Bot-Datenbank
www.google.com 2 Hops/149 ms/200 HTTP/2

Übersicht

final URL: https://www.google.com/
2
Hops
149 ms
Total
60
Bots erlaubt
0
Bots blockiert

Redirect-Chain

2 Hops
  1. 00
    301 HTTP/2 https://google.com
    TTFB 58 ms Total 58 ms Size 220 B IP 142.251.14.139 Server gws
    https://www.google.com/
    TLS Issuer C = US, O = Google Trust Services, CN = WE2 Gültig noch 64 Tage
    Alle Header (11)
    locationhttps://www.google.com/
    content-typetext/html; charset=UTF-8
    content-security-policy-report-onlyobject-src 'none';base-uri 'self';script-src 'nonce-q5r5CgxRB7IJr9TvKYWDPg' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
    dateSun, 10 May 2026 00:57:39 GMT
    expiresTue, 09 Jun 2026 00:57:39 GMT
    cache-controlpublic, max-age=2592000
    servergws
    content-length220
    x-xss-protection0
    x-frame-optionsSAMEORIGIN
    alt-svch3=":443"; ma=2592000,h3-29=":443"; ma=2592000
  2. 01
    200 HTTP/2 https://www.google.com/
    TTFB 81 ms Total 91 ms Size 26,2 KB IP 142.251.150.119 Server gws
    TLS Issuer C = US, O = Google Trust Services, CN = WE2 Gültig noch 64 Tage
    Alle Header (14)
    content-typetext/html; charset=ISO-8859-1
    dateSun, 10 May 2026 00:57:39 GMT
    expires-1
    cache-controlprivate, max-age=0
    content-security-policy-report-onlyobject-src 'none';base-uri 'self';script-src 'nonce-0PlC6nmUW0Rsg329CNTGNA' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
    accept-chSec-CH-Prefers-Color-Scheme
    p3pCP="This is not a P3P policy! See g.co/p3phelp for more info."
    content-encodinggzip
    servergws
    content-length26845
    x-xss-protection0
    x-frame-optionsSAMEORIGIN
    alt-svch3=":443"; ma=2592000,h3-29=":443"; ma=2592000

robots.txt / google.com

vorhanden 4 Gruppen 1 Sitemap
robots.txt Quelltext (6,4 KB)
google.com/robots.txt
User-agent: * User-agent: Yandex Disallow: /search Allow: /search/about Allow: /search/howsearchworks Disallow: /sdch Disallow: /groups Disallow: /index.html? Disallow: /? Allow: /?hl= Disallow: /?hl=*& Allow: /?hl=*&gws_rd=ssl$ Disallow: /?hl=*&*&gws_rd=ssl Allow: /?gws_rd=ssl$ Allow: /?pt1=true$ Disallow: /imgres Disallow: /u/ Disallow: /setprefs Disallow: /m? Disallow: /m/ Allow: /m/finance Disallow: /wml? Disallow: /wml/? Disallow: /wml/search? Disallow: /xhtml? Disallow: /xhtml/? Disallow: /xhtml/search? Disallow: /xml? Disallow: /imode? Disallow: /imode/? Disallow: /imode/search? Disallow: /jsky? Disallow: /jsky/? Disallow: /jsky/search? Disallow: /pda? Disallow: /pda/? Disallow: /pda/search? Disallow: /local? Disallow: /local_url Disallow: /products? Disallow: /product_ Disallow: /products_ Disallow: /products; Disallow: /print Disallow: /books/ Disallow: /bkshp?*dq= Disallow: /bkshp?*q= Disallow: /books?*dq= Disallow: /books?*q= Disallow: /books?*qtid= Disallow: /books?*output= Disallow: /books?*pg= Disallow: /books?*jtp= Disallow: /books?*jscmd= Disallow: /books?*buy= Disallow: /books?*zoom= Allow: /books/about Allow: /books?*zoom=1 Allow: /books?*zoom=5 Allow: /books/content?*zoom=1 Allow: /books/content?*zoom=5 Disallow: /patents? Disallow: /patents/download/ Disallow: /patents/pdf/ Disallow: /patents/related/ Disallow: /scholar Disallow: /citations? Allow: /citations?user= Allow: /citations?view_op=new_profile Allow: /citations?view_op=top_venues Allow: /scholar_share Disallow: /s? Disallow: /maps? Allow: /maps?daddr= Allow: /maps?entry=wc Allow: /maps?f= Allow: /maps?hl= Allow: /maps?q= Allow: /maps?saddr= Allow: /maps?sid= Allow: /maps?*output=classic Allow: /maps?*file= Disallow: /mapslt? Disallow: /maphp? Disallow: /maps/ Allow: /maps/$ Allow: /maps/@ Allow: /maps/?daddr= Allow: /maps/?entry=wc Allow: /maps/?f= Allow: /maps/?hl= Allow: /maps/?q= Allow: /maps/?saddr= Allow: /maps/?sid= Allow: /maps/search/ Allow: /maps/sitemap.xml Allow: /maps/sitemaps/ Allow: /maps/dir/ Allow: /maps/d/ Allow: /maps/reserve Allow: /maps/about Allow: /maps/contrib/ Allow: /maps/match Allow: /maps/ms? Allow: /maps/place/ Allow: /maps/_/ Allow: /search?*tbm=map Allow: /maps/vt? Allow: /maps/preview Disallow: /maps/api/js/ Allow: /maps/api/js Disallow: /mld? Disallow: /staticmap? Disallow: /help/maps/streetview/partners/welcome/ Disallow: /help/maps/indoormaps/partners/ Disallow: /lochp? Disallow: /ie? Disallow: /uds/ Disallow: /transit? Disallow: /trends? Disallow: /trends/music? Disallow: /trends/hottrends? Disallow: /trends/viz? Disallow: /trends/embed.js? Disallow: /trends/fetchComponent? Disallow: /trends/beta Disallow: /trends/topics Disallow: /trends/explore? Disallow: /trends/api Disallow: /musica Disallow: /musicl Disallow: /musics Disallow: /urchin_test/ Disallow: /movies? Disallow: /wapsearch? Disallow: /reviews/search? Disallow: /cbk Disallow: /profiles/me Disallow: /s2/profiles/me Allow: /s2/profiles Allow: /s2/oz Allow: /s2/photos Allow: /s2/search/social Allow: /s2/static Disallow: /s2 Disallow: /transconsole/portal/ Disallow: /aclk Disallow: /tbproxy/ Disallow: /support/forum/search? Disallow: /reviews/polls/ Disallow: /hosted/images/ Disallow: /accounts/ClientLogin Disallow: /accounts/ClientAuth Disallow: /accounts/o8 Allow: /accounts/o8/id Disallow: /quality_form? Disallow: /labs/popgadget/search Disallow: /compressiontest/ Disallow: /analytics/feeds/ Disallow: /analytics/partners/comments/ Disallow: /analytics/portal/ Disallow: /analytics/uploads/ Allow: /alerts/manage Allow: /alerts/remove Disallow: /alerts/ Allow: /alerts/$ Disallow: /phone/compare/? Disallow: /travel/clk Disallow: /travel/entity Disallow: /travel/search Disallow: /travel/flights/booking Disallow: /travel/flights/s/ Disallow: /travel/flights/search Disallow: /travel/hotels/stories Disallow: /travel/hotels/*/stories Disallow: /travel/story Disallow: /hotelfinder/rpc Disallow: /hotels/rpc Disallow: /evaluation/ Disallow: /forms/perks/ Disallow: /shopping/suppliers/search Disallow: /edu/cs4hs/ Disallow: /trustedstores/s/ Disallow: /trustedstores/tm2 Disallow: /trustedstores/verify Disallow: /shopping? Disallow: /shopping/product/ Disallow: /shopping/seller Disallow: /shopping/ratings/account/metrics Disallow: /shopping/ratings/merchant/immersivedetails Disallow: /shopping/reviewer Disallow: /shopping/search Disallow: /shopping/deals Allow: /shopping?udm=28$ Disallow: /storefront Disallow: /storepicker Disallow: /about/careers/applications/candidate-prep Disallow: /about/careers/applications/connect-with-a-googler Disallow: /about/careers/applications/jobs/results?page= Disallow: /about/careers/applications/jobs/results/?page= Disallow: /about/careers/applications/jobs/results?*&page= Disallow: /about/careers/applications/jobs/results/?*&page= Disallow: /landing/signout.html Disallow: /gallery/ Disallow: /landing/now/ontap/ Allow: /maps/reserve Allow: /maps/reserve/partners Disallow: /maps/reserve/api/ Disallow: /maps/reserve/search Disallow: /maps/reserve/bookings Disallow: /maps/reserve/settings Disallow: /maps/reserve/manage Disallow: /maps/reserve/payment Disallow: /maps/reserve/receipt Disallow: /maps/reserve/sellersignup Disallow: /maps/reserve/feedback Disallow: /maps/reserve/terms Disallow: /maps/reserve/m/ Disallow: /maps/reserve/b/ Disallow: /maps/reserve/partner-dashboard Disallow: /local/cars Disallow: /local/dealership/ Disallow: /local/dining/ Disallow: /local/place/products/ Disallow: /local/place/reviews/ Disallow: /local/place/rap/ Disallow: /local/tab/ Disallow: /localservices/ Disallow: /nonprofits/account/ Disallow: /uviewer Disallow: /landing/cmsnext-root/ # AdsBot User-agent: AdsBot-Google Disallow: /maps/api/js/ Allow: /maps/api/js Disallow: /maps/api/place/js/ Disallow: /maps/api/staticmap Disallow: /maps/api/streetview # New user agent groups must also have a user agent reference in the global (*) # group. See "Order of precedence" section in # https://goo.gle/rep#order-of-precedence-for-user-agents User-agent: Yandex Disallow: /about/careers/applications/jobs/results Disallow: /about/careers/applications-a/jobs/results # Crawlers of certain social media sites are allowed to access page markup when # google.com/imgres* links are shared. To learn more, please contact # images-robots-allowlist@google.com. User-agent: facebookexternalhit User-agent: Twitterbot Allow: /imgres Allow: /search Disallow: /groups Disallow: /hosted/images/ Disallow: /m/ Sitemap: https://www.google.com/sitemap.xml

robots.txt / www.google.com

vorhanden 4 Gruppen 1 Sitemap
robots.txt Quelltext (6,4 KB)
www.google.com/robots.txt
User-agent: * User-agent: Yandex Disallow: /search Allow: /search/about Allow: /search/howsearchworks Disallow: /sdch Disallow: /groups Disallow: /index.html? Disallow: /? Allow: /?hl= Disallow: /?hl=*& Allow: /?hl=*&gws_rd=ssl$ Disallow: /?hl=*&*&gws_rd=ssl Allow: /?gws_rd=ssl$ Allow: /?pt1=true$ Disallow: /imgres Disallow: /u/ Disallow: /setprefs Disallow: /m? Disallow: /m/ Allow: /m/finance Disallow: /wml? Disallow: /wml/? Disallow: /wml/search? Disallow: /xhtml? Disallow: /xhtml/? Disallow: /xhtml/search? Disallow: /xml? Disallow: /imode? Disallow: /imode/? Disallow: /imode/search? Disallow: /jsky? Disallow: /jsky/? Disallow: /jsky/search? Disallow: /pda? Disallow: /pda/? Disallow: /pda/search? Disallow: /local? Disallow: /local_url Disallow: /products? Disallow: /product_ Disallow: /products_ Disallow: /products; Disallow: /print Disallow: /books/ Disallow: /bkshp?*dq= Disallow: /bkshp?*q= Disallow: /books?*dq= Disallow: /books?*q= Disallow: /books?*qtid= Disallow: /books?*output= Disallow: /books?*pg= Disallow: /books?*jtp= Disallow: /books?*jscmd= Disallow: /books?*buy= Disallow: /books?*zoom= Allow: /books/about Allow: /books?*zoom=1 Allow: /books?*zoom=5 Allow: /books/content?*zoom=1 Allow: /books/content?*zoom=5 Disallow: /patents? Disallow: /patents/download/ Disallow: /patents/pdf/ Disallow: /patents/related/ Disallow: /scholar Disallow: /citations? Allow: /citations?user= Allow: /citations?view_op=new_profile Allow: /citations?view_op=top_venues Allow: /scholar_share Disallow: /s? Disallow: /maps? Allow: /maps?daddr= Allow: /maps?entry=wc Allow: /maps?f= Allow: /maps?hl= Allow: /maps?q= Allow: /maps?saddr= Allow: /maps?sid= Allow: /maps?*output=classic Allow: /maps?*file= Disallow: /mapslt? Disallow: /maphp? Disallow: /maps/ Allow: /maps/$ Allow: /maps/@ Allow: /maps/?daddr= Allow: /maps/?entry=wc Allow: /maps/?f= Allow: /maps/?hl= Allow: /maps/?q= Allow: /maps/?saddr= Allow: /maps/?sid= Allow: /maps/search/ Allow: /maps/sitemap.xml Allow: /maps/sitemaps/ Allow: /maps/dir/ Allow: /maps/d/ Allow: /maps/reserve Allow: /maps/about Allow: /maps/contrib/ Allow: /maps/match Allow: /maps/ms? Allow: /maps/place/ Allow: /maps/_/ Allow: /search?*tbm=map Allow: /maps/vt? Allow: /maps/preview Disallow: /maps/api/js/ Allow: /maps/api/js Disallow: /mld? Disallow: /staticmap? Disallow: /help/maps/streetview/partners/welcome/ Disallow: /help/maps/indoormaps/partners/ Disallow: /lochp? Disallow: /ie? Disallow: /uds/ Disallow: /transit? Disallow: /trends? Disallow: /trends/music? Disallow: /trends/hottrends? Disallow: /trends/viz? Disallow: /trends/embed.js? Disallow: /trends/fetchComponent? Disallow: /trends/beta Disallow: /trends/topics Disallow: /trends/explore? Disallow: /trends/api Disallow: /musica Disallow: /musicl Disallow: /musics Disallow: /urchin_test/ Disallow: /movies? Disallow: /wapsearch? Disallow: /reviews/search? Disallow: /cbk Disallow: /profiles/me Disallow: /s2/profiles/me Allow: /s2/profiles Allow: /s2/oz Allow: /s2/photos Allow: /s2/search/social Allow: /s2/static Disallow: /s2 Disallow: /transconsole/portal/ Disallow: /aclk Disallow: /tbproxy/ Disallow: /support/forum/search? Disallow: /reviews/polls/ Disallow: /hosted/images/ Disallow: /accounts/ClientLogin Disallow: /accounts/ClientAuth Disallow: /accounts/o8 Allow: /accounts/o8/id Disallow: /quality_form? Disallow: /labs/popgadget/search Disallow: /compressiontest/ Disallow: /analytics/feeds/ Disallow: /analytics/partners/comments/ Disallow: /analytics/portal/ Disallow: /analytics/uploads/ Allow: /alerts/manage Allow: /alerts/remove Disallow: /alerts/ Allow: /alerts/$ Disallow: /phone/compare/? Disallow: /travel/clk Disallow: /travel/entity Disallow: /travel/search Disallow: /travel/flights/booking Disallow: /travel/flights/s/ Disallow: /travel/flights/search Disallow: /travel/hotels/stories Disallow: /travel/hotels/*/stories Disallow: /travel/story Disallow: /hotelfinder/rpc Disallow: /hotels/rpc Disallow: /evaluation/ Disallow: /forms/perks/ Disallow: /shopping/suppliers/search Disallow: /edu/cs4hs/ Disallow: /trustedstores/s/ Disallow: /trustedstores/tm2 Disallow: /trustedstores/verify Disallow: /shopping? Disallow: /shopping/product/ Disallow: /shopping/seller Disallow: /shopping/ratings/account/metrics Disallow: /shopping/ratings/merchant/immersivedetails Disallow: /shopping/reviewer Disallow: /shopping/search Disallow: /shopping/deals Allow: /shopping?udm=28$ Disallow: /storefront Disallow: /storepicker Disallow: /about/careers/applications/candidate-prep Disallow: /about/careers/applications/connect-with-a-googler Disallow: /about/careers/applications/jobs/results?page= Disallow: /about/careers/applications/jobs/results/?page= Disallow: /about/careers/applications/jobs/results?*&page= Disallow: /about/careers/applications/jobs/results/?*&page= Disallow: /landing/signout.html Disallow: /gallery/ Disallow: /landing/now/ontap/ Allow: /maps/reserve Allow: /maps/reserve/partners Disallow: /maps/reserve/api/ Disallow: /maps/reserve/search Disallow: /maps/reserve/bookings Disallow: /maps/reserve/settings Disallow: /maps/reserve/manage Disallow: /maps/reserve/payment Disallow: /maps/reserve/receipt Disallow: /maps/reserve/sellersignup Disallow: /maps/reserve/feedback Disallow: /maps/reserve/terms Disallow: /maps/reserve/m/ Disallow: /maps/reserve/b/ Disallow: /maps/reserve/partner-dashboard Disallow: /local/cars Disallow: /local/dealership/ Disallow: /local/dining/ Disallow: /local/place/products/ Disallow: /local/place/reviews/ Disallow: /local/place/rap/ Disallow: /local/tab/ Disallow: /localservices/ Disallow: /nonprofits/account/ Disallow: /uviewer Disallow: /landing/cmsnext-root/ # AdsBot User-agent: AdsBot-Google Disallow: /maps/api/js/ Allow: /maps/api/js Disallow: /maps/api/place/js/ Disallow: /maps/api/staticmap Disallow: /maps/api/streetview # New user agent groups must also have a user agent reference in the global (*) # group. See "Order of precedence" section in # https://goo.gle/rep#order-of-precedence-for-user-agents User-agent: Yandex Disallow: /about/careers/applications/jobs/results Disallow: /about/careers/applications-a/jobs/results # Crawlers of certain social media sites are allowed to access page markup when # google.com/imgres* links are shared. To learn more, please contact # images-robots-allowlist@google.com. User-agent: facebookexternalhit User-agent: Twitterbot Allow: /imgres Allow: /search Disallow: /groups Disallow: /hosted/images/ Disallow: /m/ Sitemap: https://www.google.com/sitemap.xml

Bot-Matrix

Pfad / 60 erlaubt 0 blockiert
Suchmaschinen (13)
BotUA-TokenmatchedRegelVerdict
Googlebot
Google
Googlebot * implizit erlaubt
Googlebot-Image
Google
Googlebot-Image * implizit erlaubt
Googlebot-News
Google
Googlebot-News * implizit erlaubt
AdsBot-Google
Google
AdsBot-Google * implizit erlaubt
Mediapartners-Google
Google
Mediapartners-Google * implizit erlaubt
Bingbot
Microsoft
bingbot * implizit erlaubt
DuckDuckBot
DuckDuckGo
DuckDuckBot * implizit erlaubt
YandexBot
Yandex
YandexBot * implizit erlaubt
Baiduspider
Baidu
Baiduspider * implizit erlaubt
Applebot
Apple
Applebot * implizit erlaubt
Sogou web spider
Sogou
Sogou web spider * implizit erlaubt
Yeti (Naver)
Naver
Yeti * implizit erlaubt
SeznamBot
Seznam.cz
SeznamBot * implizit erlaubt
AI / LLM (17)
BotUA-TokenmatchedRegelVerdict
Google-Extended
Google
Google-Extended * implizit erlaubt
Applebot-Extended
Apple
Applebot-Extended * implizit erlaubt
GPTBot
OpenAI
GPTBot * implizit erlaubt
ChatGPT-User
OpenAI
ChatGPT-User * implizit erlaubt
OAI-SearchBot
OpenAI
OAI-SearchBot * implizit erlaubt
ClaudeBot
Anthropic
ClaudeBot * implizit erlaubt
Claude-User
Anthropic
Claude-User * implizit erlaubt
Claude-SearchBot
Anthropic
Claude-SearchBot * implizit erlaubt
PerplexityBot
Perplexity
PerplexityBot * implizit erlaubt
Perplexity-User
Perplexity
Perplexity-User * implizit erlaubt
CCBot
Common Crawl
CCBot * implizit erlaubt
Bytespider
ByteDance
Bytespider * implizit erlaubt
meta-externalagent
Meta
meta-externalagent * implizit erlaubt
Amazonbot
Amazon
Amazonbot * implizit erlaubt
cohere-ai
Cohere
cohere-ai * implizit erlaubt
Diffbot
Diffbot
Diffbot * implizit erlaubt
MistralAI-User
Mistral
MistralAI-User * implizit erlaubt
SEO-Tools (10)
BotUA-TokenmatchedRegelVerdict
AhrefsBot
Ahrefs
AhrefsBot * implizit erlaubt
AhrefsSiteAudit
Ahrefs
AhrefsSiteAudit * implizit erlaubt
SemrushBot
Semrush
SemrushBot * implizit erlaubt
MJ12bot (Majestic)
Majestic
MJ12bot * implizit erlaubt
DotBot (Moz)
Moz
DotBot * implizit erlaubt
rogerbot (Moz)
Moz
rogerbot * implizit erlaubt
BLEXBot (WebMeUp)
WebMeUp
BLEXBot * implizit erlaubt
DataForSeoBot
DataForSEO
DataForSeoBot * implizit erlaubt
Screaming Frog SEO Spider
Screaming Frog
Screaming Frog SEO Spider * implizit erlaubt
Sitebulb
Sitebulb
Sitebulb * implizit erlaubt
Social / Messenger (11)
BotUA-TokenmatchedRegelVerdict
facebookexternalhit
Meta
facebookexternalhit facebookexternalhit, Twitterbot implizit erlaubt
meta-externalfetcher
Meta
meta-externalfetcher * implizit erlaubt
Twitterbot
X (Twitter)
Twitterbot facebookexternalhit, Twitterbot implizit erlaubt
LinkedInBot
LinkedIn
LinkedInBot * implizit erlaubt
Slackbot-LinkExpanding
Slack
Slackbot-LinkExpanding * implizit erlaubt
Discordbot
Discord
Discordbot * implizit erlaubt
TelegramBot
Telegram
TelegramBot * implizit erlaubt
WhatsApp
Meta
WhatsApp * implizit erlaubt
Pinterestbot
Pinterest
Pinterestbot * implizit erlaubt
Reddit Bot
Reddit
Reddit * implizit erlaubt
TikTokSpider
ByteDance
TikTokSpider * implizit erlaubt
Archive (1)
BotUA-TokenmatchedRegelVerdict
archive.org_bot
Internet Archive
archive.org_bot * implizit erlaubt
Monitoring (3)
BotUA-TokenmatchedRegelVerdict
UptimeRobot
UptimeRobot
UptimeRobot * implizit erlaubt
Pingdom
SolarWinds
Pingdom * implizit erlaubt
StatusCake
StatusCake
StatusCake * implizit erlaubt
Security-Scanner (5)
BotUA-TokenmatchedRegelVerdict
CensysInspect
Censys
CensysInspect * implizit erlaubt
Shodan
Shodan
Shodan * implizit erlaubt
Palo Alto Expanse
Palo Alto
expanse * implizit erlaubt
l9scan / leakix
LeakIX
l9scan * implizit erlaubt
NetcraftSurveyAgent
Netcraft
NetcraftSurveyAgent * implizit erlaubt

Mehr Details, Verifikations-Hinweise und ein robots.txt-Generator in der Bot-Datenbank.