crawlerbase.de / trace Bot-Datenbank →
github.com 1 Hop/70 ms/200 HTTP/2
+ Neu Re-Run →

Übersicht

final URL: https://github.com/
1
Hops
70 ms
Total
60
Bots erlaubt
0
Bots blockiert

Redirect-Chain

1 Hop
  1. 00
    200 HTTP/2 https://github.com
    TTFB 36 ms Total 70 ms Size 122,9 KB IP 140.82.121.4 Server github.com
    TLS Issuer C = GB, O = Sectigo Limited, CN = Sectigo Public Server Authentication CA DV E36 Gültig noch 39 Tage
    Alle Header (17)
    dateWed, 24 Jun 2026 03:14:35 GMT
    content-typetext/html; charset=utf-8
    varyX-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, X-Requested-With, Accept-Language, Sec-Fetch-Site,Accept-Encoding, Accept, X-Requested-With
    content-languageen-US
    etagW/"59c5882e58ef4e7f348b64cfe5517851"
    cache-controlmax-age=0, private, must-revalidate
    strict-transport-securitymax-age=31536000; includeSubdomains; preload
    x-frame-optionsdeny
    x-content-type-optionsnosniff
    x-xss-protection0
    referrer-policyorigin-when-cross-origin, strict-origin-when-cross-origin
    content-security-policydefault-src 'none'; base-uri 'self'; child-src github.githubassets.com github.com/assets-cdn/worker/ github.com/assets/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com www.githubstatus.com collector.github.com raw.githubusercontent.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com *.rel.tunnels.api.visualstudio.com wss://*.rel.tunnels.api.visualstudio.com github.githubassets.com objects-origin.githubusercontent.com copilot-proxy.githubusercontent.com proxy.individual.githubcopilot.com proxy.business.githubcopilot.com proxy.enterprise.githubcopilot.com *.actions.githubusercontent.com wss://*.actions.githubusercontent.com productionresultssa0.blob.core.windows.net productionresultssa1.blob.core.windows.net productionresultssa2.blob.core.windows.net productionresultssa3.blob.core.windows.net productionresultssa4.blob.core.windows.net productionresultssa5.blob.core.windows.net productionresultssa6.blob.core.windows.net productionresultssa7.blob.core.windows.net productionresultssa8.blob.core.windows.net productionresultssa9.blob.core.windows.net productionresultssa10.blob.core.windows.net productionresultssa11.blob.core.windows.net productionresultssa12.blob.core.windows.net productionresultssa13.blob.core.windows.net productionresultssa14.blob.core.windows.net productionresultssa15.blob.core.windows.net productionresultssa16.blob.core.windows.net productionresultssa17.blob.core.windows.net productionresultssa18.blob.core.windows.net productionresultssa19.blob.core.windows.net github-production-repository-image-32fea6.s3.amazonaws.com github-production-release-asset-2e65be.s3.amazonaws.com insights.github.com wss://alive.github.com wss://alive-staging.github.com api.githubcopilot.com api.individual.githubcopilot.com api.business.githubcopilot.com api.enterprise.githubcopilot.com wss://production-copilot-host.webpubsub.azure.com edge.fullstory.com rs.fullstory.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com copilot-workspace.githubnext.com objects-origin.githubusercontent.com; frame-ancestors 'none'; frame-src viewscreen.githubusercontent.com notebooks.githubusercontent.com www.youtube-nocookie.com; img-src 'self' data: blob: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com avatars.githubusercontent.com private-avatars.githubusercontent.com github-cloud.s3.amazonaws.com objects.githubusercontent.com release-assets.githubusercontent.com secured-user-images.githubusercontent.com user-images.githubusercontent.com private-user-images.githubusercontent.com opengraph.githubassets.com marketplace-screenshots.githubusercontent.com copilotprodattachments.blob.core.windows.net/github-production-copilot-attachments/ github-production-user-asset-6210df.s3.amazonaws.com customer-stories-feed.github.com spotlights-feed.github.com explore-feed.github.com objects-origin.githubusercontent.com *.githubusercontent.com images.ctfassets.net/8aevphvgewt8/; manifest-src 'self'; media-src github.com user-images.githubusercontent.com secured-user-images.githubusercontent.com private-user-images.githubusercontent.com github-production-user-asset-6210df.s3.amazonaws.com gist.github.com github.githubassets.com assets.ctfassets.net/8aevphvgewt8/ videos.ctfassets.net/8aevphvgewt8/; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com; upgrade-insecure-requests; worker-src github.githubassets.com github.com/assets-cdn/worker/ github.com/assets/ gist.github.com/assets-cdn/worker/
    servergithub.com
    content-encodinggzip
    accept-rangesbytes
    set-cookie_gh_sess=foxh%2Bq1EQwqZxY%2B39uPwizm9%2B%2BsZmnrPzPc2huy3ratwJdLYLj3MtjgNDxH6Ki732mPIxGmTCMEUFEBmZC7tiKmany6RMHkjZ1kVmOp4EU%2FrZkl2tTMG2tBiIXy5Dh28aj0TJOj8Ly3f1f3SSHko17JZ7VdA%2FdVeSgCCbFStozUITOrOlFyiWGMs9ePXz3k0UTfRgdhfade7Hbl9Yz2vYQhwsaYy8FnLWT9wmgXXXTH4xF%2Fr85CnuNLm8ry9Kbnub32Af72DhL%2BJ5Zr%2BOchjbg%3D%3D--ndN9lTRdaz%2BTk3Lb--rx6fori8YA2u8lkBfFWGaQ%3D%3D; path=/; HttpOnly; secure; SameSite=Lax
    set-cookie_octo=GH1.1.1401807614.1782270878; expires=Thu, 24 Jun 2027 03:14:38 GMT; domain=.github.com; path=/; secure; SameSite=Lax
    set-cookielogged_in=no; expires=Thu, 24 Jun 2027 03:14:38 GMT; domain=.github.com; path=/; HttpOnly; secure; SameSite=Lax
    x-github-request-idCFEE:21C9D2:10065F0B:CF80E42:6A3B4B9E

robots.txt / github.com

vorhanden 5 Gruppen 0 Sitemaps
robots.txt Quelltext (2,2 KB)
github.com/robots.txt
# If you would like to crawl GitHub contact us via https://support.github.com?tags=dotcom-robots # We also provide an extensive API: https://docs.github.com User-agent: bingbot Disallow: /ekansa/Open-Context-Data Disallow: /ekansa/opencontext-* Disallow: /account-login Disallow: */tarball/ Disallow: */zipball/ Disallow: /Explodingstuff/ Disallow: /copilot/ Disallow: /copilot/c/ User-agent: adidxbot Disallow: /ekansa/Open-Context-Data Disallow: /ekansa/opencontext-* Disallow: /account-login Disallow: */tarball/ Disallow: */zipball/ Disallow: /Explodingstuff/ Disallow: /copilot/ Disallow: /copilot/c/ User-agent: BingPreview Disallow: /ekansa/Open-Context-Data Disallow: /ekansa/opencontext-* Disallow: /account-login Disallow: */tarball/ Disallow: */zipball/ Disallow: /Explodingstuff/ Disallow: /copilot/ Disallow: /copilot/c/ User-agent: baidu crawl-delay: 1 User-agent: * Disallow: /*/*/pulse Disallow: /*/*/projects Disallow: /*/*/forks Disallow: /*/*/issues/new Disallow: /*/*/milestones/new Disallow: /*/*/issues/search Disallow: /*/*/commits/ Disallow: /*/*/branches Disallow: /*/*/contributors Disallow: /*/*/tags Disallow: /*/*/stargazers Disallow: /*/*/watchers Disallow: /*/*/network Disallow: /*/*/graphs Disallow: /*/*/compare Disallow: /*/tree/ Disallow: /gist/ Disallow: /*/download Disallow: /*/revisions Disallow: /*/commits/*?author Disallow: /*/commits/*?path Disallow: /*/comments Disallow: /*/archive/ Disallow: /*/blame/ Disallow: /*/raw/ Disallow: /*/cache/ Disallow: /.git/ Disallow: */.git/ Disallow: /*.git$ Disallow: /search/advanced Disallow: /search$ Disallow: /*q= Disallow: /*.atom$ Disallow: /ekansa/Open-Context-Data Disallow: /ekansa/opencontext-* Disallow: */tarball/ Disallow: */zipball/ Disallow: /*source=* Disallow: /*ref_cta=* Disallow: /*plan=* Disallow: /*return_to=* Disallow: /*ref_loc=* Disallow: /*setup_organization=* Disallow: /*source_repo=* Disallow: /*ref_page=* Disallow: /*source=* Disallow: /*referrer=* Disallow: /*report=* Disallow: /*author=* Disallow: /*since=* Disallow: /*until=* Disallow: /*commits?author=* Disallow: /*report-abuse?report=* Disallow: /*tab=* Allow: /*?tab=achievements&achievement=* Disallow: /account-login Disallow: /Explodingstuff/ Disallow: /copilot/ Disallow: /copilot/c/

Bot-Matrix

Pfad / 60 erlaubt 0 blockiert
Suchmaschinen (13)
BotUA-TokenmatchedRegelVerdict
Googlebot
Google
 Googlebot * implizit erlaubt
Googlebot-Image
Google
 Googlebot-Image * implizit erlaubt
Googlebot-News
Google
 Googlebot-News * implizit erlaubt
AdsBot-Google
Google
 AdsBot-Google * implizit erlaubt
Mediapartners-Google
Google
 Mediapartners-Google * implizit erlaubt
Bingbot
Microsoft
 bingbot * implizit erlaubt
DuckDuckBot
DuckDuckGo
 DuckDuckBot * implizit erlaubt
YandexBot
Yandex
 YandexBot * implizit erlaubt
Baiduspider
Baidu
 Baiduspider * implizit erlaubt
Applebot
Apple
 Applebot * implizit erlaubt
Sogou web spider
Sogou
 Sogou web spider * implizit erlaubt
Yeti (Naver)
Naver
 Yeti * implizit erlaubt
SeznamBot
Seznam.cz
 SeznamBot * implizit erlaubt
AI / LLM (17)
BotUA-TokenmatchedRegelVerdict
Google-Extended
Google
 Google-Extended * implizit erlaubt
Applebot-Extended
Apple
 Applebot-Extended * implizit erlaubt
GPTBot
OpenAI
 GPTBot * implizit erlaubt
ChatGPT-User
OpenAI
 ChatGPT-User * implizit erlaubt
OAI-SearchBot
OpenAI
 OAI-SearchBot * implizit erlaubt
ClaudeBot
Anthropic
 ClaudeBot * implizit erlaubt
Claude-User
Anthropic
 Claude-User * implizit erlaubt
Claude-SearchBot
Anthropic
 Claude-SearchBot * implizit erlaubt
PerplexityBot
Perplexity
 PerplexityBot * implizit erlaubt
Perplexity-User
Perplexity
 Perplexity-User * implizit erlaubt
CCBot
Common Crawl
 CCBot * implizit erlaubt
Bytespider
ByteDance
 Bytespider * implizit erlaubt
meta-externalagent
Meta
 meta-externalagent * implizit erlaubt
Amazonbot
Amazon
 Amazonbot * implizit erlaubt
cohere-ai
Cohere
 cohere-ai * implizit erlaubt
Diffbot
Diffbot
 Diffbot * implizit erlaubt
MistralAI-User
Mistral
 MistralAI-User * implizit erlaubt
SEO-Tools (10)
BotUA-TokenmatchedRegelVerdict
AhrefsBot
Ahrefs
 AhrefsBot * implizit erlaubt
AhrefsSiteAudit
Ahrefs
 AhrefsSiteAudit * implizit erlaubt
SemrushBot
Semrush
 SemrushBot * implizit erlaubt
MJ12bot (Majestic)
Majestic
 MJ12bot * implizit erlaubt
DotBot (Moz)
Moz
 DotBot * implizit erlaubt
rogerbot (Moz)
Moz
 rogerbot * implizit erlaubt
BLEXBot (WebMeUp)
WebMeUp
 BLEXBot * implizit erlaubt
DataForSeoBot
DataForSEO
 DataForSeoBot * implizit erlaubt
Screaming Frog SEO Spider
Screaming Frog
 Screaming Frog SEO Spider * implizit erlaubt
Sitebulb
Sitebulb
 Sitebulb * implizit erlaubt
Social / Messenger (11)
BotUA-TokenmatchedRegelVerdict
facebookexternalhit
Meta
 facebookexternalhit * implizit erlaubt
meta-externalfetcher
Meta
 meta-externalfetcher * implizit erlaubt
Twitterbot
X (Twitter)
 Twitterbot * implizit erlaubt
LinkedInBot
LinkedIn
 LinkedInBot * implizit erlaubt
Slackbot-LinkExpanding
Slack
 Slackbot-LinkExpanding * implizit erlaubt
Discordbot
Discord
 Discordbot * implizit erlaubt
TelegramBot
Telegram
 TelegramBot * implizit erlaubt
WhatsApp
Meta
 WhatsApp * implizit erlaubt
Pinterestbot
Pinterest
 Pinterestbot * implizit erlaubt
Reddit Bot
Reddit
 Reddit * implizit erlaubt
TikTokSpider
ByteDance
 TikTokSpider * implizit erlaubt
Archive (1)
BotUA-TokenmatchedRegelVerdict
archive.org_bot
Internet Archive
 archive.org_bot * implizit erlaubt
Monitoring (3)
BotUA-TokenmatchedRegelVerdict
UptimeRobot
UptimeRobot
 UptimeRobot * implizit erlaubt
Pingdom
SolarWinds
 Pingdom * implizit erlaubt
StatusCake
StatusCake
 StatusCake * implizit erlaubt
Security-Scanner (5)
BotUA-TokenmatchedRegelVerdict
CensysInspect
Censys
 CensysInspect * implizit erlaubt
Shodan
Shodan
 Shodan * implizit erlaubt
Palo Alto Expanse
Palo Alto
 expanse * implizit erlaubt
l9scan / leakix
LeakIX
 l9scan * implizit erlaubt
NetcraftSurveyAgent
Netcraft
 NetcraftSurveyAgent * implizit erlaubt

Mehr Details, Verifikations-Hinweise und ein robots.txt-Generator in der Bot-Datenbank.