Common Crawl Foundation
Common Crawl is a non-profit foundation dedicated to the Open Web.
- The latest Web Graphs from the November and December 2025 and January 2026 crawls are now available, comprising 279.4 million host-level nodes with 13.4 billion edges, and 122.3 million domain-level nodes with 6.1 billion edges. www.commoncrawl.org/blog/host--a...
- We are pleased to announce the release of the January 2026 crawl archive, containing 2.3 billion web pages, or 398 TiB of uncompressed content. www.commoncrawl.org/blog/january...
- Recently, a two-day Bristol datathon used Common Crawl web archives to analyse UK industries and policy, strengthening social science research through hands-on, team-based work. www.commoncrawl.org/blog/web-arc...
- As SEOs grapple with the shift from traditional Search Engine Optimization to AI visibility, they're discovering a resource that's been powering AI training for years: Common Crawl's Web Graph. commoncrawl.org/blog/how-seo...
- GneissWeb Annotations Examples A new Common Crawl index annotation has been added to Hugging Face and our S3 bucket. commoncrawl.org/blog/gneissw...
- From the 6th to the 10th of November 2025, Pedro Ortiz Suarez attended Mozfest in Barcelona, as well as some satellite events. www.commoncrawl.org/blog/common-...
- We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of October, November, and December 2025. commoncrawl.org/blog/host--a...
- The crawl archive for December 2025 is now available, consisting of 2.16 billion web pages (or 364 TiB of uncompressed content). commoncrawl.org/blog/decembe...
- As another year here at Common Crawl comes to a close, we present a dozen papers from 2025 that demonstrate the range of topics and areas of study for which Common Crawl’s datasets are used and referenced. commoncrawl.org/blog/a-sampl...
- Reposted by Common Crawl Foundation[Not loaded yet]
- We are pleased to announce the release of the web graphs based on the crawls of September, October, and November of 2025, consisting of 235.7 million nodes and 9.5 billion edges at the host level, and 100.7 million nodes and 6.6 billion edges at the domain level. commoncrawl.org/blog/host--a...
- We are pleased to announce that the crawl archive for November 2025 is now available, containing 2.29 billion web pages or 378 TiB of uncompressed content. commoncrawl.org/blog/novembe...
- Common Crawl celebrates World Digital Preservation Day Nov. 6, which invites the community to unite in answering a powerful question: Why Preserve? commoncrawl.org/blog/common-...
- Setting the Record Straight A recent article in The Atlantic makes several false and misleading claims about the Common Crawl Foundation, including the accusation that our organization has “lied to publishers” about our activities. commoncrawl.org/blog/setting...
- Check out our newsletter for October/November 2025, with updates on what we've been up to commoncrawl.org/blog/october...
- The Common Crawl team presented a seminar at Stanford HAI entitled “Preserving Humanity's Knowledge and Making it Accessible: Addressing Challenges of Public Web Data”. commoncrawl.org/blog/common-...
- We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of August, September, and October 2025, consisting of of 468.4 million nodes and 8.0 billion edges at the host level, and 97.7 million nodes and 6.0 billion edges at the domain level.
- We are pleased to announce the release of the October 2025 crawl, containing 2.61 billion web pages or 468 TiB of uncompressed content. commoncrawl.org/blog/october...
- The Common Crawl team attended the 2nd Conference on Language Modeling in Montréal, organizing a workshop, giving invited talks, and strengthening links with the research community. commoncrawl.org/blog/common-...
- Reposted by Common Crawl Foundation[Not loaded yet]
- Reposted by Common Crawl Foundation[Not loaded yet]
- Reposted by Common Crawl Foundation[Not loaded yet]
- Reposted by Common Crawl Foundation[Not loaded yet]
- Reposted by Common Crawl Foundation[Not loaded yet]
- Reposted by Common Crawl Foundation[Not loaded yet]
- Common Crawl has added IBM’s GneissWeb quality and category annotations to its web dataset, enabling users to filter high-quality content and explore topics like medical, education, and technology. commoncrawl.org/blog/announc...
- Common Crawl’s Web Languages initiative has had many contributions since its introduction. We’re calling for native speakers of certain languages to review language contributions, to ensure that links we’re adding to our seed crawl are of good quality. commoncrawl.org/blog/web-lan...
- We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of July, August, and September 2025. The host-level graph consists of 628.7 million nodes and 6.9 billion edges, and the domain-level graph consists of 184.6 million nodes and 5.4 billion edges.
- The era of traditional search engine optimization is rapidly evolving into "AIO" (AI optimization), where businesses must ensure their content exists in AI training datasets to remain discoverable as users increasingly turn to AI assistants for answers. commoncrawl.org/blog/from-se...
- We are pleased to announce the release of our September 2025 crawl, containing 2.39 billion web pages, or 421 TiB of uncompressed content. www.commoncrawl.org/blog/septemb...
- Publishers have been sending Common Crawl legal opt-out requests. In the interest of transparency and to better serve our ecosystem, we are publishing the full opt-out list for every legal request we have received. commoncrawl.org/blog/common-...
- On the 28th and 29th of August 2025, Thom Vaughan, Pedro Ortiz Suarez, and Thijs Dalhuijsen attended the Linux Foundation’s AI_dev event in Amsterdam. commoncrawl.org/blog/trip-re...
- On October 22, the Common Crawl team will lead a seminar at Stanford HAI. Our topic of discussion is “Preserving Humanity's Knowledge and Making it Accessible: Addressing Challenges of Public Web Data”. Please register at: hai.stanford.edu/events/commo...
- We’re Walling Off The Open Internet To Stop AI—And It May End Up Breaking Everything Else www.techdirt.com/2025/09/08/w...
- Stanford HAI and Common Crawl are joining forces to explore how open data can shape the future of AI. On 22 October 2025, their seminar will address privacy, safety, and security while showcasing new ways to preserve and share humanity’s knowledge. www.commoncrawl.org/blog/common-...
- We are pleased to release our newsletter for July and August 2025, with updates on our team's activities. commoncrawl.org/blog/july-au...
- We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of June, July, and August 2025. commoncrawl.org/blog/host--a...
- We are pleased to announce the release of our August 2025 crawl, containing 2.44 billion web pages (or 424 TiB of uncompressed content). commoncrawl.org/blog/august-...
- Publishers and brands are shifting from SEO to AIO. Many SEOs unknowingly block their sites from AI search by restricting CCBot in robots.txt. As Search 2.0 transforms discovery, ensuring content can train AI models becomes as crucial as traditional SEO. commoncrawl.org/blog/ai-opti...
- The Enclosure Of The Open Web And The Open Internet Toll Booth: What’s Behind Pay-By-Crawl digitalmedusa.org/the-enclosur...
- A report on IETF 123 in Madrid, including sessions on AI content preferences, bot authentication, and web measurement. commoncrawl.org/blog/ietf-12...
- Our Web Graph release for July 2025 is now available, consisting of 481.6 million nodes and 3.4 billion edges at the host level, and 209.5 million nodes and 2.6 billion edges at the domain level. commoncrawl.org/blog/host--a...
- The crawl archive for July 2025 is now available. Crawled between July 7th and July 21st, the data contains 2.42 billion web pages, or 419 TiB of uncompressed content. commoncrawl.org/blog/july-20...
- The Common Crawl Foundation, MLCommons, EleutherAI, and John Hopkins' Center for Language and Speech Processing have the pleasure of inviting you to register for the 1st shared task on Language Identification for web data. commoncrawl.org/blog/wmdqs-s...
- "MOIC will also partner with Common Crawl, one of the largest free and open repositories of web crawled data. MOIC will fund work at Common Crawl, leveraging native speakers to annotate and seed European language data in the publicly available Common Crawl data set."
- In June 2025 the Common Crawl Foundation, MLCommons, and EleutherAI had the pleasure of hosting a virtual hackathon in partnership with Masakhane in order to collect language identification annotations for African languages. commoncrawl.org/blog/the-fir...
- We are pleased to announce that the Web Graph for June 2025 is now available. The graph consists of 371.6 million nodes and 3.1 billion edges at the host level, and 161.8 million nodes and 2.2 billion edges at the domain level. commoncrawl.org/blog/host--a...
- The Common Crawl Foundation team took part in the United Nations Open Source Week in New York City this June, meeting with global developers, researchers, and policymakers to discuss all things open source and AI. commoncrawl.org/blog/common-...
- We are pleased to announce that the crawl archive for June 2025 is now available. www.commoncrawl.org/blog/june-20...
- We're happy to share our newsletter for May/June 2025 with updates from our team. commoncrawl.org/blog/may-jun...
- The deadline for paper submissions has been extended! The new deadline is July 3, 2025. AoE. For more information, please visit: wmdqs.org
- Call for papers! We are organising the 1st Workshop on Multilingual Data Quality Signals with @mlcommons.org and @eleutherai.bsky.social, held in tandem with @colmweb.org. Submit your research on multilingual data quality! Submission deadline is 23 June, more info: wmdqs.org
- The AI Alliance Forms Non-profit AI Lab and AI Technology & Advocacy Association to Scale Open-Source Innovation www.prnewswire.com/news-release...
- Announcing a refreshed version of the Whirlwind Tour in Python. Get to know how to make the most of our crawl data. commoncrawl.org/blog/announc...
- The Common Crawl Foundation, together with IBM, the AI Alliance, and BrightQuery will be hosting an "UN Conference" at IBM's new flagship NYC HQ at One Madison Avenue on Friday, June 20, from 12:30-5pm. If you are in NYC, it would be great to see you there! lu.ma/p0a1scde
- We are pleased to announce that the Web Graph for May 2025 is now available. The graph consists of 326.8 million nodes and 2.9 billion edges at the host level, and 156.1 million nodes and 2.1 billion edges at the domain level. commoncrawl.org/blog/host--a...
- We are pleased to announce that the crawl archive for May 2025 is now available. The data was crawled between May 11th and May 25th, and contains 2.47 billion web pages, or 429 TiB of uncompressed content. commoncrawl.org/blog/may-202...
- Call for papers! We are organising the 1st Workshop on Multilingual Data Quality Signals with @mlcommons.org and @eleutherai.bsky.social, held in tandem with @colmweb.org. Submit your research on multilingual data quality! Submission deadline is 23 June, more info: wmdqs.org
