Discover / Data & Research

Colly Fast Go Scraping Framework

Name: Colly Fast Go Scraping Framework
Author: gocolly

by gocollyGo

Fast and elegant scraping and crawling framework for Go.

Toolactive

Maturity: active because commit 46d ago, latest release v2.2.0. Derived from release and commit history, not a rating.

Stars: 25k
Forks: 1.9k
Downloads / mo: —
Last commit: 2026-06-18
License: Apache-2.0
Open issues: 191

Market and trust evidence

Edition not yet matched

No exact skills.sh identity match is available for this repository. Repository adoption and freshness remain visible above; install momentum is not inferred.

Trust analysis is a screening signal, not a security warranty. Read the ranking and trust methodology.

In practice

Written by AI from this repository’s README · high confidence

Writing a crawler in Go means hand rolling concurrency limits, cookie handling, caching and robots.txt support.

Use it when

When you are building a Go crawler and want HTML element callbacks with built in rate limiting and caching.

Not the right pick when

Wrong pick outside Go, since the API is distributed as a Go package and all examples are Go code.

Capabilities

clean API with OnHTML and OnRequest callbacks
manages request delays and maximum concurrency per domain
automatic cookie and session handling
sync, async and parallel scraping
caching and robots.txt support
distributed scraping and extensions

Cost: Free and open source

Compare Repository ↗Homepage ↗

Video walkthroughs

How to Build a Go Web Scraper with Colly

ShandonCodes · 2K views · ≈Aug 2021

Scrape IMDB 🎬🎬 With GOLANG! - Golang Project 2021 🎉🎉 [PART 3 ] #IMDBcrawler #scrapeIMDB #gocolly

Akhil Sharma · 901 views · ≈Aug 2021

Third-party YouTube uploads matched to this tool by title, channel and repository name on 2026-08-03. Not made, reviewed or endorsed by SkillPilot. View counts and publish months are as of the match date and the month is approximate. Nothing loads from YouTube until you press play.

What the repository ships

CI configured

Detected from the actual files in the repository root.

Latest release v2.2.0

Published 2025-03-27

Tons of bug fixes & little improvements & updated dependencies

What's Changed

Added new project that uses Colly by @twiny in https://github.com/gocolly/colly/pull/525
adding docker-slim to the list of open source projects using Colly by @kcq in https://github.com/gocolly/colly/pull/526
update coursera example by @ysung6 in https://github.com/gocolly/colly/pull/537
Update openedx_courses.go by @lingt-xyz in https://github.com/gocolly/colly/pull/539
feat: support Stop() in queue by @PalanQu in https://github.com/gocolly/colly/pull/531
Fix invalid base URL assignment by @dasshield in https://github.com/gocolly/colly/pull/549
Add context.Context support by @WGH- in https://github.com/gocolly/colly/pull/548
Fix handling of redirected URLs in OnResponseHeaders by @WGH- in https://github.com/gocolly/colly/pull/561
Fix race on count fields of Collector type by @shubham14bajpai in https://github.com/gocolly/colly/pull/563
SetProxyFunc: fix proxies rotation by @wowinter13 in https://github.com/gocolly/colly/pull/567
Update xkcd_store.go by @BigManing in https://github.com/gocolly/colly/pull/570
Updated xmlquery to close a known security vulnerability by @ijomeli in https://github.com/gocolly/colly/pull/582
Fix default User-Agent when using custom headers by @WGH- in https://github.com/gocolly/colly/pull/592
Update openedx_courses.go by @BigManing in https://github.com/gocolly/colly/pull/576
Add httpBackend cache response callback by @slashapp in https://github.com/gocolly/colly/pull/574
Update README.md by @seversky in https://github.com/gocolly/colly/pull/595
Change the type assersion for Attr by @moritamori in https://github.com/gocolly/colly/pull/588
Update User-Agent strings for Desktop by @guessi in https://github.com/gocolly/colly/pull/605
add quote crawler using colly by @eval-exec in https://github.com/gocolly/colly/pull/603
Fix golint issues by @waltton in https://github.com/gocolly/colly/pull/620
Replace Travis with GitHub Actions by @alexhraber in https://github.com/gocolly/colly/pull/669
Use github.com/nlnwa/whatwg-url for URL parsing by @WGH- in https://github.com/gocolly/colly/pull/673
Fix Async ignoring its arguments by @ljbink in https://github.com/gocolly/colly/pull/652
fix: generated code has compile error by @darjun in https://github.com/gocolly/colly/pull/629
moving filterings to one function (checkFilters) and use it in case of redirect too by @joelazar in https://github.com/gocolly/colly/pull/638
Fix Contributing.md link by @SaladinoBelisario in https://github.com/gocolly/colly/pull/641
Work around invalid URL encoding in path by @WGH- in https://github.com/gocolly/colly/pull/675
Host & Custom Header Support by @ErikOwen in https://github.com/gocolly/colly/pull/691
Code cleanup by @WGH- in https://github.com/gocolly/colly/pull/678
Add GitHub issue templates and nudge questions to Stack Overflow by @WGH- in https://github.com/gocolly/colly/pull/702
Fix redirects ignoring AllowURLRevisit=false by @WGH- in https://github.com/gocolly/colly/pull/681
Bump whatwg-url version by @WGH- in https://github.com/gocolly/colly/pull/713
fix some typos by @cuishuang in https://github.com/gocolly/colly/pull/736
feat: Add MaxRequests paramete by @guilhem in https://github.com/gocolly/colly/pull/722
Update CI configuration by @WGH- in https://github.com/gocolly/colly/pull/749
upgrade Cascadia version by @kinoute in https://github.com/gocolly/colly/pull/650
Bump golang.org/x/net from 0.0.0-20220114011407-0dd24b26b47d to 0.7.0 by @dependabot in https://github.com/gocolly/colly/pull/756
Support websites redirecting to the same page when AllowURLRevisit is disabled by @WGH- in https://github.com/gocolly/colly/pull/763
Update User-Agent strings for 2023 by @guessi in https://github.com/gocolly/colly/pull/765
Fix data races by @WGH- in https://github.com/gocolly/colly/pull/771
Replaced deprecated io/ioutil packa

README

Colly

Lightning Fast and Elegant Scraping Framework for Gophers

Colly provides a clean interface to write any kind of crawler/scraper/spider.

With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.

GoDoc

Backers on Open Collective Sponsors on Open Collective build status

Features

Clean API
Fast (>1k request/sec on a single core)
Manages request delays and maximum concurrency per domain
Automatic cookie and session handling
Sync/async/parallel scraping
Caching
Automatic encoding of non-unicode responses
Robots.txt support
Distributed scraping
Configuration via environment variables
Extensions

Example



import (
	"fmt"

	"github.com/gocolly/colly/v2"
)

func main() {
	c := colly.NewCollector()

	// Find and visit all links
	c.OnHTML("a[href]", func(e *colly.HTMLElement) {
		e.Request.Visit(e.Attr("href"))
	})

	c.OnRequest(func(r *colly.Request) {
		fmt.Println("Visiting", r.URL)
	})

	c.Visit("http://go-colly.org/")
}

See examples folder for more detailed examples.

Installation

go get github.com/gocolly/colly/v2

Bugs

Bugs or suggestions? Visit the issue tracker or join #colly on freenode

Other Projects Using Colly

Below is a list of public, open source projects that use Colly:

greenpeace/check-my-pages Scraping script to test the Spanish Greenpeace web archive.
altsab/gowap Wappalyzer implementation in Go.
jesuiscamille/goquotes A quotes scraper, making your day a little better!
jivesearch/jivesearch A search engine that doesn't track you.
Leagify/colly-draft-prospects A scraper for future NFL Draft prospects.
lucasepe/go-ps4 Search playstation store for your favorite PS4 games using the command line.
yringler/inside-chassidus-scraper Scrapes Rabbi Paltiel's web site for lesson metadata.
gamedb/gamedb A database of Steam games.
lawzava/scrape CLI for email scraping from any website.
eureka101v/WeiboSpiderGo A sina weibo(chinese twitter) scraper
Go-phie/gophie Search, Download and Stream movies from your terminal
imthaghost/goclone Clone websites to your computer within seconds.
superiss/spidy Crawl the web and collect expired domains.
docker-slim/docker-slim Optimize your Docker containers to make them smaller and better.
seversky/gachifinder an agent for asynchronous scraping, parsing and writing to some storages(elasticsearch for now)
eval-exec/goodreads crawl all tags and all pages of quotes from goodreads.

If you are using Colly in a project please send a pull request to add it to the list.

Contributors

This project exists thanks to all the people who contribute. [[Contribute]](CONTRIBUTING.md).

Backers

Thank you to all our backers! 🙏 [Become a backer]

License