README
¶
See the video on the Youtube.
Introduction
DHT implements the bittorrent DHT protocol in Go. Now it includes:
It contains two modes, the standard mode and the crawling mode. The standard mode follows the BEPs, and you can use it as a standard dht server. The crawling mode aims to crawl as more metadata info as possiple. It doesn't follow the standard BEPs protocol. With the crawling mode, you can build another BTDigg.
bthub.io is a BT search engine based on the crawling mode.
Installation
go get github.com/shiyanhui/dht
Example
Below is a simple spider. You can move here to see more samples.
import (
"fmt"
"github.com/shiyanhui/dht"
)
func main() {
downloader := dht.NewWire(65535)
go func() {
// once we got the request result
for resp := range downloader.Response() {
fmt.Println(resp.InfoHash, resp.MetadataInfo)
}
}()
go downloader.Run()
config := dht.NewCrawlConfig()
config.OnAnnouncePeer = func(infoHash, ip string, port int) {
// request to download the metadata info
downloader.Request([]byte(infoHash), ip, port)
}
d := dht.New(config)
d.Run()
}
Download
You can download the demo compiled binary file here.
Note
- The default crawl mode configure costs about 300M RAM. Set MaxNodes and BlackListMaxSize to fit yourself.
- Now it cant't run in LAN because of NAT.
TODO
- NAT Traversal.
- Implements the full BEP-3.
- Optimization.
FAQ
Why it is slow compared to other spiders ?
Well, maybe there are several reasons.
- DHT aims to implements the standard BitTorrent DHT protocol, not born for crawling the DHT network.
- NAT Traversal issue. You run the crawler in a local network.
- It will block ip which looks like bad and a good ip may be mis-judged.
License
MIT, read more here
Documentation
¶
Overview ¶
Package dht implements the bittorrent dht protocol. For more information see http://www.bittorrent.org/beps/bep_0005.html.
Index ¶
- Constants
- Variables
- func Decode(data []byte) (result interface{}, err error)
- func DecodeDict(data []byte, start int) (result interface{}, index int, err error)
- func DecodeInt(data []byte, start int) (result interface{}, index int, err error)
- func DecodeList(data []byte, start int) (result interface{}, index int, err error)
- func DecodeString(data []byte, start int) (result interface{}, index int, err error)
- func Encode(data interface{}) string
- func EncodeDict(data map[string]interface{}) string
- func EncodeInt(data int) string
- func EncodeList(data []interface{}) string
- func EncodeString(data string) string
- func ParseKey(data map[string]interface{}, key string, t string) error
- func ParseKeys(data map[string]interface{}, pairs [][]string) error
- type Config
- type DHT
- type Peer
- type Request
- type Response
- type Wire
Constants ¶
const ( // StandardMode follows the standard protocol StandardMode = iota // CrawlMode for crawling the dht network. CrawlMode )
const ( // REQUEST represents request message type REQUEST = iota // DATA represents data message type DATA // REJECT represents reject message type REJECT )
const ( // BLOCK is 2 ^ 14 BLOCK = 16384 // MaxMetadataSize represents the max medata it can accept MaxMetadataSize = BLOCK * 1000 // EXTENDED represents it is a extended message EXTENDED = 20 // HANDSHAKE represents handshake bit HANDSHAKE = 0 )
Variables ¶
var ( // ErrNotReady is the error when DHT is not initialized. ErrNotReady = errors.New("dht is not ready") // ErrOnGetPeersResponseNotSet is the error that config // OnGetPeersResponseNotSet is not set when call dht.GetPeers. ErrOnGetPeersResponseNotSet = errors.New("OnGetPeersResponse is not set") )
Functions ¶
func DecodeDict ¶
DecodeDict decodes a map value.
func DecodeList ¶
DecodeList decodes a list value.
func DecodeString ¶
DecodeString decodes a string in the data. It returns a tuple (decoded result, the end position, error).
func Encode ¶
func Encode(data interface{}) string
Encode encodes a string, int, dict or list value to a bencoded string.
func EncodeDict ¶
EncodeDict encodes a dict value.
Types ¶
type Config ¶
type Config struct {
// in mainline dht, k = 8
K int
// for crawling mode, we put all nodes in one bucket, so KBucketSize may
// not be K
KBucketSize int
// candidates are udp, udp4, udp6
Network string
// format is `ip:port`
Address string
// the prime nodes through which we can join in dht network
PrimeNodes []string
// the kbucket expired duration
KBucketExpiredAfter time.Duration
// the node expired duration
NodeExpriedAfter time.Duration
// how long it checks whether the bucket is expired
CheckKBucketPeriod time.Duration
// peer token expired duration
TokenExpiredAfter time.Duration
// the max transaction id
MaxTransactionCursor uint64
// how many nodes routing table can hold
MaxNodes int
// callback when got get_peers request
OnGetPeers func(string, string, int)
// callback when receive get_peers response
OnGetPeersResponse func(string, *Peer)
// callback when got announce_peer request
OnAnnouncePeer func(string, string, int, int)
// blcoked ips
BlockedIPs []string
// blacklist size
BlackListMaxSize int
// StandardMode or CrawlMode
Mode int
// the times it tries when send fails
Try int
// the size of packet need to be dealt with
PacketJobLimit int
// the size of packet handler
PacketWorkerLimit int
// the nodes num to be fresh in a kbucket
RefreshNodeNum int
}
Config represents the configure of dht.
func NewCrawlConfig ¶
func NewCrawlConfig() *Config
NewCrawlConfig returns a config in crawling mode.
func NewStandardConfig ¶
func NewStandardConfig() *Config
NewStandardConfig returns a Config pointer with default values.
type DHT ¶
DHT represents a DHT node.
func New ¶
New returns a DHT pointer. If config is nil, then config will be set to the default config.
func (*DHT) IsCrawlMode ¶
IsCrawlMode returns whether mode is CrawlMode.
func (*DHT) IsStandardMode ¶
IsStandardMode returns whether mode is StandardMode.
type Peer ¶
Peer represents a peer contact.
func (*Peer) CompactIPPortInfo ¶
CompactIPPortInfo returns "Compact node info". See http://www.bittorrent.org/beps/bep_0005.html.
type Wire ¶
type Wire struct {
// contains filtered or unexported fields
}
Wire represents the wire protocol.
func NewWire ¶
NewWire returns a Wire pointer.
- blackListSize: the blacklist size
- requestQueueSize: the max requests it can buffers
- workerQueueSize: the max goroutine downloading workers
Source Files
Directories