***UPDATE***
Ibitzy 0.2 Beta NOW OUT.

1 major change, halved the sql load for processing torrents. (dl link updated)

Well as promised myself and jaggi have been working over the past couple of weeks on a FREE hughly configurable web spider and here it is: IBITZY

I have been running it for just over 3 days now and it is indexing roughly 2000 UNIQUE torrents a day.

Features:

• Highly configuable, all sql feilds can bee edited within the config file, along with many internal config options
• Loose categorization of spidered torrents (it will always try to categorize a torrent baised on the category of the site it was reteved from)
• Designed to run silently with no output (for running from crontab)
• Standized pluggin design (means you can add new sites to the system just by copying the existing pluggins)
• Release version supports 5 different sites (meganova, mininova, piratebay, torrentportal, torrentspy)
• Comes with its own stats aggreator

you can see a demo of Ibitzy in action by visiting the demo site http://ibitzy.wooptydoo.com/ (NOTE: NONE of the torrents can bee downloaded from that site)

And the stats page for the demo site http://ibitzy.wooptydoo.com/stats.php

BASIC setup instrusions are included though a little php knlodge is recommended to operate this script

Ibitzy support site: http://www.ipbtracker.eu/index.php?showforum=26

for the download visit: http://www.ipbtracker.eu/index.php?showtopic=182 (Updated link)

Heres a FAQ, its a work in progress so expect lots of updates as questions arise.

FAQ

Q: What is IBITZY?
A: IBITZY is a torrent site indexer that will go to any torrent site that has a working pluggin in the script and gather their torrent data and torrent files and download them into a database for use by you however you see fit.

Q: Sound good but whats the point of that?
A: Well with the data gathered people will be able to make their own front end enabling them to setup their own torrent site or to intergrate it into one of their existing torrent sites.

Q: Will this gather peer data too?
A: Unfortunetly not, this script is not designed to do that but it can be done using the data in the torrents database and a torrent scraper.

Q: I had 200 torrents in un_processed table but only 150 made it through to torrents table, why?
A: This is because some torrents aren't inserted into the torrents table because they don't pass the final checking, this could be missing fields or that they already exist and so this is to stop dupe checking.

Q: Where is the torrents download page? how am i meant to download torrents?!?!
A: IBITZY is designed to be used with a front end and therefore one must be installed before you can get access to any features that you used to on normal torrent sites.

Q: I already have a torrent site, do i have to start all over again?
A: Nope its very simple to use IBITZY with any existing torrent site, simple edit the configuration file torrents table to match the fields and name of your torrents table.

Q: I have different categories on my site, can i modify IBITZY category system to match?
A: Yes, this is a big feature and spent lots of time on perfecting this, just change the category id to match the id of your.

Q: I want to index a site not part of the script, how can i do this?
A: IBITZY uses a pluggin system, this means you can make a file based on an existing pluggin and simply upload it to the pluggin system and add it to the index pluggins array and the script will automatically begin to index that site.

Q: Does IBITZY guard against dupes?
A: Yes we have a few things to stop dupes in the DB including locking the script while its running and checking against info hashes and keeping a cache time of when a site was last indexed.

Q: How do i stop indexing from a certain site?
A: Well the easiest way would be to comment out the line in the pluggin array linked to the pluggin you want to stop indexing, the pluggin array is in the index file.

This post has been edited by Jaggi: Sep 15 2006, 02:02 AM

Looks great! I like the layout

Nice job guys! My windows crawler will be coming out any day now as well (would've been sooner but I broke my leg heh). Current tests are showing it can pull in about 41,000 - 47,000 torrents a day on a DSL connection (A LOT more if you run multiple copies of the program at once of course)! Anyhow, thanks again for this, your code gave me some good ideas.

Also, if you have problems with sites limiting how many torrents you can download a day thus making the spider not download as many torrents per day as it could, I can help out with some code that will get around it

EDIT: Friggin nice job on the stats page btw

This post has been edited by Big Pox: Sep 15 2006, 05:00 PM

Well ibitzy's therotecal daily limit is 57'600 torrents a day assuming everything is running at peek efficiency (server having little/no overhead, external sites connecting fast etc), the 2000 a day is roughly everything thats posted on the 5 currently supported sites (with the exemption of tspy scince they do seem to have a 100 torrent/day dl limit).

We are planning to impliment proxy support for later versions if you have idea's to bypass the daily limit without going through a proxy i love to hear from ya

Well, if it'll have proxy support i'd bet using the Tor network would solve any limit problems (tor can't work on shared hosting though). Without Tor, proxy support would work just fine too of course. Sometimes sites don't limit based on IP, but rather browser info (cookies sometimes) so, you could also write some code to mimic the browser info sent to sites. This way ibitzy could download a torrent with Internet Explorer browser info, then the next torrent it downloads could tell the sites it's downloading on a fresh connection with Mozilla and etc etc etc. It's very easy to write a program that can do this in C# so im hoping it won't be too hard for me to get this pounded out in php.

edit: I don't think any of the sites ibitzy currently works with will limit downloads except for torrentspy (and i don't know about torrentportal). It probably actually won't even be a problem heh, but who knows.

This post has been edited by Big Pox: Sep 15 2006, 07:17 PM

yer the tor idea sounds good though as you said it would limit ibitzy's usage somewot so ill have to think on that

as for browser headers thats pretty simple in php (infact ibitzy already dose that for tpb (to send the language cookie)).

good

nice!
so can this be integrated with ipbtracker and the invision forum or is it completely separate?

it can bee intergrated with anything designed to receive external torrents (ipbtracker is designed for internal torrents)

It can be intergrated with IPB but as a seperate mod, as said IPBTracker is a internal tracker system. If you wanna have a go the torrents page mod on ipbtracker would be a start.

do you have to have btsource installed also? cause i installed Ibitzy on my computer and then its just a blank page after i put debug off and when debug is off is says "The present file is currently locked" and stuff so please help me

because ibitzy is designed to run from cron when debug mode is disabled there will bee NO OUTPUT to the screen, so that is perfictly normal, as for the file locked message when turning debug on that is most likly because the previous run (with debug disabled) hasnt finished running yet (to provent dupes ibitzy locks the script while its running and unlocks it when its finished)

o okay thanks man, and how do i ut btsource with it or does it not need it?

BTsource as it is now is a TRACKER frountend what you need for ibitzy is softwere that deals with external torrents (such as phpmybittorrent). the "btsource" you see in the demo is an edited version i setup simply to demo what ibitzy can do.

As i said earlier you can easily intergrate ibitzy into ANY softwere designed to deal with EXTERNAL torrents