Filesoup
X   Site Message
(Message will auto close in 2 seconds)


> 

If you would like to advertise your Hosting, Services or Products on Filesoup, please contact Geeker for a quote.

 
 Reply to this topicStart new topic
> Pro RSS crawlers?, are there any?
juiced
post May 7 2007, 05:12 PM
Post #1


Chicken Noodle
Group Icon

Group: Member
Posts: 17
Joined: 7-May 07
Member No.: 452,851
Hi, new here and trying to get a good indexer started and just wondering if there were any pro RSS crawlers around that will drop info into a db, download torrents, run automatically at specified intervals (to keep on getting new torrents) etc? I coded out a few of the crawlers for a couple of the larger indexer sites and since not everyone has the same rss format I need site specific code so figured i'd try and find out if someone already made something similar before I ended up making agagillion other crawlers with site specific code. All I have done is crawlers for btjunkie, and mega/mini nova thus far sad.gif Thanks for any help.
Go to the top of the pageReport Post
+
Jaggi
post May 7 2007, 05:29 PM
Post #2


Spam Soup
Group Icon

Group: BT Community Leader
Posts: 789
Joined: 11-August 04
Member No.: 170,409
 


you can use ibitzy although it crawls the site not the rss but you could modify all that using the pluggin system, it does all the other things u mentioned.


--------------------
CODE9 ADMIN
Check out Code9-Group.org for the best stuff around, NOW running my IPBTracker (only IPB forum with a passkey system), membership now open.

IPBTracker (Invision 2.1.* intergrated forum tracker using XBT) going final in nxt release, IPBTracker 2.0 begins development for IPB 2.2 soon.

Check out IBITZY at my IPBTracker site, the best torrent spider around.
Go to the top of the pageReport Post
+
juiced
post May 7 2007, 05:37 PM
Post #3


Chicken Noodle
Group Icon

Group: Member
Posts: 17
Joined: 7-May 07
Member No.: 452,851
Does ibitzy obtain the category (i.e music, video, games etc) of individual torrents as it's crawling?

edit: nvm, i just installed it, I see how it works kinda. Thx

This post has been edited by juiced: May 7 2007, 06:19 PM
Go to the top of the pageReport Post
+
expert01
post May 7 2007, 08:44 PM
Post #4


Spam Soup
Group Icon

Group: Member
Posts: 140
Joined: 27-April 05
Member No.: 272,613
I've been working on a new Bittorrent XML specification. for a few months. Once I complete it and have sample code, I'll try to get other sites to support it (and get it integrated into most of the tracker sources).


--------------------
http://torrentbits.sourceforge.net/
Links to BitTorrent sites and tracker sources, install guides, support forums, RSS feeds, and more!
Go to the top of the pageReport Post
+
juiced
post May 7 2007, 08:58 PM
Post #5


Chicken Noodle
Group Icon

Group: Member
Posts: 17
Joined: 7-May 07
Member No.: 452,851
QUOTE(expert01 @ May 7 2007, 06:44 PM) *
I've been working on a new Bittorrent XML specification. for a few months. Once I complete it and have sample code, I'll try to get other sites to support it (and get it integrated into most of the tracker sources).


That'd be a relief.
Go to the top of the pageReport Post
+
Jaggi
post May 7 2007, 09:43 PM
Post #6


Spam Soup
Group Icon

Group: BT Community Leader
Posts: 789
Joined: 11-August 04
Member No.: 170,409
 


yes it does and i'd like to see the code when your done expert, i would also like if you opened it up for some suggestions as a xml spec for bittorrent is a pretty big thing.
Go to the top of the pageReport Post
+
SqrtBoy
post May 9 2007, 05:49 PM
Post #7


Spam Soup
Group Icon

Group: BT Community Leader
Posts: 1,157
Joined: 30-July 04
Member No.: 160,009
 


Here is a RSS specification I wrote some time ago for bittorrent:
http://borget.info/?page_id=72

It is compatible with what has been commonly adopted

To write a RSS Crawler, I would recommend you to use curl + magPie in php.


--------------------
Need a coder ? Check http://www.borget.info/
Go to the top of the pageReport Post
+
Jaggi
post May 9 2007, 08:46 PM
Post #8


Spam Soup
Group Icon

Group: BT Community Leader
Posts: 789
Joined: 11-August 04
Member No.: 170,409
 


i don't get the use of live stats in your rss sqrt? they'd always be out of data and alot of if not most rss's are cached.
Go to the top of the pageReport Post
+
horoxxx
post Yesterday, 02:15 PM
Post #9


Clam Chowder
Group Icon

Group: Member
Posts: 95
Joined: 24-May 05
Member No.: 284,605
torrentz.ws is based on rss, but its just 1/10 of whole indexer, it tooked me about 3-4 months to finish it, but it works great now


--------------------
BitDig.com | torrentz.ws
Go to the top of the pageReport Post
+
proxykill
post Yesterday, 03:35 PM
Post #10


PK-Torrents
Group Icon

Group: BT Community Leader
Posts: 263
Joined: 28-October 06
From: Isle Of Wight,UK
Member No.: 422,304
 
System Specs: (show/hide)
Thats a nice site, is there any chance your release it?. it would be a nice contribution to the bit-torrent community.


--------------------
Torrent Listers/Crawlers/Trackers/Clients Downloads Get everything in one place

PHP Photo album by me coded for my brother with a p0rnrip.net theme because thats his favorit site. ;~P

PK-Torrents PKTv0.1beta - DEMO

Go to the top of the pageReport Post
+
juiced
post Yesterday, 04:02 PM
Post #11


Chicken Noodle
Group Icon

Group: Member
Posts: 17
Joined: 7-May 07
Member No.: 452,851
QUOTE(SqrtBoy @ May 9 2007, 03:49 PM) *
To write a RSS Crawler, I would recommend you to use curl + magPie in php.


Why? Seems so easy with just php.
Go to the top of the pageReport Post
+
phrostwave
post Yesterday, 09:16 PM
Post #12


AK Nova
Group Icon

Group: Member
Posts: 31
Joined: 7-September 06
Member No.: 411,795

System Specs: (show/hide)
it looks like you used some of scuttles backend, very nice =D


--------------------
may 15, 2007 - ak nova 1.4 sourcecode.
t-xore 0.4 preconfigured with ibitzy and lots of other stuff!

dj taktikz - str records
Go to the top of the pageReport Post
+
horoxxx
post Today, 09:18 AM
Post #13


Clam Chowder
Group Icon

Group: Member
Posts: 95
Joined: 24-May 05
Member No.: 284,605
QUOTE(phrostwave @ May 14 2007, 07:16 PM) *
it looks like you used some of scuttles backend, very nice =D


template only smile.gif
Go to the top of the pageReport Post
+
« Next Oldest · BitTorrent Tracker Software · Next Newest »
 

Fast ReplyReply to this topicStart new topic
1 User(s) are reading this topic (0 Guests and 0 Anonymous Users)
1 Members: OPPZeroCool

 

Display Mode: Standard · Switch to: Linear+ · Switch to: Outline

Track this topic · Email this topic · Print this topic · Subscribe to this forum


Problem Remortgage | Personal Loans Online | Best MySpace Layouts | Spanish Bible | Nissan Armada Forum
-
Lo-Fi Version 0.3345 sec    --    12 queries    GZIP Enabled
Time is now: 15th May 2007 - 03:12 PM
Powered By IP.Board © 2007  IPS, Inc.