Filesoup
X   Site Message
(Message will auto close in 2 seconds)


> 

If you would like to advertise your Hosting, Services or Products on Filesoup, please contact Geeker for a quote.

 
Reply to this topicStart new topic
> Juiced's RSS crawler minipack v 0.0.1
juiced
post May 19 2007, 07:06 AM
Post #1


Vegetable
Group Icon

Group: Member
Posts: 38
Joined: 7-May 07
Member No.: 452,851



Juiced's RSS Crawler mini-pack version 0.0.1

What does it do?
-Crawls the rss feeds for mininova, meganova, torrentspy, btjunkie, thepiratebay and downloads found torrents.

What does it not do?
-Drop info into DB
-Get torrent metadata

Ok, so here is the first mini-pack release of my rss crawler project. Keep in mind this is version 0.0.1 so don't expect it to be perfect and do everything you could ever want. It's just the basic code for the beginning of this project. The meganova cralwer still picks up a bit of excess crap. You will have to alter the $start variable and the while statement based on your needs to determine what specific RSS feeds will be crawled so I want no lame comments about how it's not crawling all the feeds or some nonsense because you decide.

Also, if anyone feels like writing the code to have the torrents drop into a DB, get scrape/metadata info (maybe have it integrated into t-xore or something) by all means go ahead. Do whatever you want and do a re-release, all I ask is you keep and do not alter the project info message at the top of each of the php pages.

Future releases will include crawlers for more sites, more advanced features, and an easy to use interface.

Attached File  JuicedsRSSCrawlerminipack.zip ( 3.64k ) Number of downloads: 16


Go to the top of the pageReport Post
+
phrostwave
post May 19 2007, 07:11 AM
Post #2


DJ Taktikz / STR Records
Group Icon

Group: Member
Posts: 52
Joined: 7-September 06
Member No.: 411,795

System Specs: (show/hide)




excellent work juiced, this will definately be a project i'm gonna start working on asap! Wink.gif

This post has been edited by phrostwave: May 19 2007, 07:16 AM


--------------------
may 15, 2007 releases - ak nova 1.4 (snapshot), ak tracker 1.0.1 (snapshot).
t-xore 0.4 preconfigured with ibitzy, registration captcha (ive seen it on 18 sites so far!), admin search, user search, and tons of other stuff by me and various other filesoup members!

bNova stylesheet for ak nova 1.4


look for ak nova 1.7 soon! lots of features like forums, dhtml/ajax, two mass scrapes, nicer admin area, plus a ton of other stuff!!
Go to the top of the pageReport Post
+
juiced
post May 19 2007, 08:00 AM
Post #3


Vegetable
Group Icon

Group: Member
Posts: 38
Joined: 7-May 07
Member No.: 452,851



Cool, good to hear ThumbsUp.gif
Go to the top of the pageReport Post
+
SumoTorrent
post May 19 2007, 08:55 AM
Post #4


Vegetable
Group Icon

Group: BT Community Leader
Posts: 42
Joined: 26-February 07
Member No.: 447,369
 


Interesting. I coded myself sor SUMO a generic RSS fetcher (not crawler as it actually crawls nothing, it just get the URL for the enclosed content or to the details page), then I have a bunch of crawlers multi threaded that extract the torrents from various sources on the web.
Go to the top of the pageReport Post
+
juiced
post May 19 2007, 11:34 AM
Post #5


Vegetable
Group Icon

Group: Member
Posts: 38
Joined: 7-May 07
Member No.: 452,851



Some of the download code is messed up at the moment. This line of code
CODE
$filelink = file_get_contents("$actualLink/download.torrent");
must be altered for each website depending on how the site is set up... I'll put a fixed version up tomorrow sometime. I totally spaced this somehow hah.

EDIT: Haven't been able to finish them all but here is the fixed mininova rss crawler (download code corrected).

Attached File  mininova.php ( 1.47k ) Number of downloads: 6




This post has been edited by juiced: May 19 2007, 09:33 PM
Go to the top of the pageReport Post
+

Fast ReplyReply to this topicStart new topic
1 User(s) are reading this topic (0 Guests and 0 Anonymous Users)
1 Members: OPPZeroCool

 




Fast Loans | Loans | Online Loan | Charity | Final Fantasy 13
-
Lo-Fi Version 0.1881 sec    --    13 queries    GZIP Enabled
Time is now: 21st May 2007 - 05:34 PM