sounds like its possible getting corrupted along the way since it doesn't grab data it just dl's the file its even more weird, i'll look into it but its definitely a issue caused by your server and setup.

Ok, well I dunno if anyone is really interested in an rss mod or not and I probably don't have the time to make a full mod anyway so below is some example code from the btjunkie rss crawler I wrote. I decided to use straight up string manipulation instead of preg_match or anything related (I find string manipulation easier to code and faster to be executed for this type of thing). The download part of the code is commented out because you should run it once first without it downloading just to see how fast it is (gets the info from all 8 main rss feeds in mere seconds). Anyhow, the code is pretty straight forward and you can probably see how to use similar code for other sites but remember this IS site specific code so it's not gonna work if you plug a mininova url into it or something.




Welp, have fun.

edit: Eventually (gonna be a bit) ill release more RSS crawling code for other sites.

This post has been edited by juiced: May 10 2007, 08:42 PM

i'll look into that a little more jaggi, thanks for the tip.

and very nice script juiced, that was cool to see the screen fill up like that! small question though, it looks like it needs to be constructed into a torrent and inserted into the database, right?

This post has been edited by phrostwave: May 10 2007, 08:46 PM

Yea, if you uncomment the download code it'll download the torrents to the directory you run the script from (you can write code to put it in whatever directory you want though). It does although need the code for getting it dropped into the database and whatnot.

major props for that, this is going to be a beast to work with, because its a lot faster than ibitzy (and configurable) at this point, we'll have to see how it handles once it starts inserting stuff into the db....

looks like im gonna be writing a nice new function!

Thx dude. Eventually when i'm done coding my crawlers (not really sure when tha'll be at this point) it'll work so that the RSS crawlers are running periodically throughout each day (over top of the full on crawler) so that new torrents submitted around the net will be easily indexed. The goal is to make it so that any new torrent submitted to other sites can be quickly indexed close to the time it was actually submitted. My full on crawler (which is what's taking the most time) will also be more basic string manipulation code so that it can be kept clean, simple, and fast. It'll have a good visual interface to work with too.

awesome

right now im looking at torrenttraders import.php, just to get an idea on how to properly write this function (if i can.....oh boy). i mean its a pretty simple process to do, but its a lot of lines to write, and a lot of ways to do it (do like ibitzy and have an 'unprocessed_rss' row?). i guess it would depend on how it functions with the least amount of stress on your server and the server youre grabbing stuff from - i know they wouldnt enjoy a 1 minute DOS from having a ton of crawlers ripping up their feeds.

Thinking ahead for the full crawlers behavior...

Well, what's going to bog down servers the most is the actual downloading and transferring of torrents from some web site to another (i.e your own site). So, I was thinking that the crawler will run multiple instances of itself at a time (within reason) so it'll be crawling a variety of sites at the same time. When the crawler gets info about a torrent and it's download location, that info can be sent to a download queue that will alternate downloads depending on how long it's been since the last download from the particular site the torrent is from. So, for example if the crawler just downloaded some torrents from site#1 and at the same time is getting info about site#2, 3, 4, 5 etc it will get downloads from sites#2, 3, 4, 5 etc before downloading again from site#1. There are a variety of ways it could be handled I suppose. I'll think of something cool.

This post has been edited by juiced: May 10 2007, 09:56 PM

btw: if you have managed to make it all the way through this thread, this release does still contain the same sql injection bug as the normal version, the only altercation i've done is remove the (mysql_error) statement, to at least hide the problem for now (hiding the problem doesnt solve it, remember that!)

something i worked on for a few hours this morning: a fade in dhtml pop up!

see exactly what im talking about in this little example, simply click 'view metadata'!

im working on integrating this into the comments, as well as a little summary of the details for the index.php

any good??

very good thank you.
i think that you must create a site or blog for developping all those scripts...

few things, you should make it only clickable once as if you click it again then the script runs again which is pointless. Also you should make it open more closer to the link cus if you scroll down and click it you miss the box.

Neato burrito.

damn...it seems that when you fix one thing you just make another issue (we've all been there and done that).

so i get the popup to at least center itself on the screen, which cleared the 'can you see me?' problem, but i went a set further and added a toggling-like ability to it, so when you scroll down it follow you. heres where the issue arises: try to scroll to the 'bottom' of the page; it just doesnt happen!

so my question is this: should i even include the toggle ability? it centers just fine without it.

also, im still working on making it autoclose if you click the link again when its open - if anyone can help me out with this i will post the script half of the function so you can have a go at it.

This post has been edited by phrostwave: May 17 2007, 08:45 AM

couple of updates on the ak nova:

removed the toggling from the metadata popup, seems kinda useless and more for show - and have instead modified the (insert cuss word) out of the login area, giving it the toggle-like ability that the popup had. and even on-top of that, i have integrated a drop down login area!

ive been doing some work to ibitzy to make it a little more friendly with t-xores db scheme (multiple values such as 'submitter' for ibitzy and 'posted_id + posted_user' for t-xore), as well as working like a madman to solve this utterly incurable problem with the subcategories & submitter in ibitzy (if ftorrent can do it, i can too ).

last thing i need to say is im abandoning the current tracker, and am going to try to focus on integrating defiant v2.? (cant remember the version, its the most current) with t-xore. im hacking away at removing defiants user/torrent system (flat files) - and at the same time integrating t-xores login system, ban feature, categories, and something else i cant seem to remember...

any suggestions or ideas? im trying to make this as much a group project as possible, as they say the more the merrier!