DYNAMIC. ENEMY. STORM. CROOVE!

http://technika.integriti.net/

for the tracker, scoring and general stats for the Australian, singaporean and phillo servers (that i know of atm) come to this site =]. Credits: Ponyoooo


got something to say/ask/request? direct it here =]

linked back to this forum on the menu bar from tracker, might as well get some pagerank on this site

I am also trying to make my own tracker. This time I will try in PHP. I tried with Python and got it to work on Google App Engin, but it took me a lot of time to code.

How did you fetch the scores? By crawling and screen-scraping or got them from some other data source?

I'm thinking about using this to scrape the site: http://thejakemarsh.com/1982/

i use php. compared to that site, my method is 'similar' but not really. and yeah i just open every page sequentially until i hit a flag i set that signals to stop.

i use fopen and fgets to parse the site, because there are a lot of irrelevant information in the pages, and since im parsing so many, its vital that i break out of reading a file AS SOON AS POSSIBLE. using the command $raw = file_get_contents($url); is very bad for this purpose, as it will download the whole thing first, then parse later. also this will present EXTREME problems later on, when people have equal rankings (such as 2 people being places 1st for having equal scores). one piece of info may be lost, and chances are that rank will become corrupted. this also presents problems in the actual coding, as you will not be able to sequentially alter your variables while the page is being loaded, but instead be forced to use preg_match_all to parse your information, which is extremely painful. i tried it. didnt like it.

literally, i use around 10 lines in total to get all the information i need for each type of page and store it into a 3-d array that holds all the information for that type (by type, i mean dj title info, or weekly scores, or TP scores, or PP scores). also, when you consider that each part of information may or may not be slightly different in format than the last (especially true when looking at popular play), then preg_match_all tends to act awkwardly.

overall im parsing around 400 pages every hour on the hour, takes around 3 minutes due to optimizations i did a while ago, normally i would have taken 5-6 minutes but the new code i did really quickened it up a lot

i also use mysql for database purposes

this guy knows what he's talking about aye!

How do you make it run every hour?

Do you use cronjob (with PHP CLI), webcron (that hits a webserver once a while) or fakecron (that checks the time elapsed every page load)?

I'm asking because I don't know how to set cronjobs on my web hosting and I can't decide what other ways to use.

yeah i use cronjobs

i set up a page which just does include() a whole of other pages, all with subtasks to update my databases one section at a time

if you dont know how to set it up with your host, then try connecting via shell/telnet or something and setting it up through there
if not, then .. no clue, really

just a curious couple of questions.. does it all have to be done on one computer.. or is it all done on some server online or... yeah.. i'm just curious cuz i dunno ^_^. can you save all your work on a usb? Why did you want to work with us? What is your biggest asset? What's your biggest weakness? do you like lobster at all?

all hosted online on my webserver, everythings done there now
my work is like, what, 50kb total of coding, most of it copypasta of previous code because its all so similar
because youre awesome
wouldnt you like to know
hexad tp
hells yeah

Oh, so you host them on your own server? My Desktop PC is a server too, but it's not always up. By the way it runs Ubuntu Karmic.

But I also use a paid web hosting and thinking about running my tracker there. Just need to find out how to use cron (or at least ssh to that web server).

ah i meant my webhost server, its all on some box somewhere i dont know

Finally I finished my own crawler, but I have to choose between flexibility / readability vs speed.

I chose flexibility / readability, so I made it using phpQuery and cURL, and some queue based class that lets me pause and resume the process at anytime.

I tried running the code, and it took around 12 minutes. x_x

At last the code was... well.... a bit clean.

i went for speed, thats how im used to programming

also because of the fact that the database is actually growing, and the number of pages will be rising every day (it went up from like 150 to 400 last month, and im expecting exponential growth) to therefore speed will be playing a factor

then again, the website itself actually doesnt even update hourly, so im not even sure why i myself update hourly.

actually thats a good point, i really should just find out exactly when they update and sync with them ..

Their website updates as soon as new data is available, I guess. Because I can check my new rank after playing that song right away.

Edit: I reran the script... It took about 10 minutes and it fetches about 725 pages. So, maybe the speed depends on the mood of the server and the internet connection.

It's true that the data is updated almost immediately. Dango usually use his psp to check online immediately after his play to check the ranking and it reflects his score on the site.

hmm really now .. well thats good then i guess, i dont think CONSTANT updating on our sides would be best way to go, so i guess hourly/timed updates would be the way to go, unless youre doing constant live updates for maybe the top page for all sets, which wouldnt be too bad. otherwise youre going to have literally thousands of page hits per hour all day, and they might think something is up with that ..

and yeah 1page/sec is pretty much a decent speed, im finding it to be more like 1.2/page though

hey you, your area 7 pp says 288 max combo, but ppl are getting 290. just letting you know broskies.

Looks like my web hosting doesn't support cron jobs.

So I'm using webcrons! Powered by Google App Engine.

can you track down percentage?

What are percentages?

percentage MAX/COOL/GOOD/MISS/BREAK? That's not possible from what I see on the site.

i mean EXP percentage?

nice work bro - you got the missions up on your site.

ran00 wrote:

i mean EXP percentage?

It might require a password to access the data, but I can't be too sure, I thought Mission data needed a pass to be accessed

Is it just me or is the tracker not updating scores? (10:28PM PH time)