General DiscussionI wish there is a way to find a solution. Please, find the solution.

to be honest, plus service was a joke. dotabuff got data from valve without paying a dime. it was all fun and games until dotabuff started charging people for additional stats and shit.

there won't be a solution and players will have the right to hide their game history from third party websites from now on.

Valve isn't going to reverse their decision, regardless if Dotabuff reverts or not. It sucks that Valve cut off their data-gathering method without notice. But Valve is basically saying "Switch to the API, or get nothing".

I was just wondering, what is the API?

I do not understand what the API is?

From what I understand the API is just a file that stats are recorded in (I'm thinking realtime, but I'm speculating).

This situation isn't entirely bad. If valve shuts down 3rd party stats, I would bet money they will add stats themselves. 1st party stats>3rd party stats.

@Infinite Regen
'to be honest, plus service was a joke. dotabuff got data from valve without paying a dime. it was all fun and games until dotabuff started charging people for additional stats and shit.'

'From what I understand the API is just a file that stats are recorded in (I'm thinking realtime, but I'm speculating). '

You are the usual clueless fuck. DotaBuff didn't get anything without paying a dime, Valve PROVIDED the data for free and publicly. What DotaBuff did was build a site that sorts and presents that data and serves 750.000 unique visitors each day. By your second comment I see that it's utterly useless to try and explain to you how hard (and expensive) that is to accomplish.

DotaBuff charge for a subset of functions on their site, NOT FOR THE FUCKING DATA. The data is out there, publicly available, so why are you even here complaining. Get your own data and do your own stats, nothing stopping you.

@jams "Valve PROVIDED the data for free and publicly"
IF you use the WebAPI (API = application programming interface) and not forbidden methods like reverse engineering.

Dotabuff was downloading replays and this cannot be called "reverse engineering".

@jams
Yeah, the WebAPI is the free and public data. Parsing Replays for player information is NOT the free and public way of getting that. VALVe didn't want it and put a stop to it. Well within their rights to.

@jams

Don't flame the wrong person, he did not post it.

@CM

It's still possible for players to parse replays on their own. Bruno the Statsman JUST released a replay parser for that exact purpose. So parsing replays is still free and public.

Dotabuff was clearly using an unsupported method of gathering data and they knew it. There was never any guarantee that their unsupported method would stay the same and remain working for any amount of time. That's why you use the API, it's designed to stay working, and remain backward compatible as Valve changes Dota.

I understand the API has been down in the past and unstable, but that's changed now. It's up and ready for business. Everyone needs to use it.

what if dotabuff sets up some computers that manually download the replays thru the client, then pass it to their parser like normal

yep, dotabuff can charge players to parse their own data and facilitate the gathering and upstream DB process. Crowd sourcing is way better than anything else.

The fact is that it's not really simple to provide us a tiny program using our SteamAPI key and collecting data for Dotabuff...

maybe P2P and meshed stat network is the solution. We are the server, not DB anymore ...

@CMMiller89
Actually the replay parser was provided by Valve and they said "looking forward to seeing what the community will do with it". And the match result data was public data, the method of collecting it is irrelevant. For all we know DotaBuff could have had 1000 kids that manually copied data from the screen with the client - the data is PUBLIC!

@Roger Feederer
You would use the WebAPI if it was working. But it wasn't and it was clearly stated by Valve that it isn't their priority. So the community had the choice of not making a site of any kind, or scraping data from the client. So they scraped... The removal of that method is not the problem, the real problem is that now that they have reinstated the WebAPI they removed some very important components - the replay seed which is necessary in order to obtain the replay link and the account id of most players. Without replay parsing (and we already established that parsing was encouraged by Valve) and with anonymous players much of DotaBuffs functionality is gone.

Bottom line is, Valve forced the community to using alternative methods for data gathering. While removing that access and providing the WebAPI one is Valve's right, the way they handled it without any prior notice was wrong. They could have given the community reasonable time, for example they could have shut down the client scraping 2 weeks after bringing the WebAPI online and giving notice to the community sites. And while the argument for and against the privacy setting is a long one with (somewhat) valid points on both sides, outright removing the replay access just to facilitate enforcement of that rule is a very negative move towards the community. As I said before - for a company that has built most of their success on the community, Valve better think deeper before alienating it.

well stated jams

does anyone understand the part in dotabuffs announcement:

"They have removed the crucial part which makes up the replay url. Replay downloading and parsing was one of our biggest features (both existing and upcoming), which we spent months developing."

my understanding of this is replays can still be downloaded just not thru a url?

so the only way to get replays now is to download them thru the dota 2 client.

so theoretically, someone could sit there and download every replay 1 by 1 as they are played and send them to dotabuff, then dotabuff could keep parsing them and functioning.

so really, the only thing that would need to be done is set up some bots that automate this process within dota 2 and dump them to dotabuff for parsing.

then dotabuff can keep doing their replay parsing thing as before?

by bots i mean you can set up macros to just move your mouse in repetitive patterns or increment the replay # by 1 each time and click download again and again... this isnt hard, wouldnt it work?

i believe the other issue is that playerid are not being included in the replays now?

this is a problem if, for example, there are 10 different players using the same username.

it seems like a difficult thing to get around..

It is possible but very slow, unreliable and can be easily restricted by Valve. For example, 100 replays per day limit is sufficient for anyone but not dotabuff.

Don't forget that Dota 2 recently hit 100.000.000 games played. Yes that is 100 MILLION! You can't build anything serious on manually working with that kind of amount of data.

The thing that was removed was the so called "replay_salt" which is basically a secret number that is part of every replay url. So the replays are still out there downloadable through http, but the right url is not known any more. And the worst thing is they didn't remove it because they don't agree with replay parsing, but rather because the publicly available replays are a loophole to their new privacy settings (player Steam ID's are available in the parsed data even for players with privacy on).

assuming about 50000 players online at a time
=> 50000/10 = 5000 games being played at once
assuming avg game interval = 60 min
=> 5000/60 = 80 games played per minute
assuming 1/4 min (15 sec) to download a replay
=> 80/4 = 20 : number of computers you'd need to have downloading replays simultaneously to download replays as fast as new games are played

well i dont know if this is a good estimate but i assume you could also have multiple instances of dota 2 open within multiple copies of windows , and get htis done with about 4 computers

Well I'm not gonna comment of all that is wrong in you post, just don't assume random shit. But I will say that current peak on Dota 2 is around 260.000 people online.

well based on the 100,000,000 games played so far, i think the number of games played per minute is more like 100-150, not 80

games/ time

After a view games without my dotabuff analysis afterwards I can say that Dota lost some of its interest for me. Sad :(

How did you ever enjoy DotA before DotA2 and dotabuff Kirk!? It must have been brutal!

I didn't say I don't enjoy it. What I said was that it made it less pleasurable for me. I still very much dig the game, just not as much as I used to

Hey Kirk do you really believe DBR was representative of where you are at in skill level in Dota 2?

Because I can tell you without a shadow of a doubt it's not even close. There are too many variables in Dota for DB to have a system in place that is representative. One of the reasons although not the main reason that Valve is not making MMR public is because they don't actually have one which they deem finely tuned enough. It's constantly evolving and changing.

The other thing is you just admitted that stats have impacted on you negatively. So you are one of the examples Valve talks about wanting to protect. You can't deal with labels/numbers to represent your standing.

uve never played in DL "drunken dota MASTER" ?

Help Dotabuf

https://www.change.org/petitions/dota-2-community-revert-past-actions-concering-data-and-third-parties
Already almost 150 signatures!

Why would we want to help dotabuff? There is a completely legit way for them to keep their site useful and functioning but they decided that they are going to be 10 years old and take their ball and go home. Im sure there is a new site in the works already by someone willing to use supported methods to achieve the goal we all want.

@ Mr Affluenza:

Nooo I really don't care that much about the DBR. I was all about the item statistics, machups, stalking my former enemies to see how good they were etc. For me as a noob the item statistics or hero statistics were pure gold and I learnt a lot from it.

@Mr Affluenza
Yes I believe the hidden Valve's MMR (and to a somewhat lesser extent DBR) are a pretty good representation of skill level. I can definitely feel it in my games. Example I'm on the border of high/very high, I'm right at home at the lower tier, but often over my head in very high. I know when people are better than me and when they are worse than me and I can compare skill with my friends and I know their rating is a close (relative) representation of their skill. And mind you - Dota skill is foremost about winning game, game awareness and sense is maybe even more important than clickety skill.

Funny thing is I was ~1700 MMR in hon (I know I will get flamed for this) and my dbr at time of publishing was 1648 (I increased it since as my win ratio past few days is higher than 50%). Considering how old I am, I'm probably not going to improve much with time - so it's quite amazing how 2 completely different system rated my skill the same.

PS.. Oh and btw I dont care for DBR much either, the current private implementation is fine as it's only good to check own progress and compare skill with friends. I much more care about the other nicely presented stats, again mostly mine and friends. I also like the ability when talking with friends about a game, to just post a link with all relevant data...