PECOTA

Message Bookmarked

Bookmark Removed

Is there any reasonable method of measuring the accuracy of PECOTA? Like a hold-out sample? r-squared? cross-validation?

It seems like PECOTA is akin to the weatherperson trade, or is there some background accountability that I'm not aware of?

― gygax! (gygax!), Wednesday, 16 February 2005 20:47 (nineteen years ago) link

okay i googled:

http://www.baseballprospectus.com/article.php?articleid=2515

correlation coefficient, mean error and r-squared.

― gygax! (gygax!), Wednesday, 16 February 2005 20:54 (nineteen years ago) link

I just skimmed that article, but it's -- ahem -- intruiging that PECOTA turns out to be the best method according to BP's math.

When Diamond Mind makes those sorts of comparisons, they typically end up on top (of course, they're not comparing the same things).

― MindInRewind (Barry Bruner), Wednesday, 16 February 2005 21:08 (nineteen years ago) link

Well, since they adjust it from year to year to improve the forecasting, why wouldn't it be the best by their measure? (I know I'm asking for it saying this to a science guy...)

― Dr Morbius (Dr Morbius), Wednesday, 16 February 2005 21:23 (nineteen years ago) link

I just skimmed that article, but it's -- ahem -- intruiging that PECOTA turns out to be the best method according to BP's math.

When Diamond Mind makes those sorts of comparisons, they typically end up on top (of course, they're not comparing the same things).

I thought it was odd that they used a ranking system to judge the results, rather than the actual variance/deviation from actuals.

― gygax! (gygax!), Wednesday, 16 February 2005 21:34 (nineteen years ago) link

yeah, it was obvious that their hitting projections are much stronger than pitching ones

― Jams Murphy (ystrickler), Wednesday, 16 February 2005 21:35 (nineteen years ago) link

Well, since they adjust it from year to year to improve the forecasting, why wouldn't it be the best by their measure?

I'm not questioning the legitimacy of their calculations. But PECOTA ranks themselves against the others by comparing OPS projections, whereas DMB will compare regular season W-L records (IIRC). Both PECOTA and DMB end up on top given the comparisons they're making. They're just comparing different things.

The conclusion is the same in both cases -- "see, we're the best!"

― MindInRewind (Barry Bruner), Wednesday, 16 February 2005 21:46 (nineteen years ago) link

well pecota rightly dismisses w-l records in their projections cuz it's a team -- not an individual -- stat!

― Jams Murphy (ystrickler), Wednesday, 16 February 2005 21:51 (nineteen years ago) link

one month passes...

Nate Silver's PECOTA projections for AL team wins this year on BP site:

Bosox 99
Yanks 95
A's 88
Twins 86
Indians 85
Angels 83

Everyone else is below .500 (Chisox 80). They haven't put up the NL yet.

― Dr Morbius (Dr Morbius), Thursday, 24 March 2005 22:01 (nineteen years ago) link

Indians! obv. is the surprisey one here. In the NL, I predict that the Reds! will be the surprisey one.

― The Obligatory Sourpuss (Begs2Differ), Thursday, 24 March 2005 22:07 (nineteen years ago) link

I predict 71-91 for the Rangers. I've been tempted to call in to a local radio show to say that, just to hear the outrage.

― milozauckerman (miloaukerman), Thursday, 24 March 2005 22:10 (nineteen years ago) link

I definitely would believe the Rangers will backslide some on last year. Too many things went right with their pitching last year that cannot happen again and their starting staff looks just as bad going into the season as last year.

The Indians should be improved. Millwood isn't great but he will fit well with CC and Westbrook in that starting staff and keep that bullpen out of the game.

If Cinci can actually have their first season in a half dozen without half the team getting laid out by the Allstar break, they could actually stick around two weeks longer when their pitching goes south. They have gotten off to a good start each of the past four seasons, then injuries and the terrible starting pitching catches up to them. What would help is being able to field the ball. The Reds again are loaded in the outfield. I'd like to see them deal Sean Casey for some pitching and move the Griff or Dunn to 1b, but it won't happen. They will probably give away Kearns or Pena for nothing in particular like usual.

― Earl Nash (earlnash), Friday, 25 March 2005 01:28 (nineteen years ago) link

PECOTA win projections, NL:

Cardinals 92
Phillies 90
Cubs 89
Giants 85
Padres 84
Dodgers 83
Braves 82
Marlins 81
Mets 81

Caveats -- the Giants' proj is with Bonds getting 464 PA; if he gets 0, SF wins 78.

Cubs' total is with Prior going 170 innings, Wood 155.

― Dr Morbius (Dr Morbius), Wednesday, 30 March 2005 17:37 (nineteen years ago) link

holy crap, the phillies 8 games up in the east? i hope so.

― the leglo (the leglo), Wednesday, 30 March 2005 21:14 (nineteen years ago) link

If the Phils win 90 games I will eat my food.

― gygax! (gygax!), Wednesday, 30 March 2005 21:15 (nineteen years ago) link

Frankly these PECOTA numbers look kind of weird to me. I don't get the Angels 83 Wins at all.

― Alex in SF (Alex in SF), Wednesday, 30 March 2005 21:20 (nineteen years ago) link

PECOTA is conservative ... look at all the teams in the 85-win range. Some of those teams will get lucky and finish four or five games above their predicted worth. In other words, there'll probably be around six 90-win teams rather than four (as predicted by PECOTA).

― MindInRewind (Barry Bruner), Wednesday, 30 March 2005 21:31 (nineteen years ago) link

That makes more sense. So do the wins actually balance out for the entire league?

― Alex in SF (Alex in SF), Wednesday, 30 March 2005 21:37 (nineteen years ago) link

PECOTA is not really meant to be "predictive" in that there will always be teams who surpass and fall short of expectations significantly. In fact, the BP guys point out the system's blind spots and their own sense of strengths or weaknesses it overlooks when they handicap the season.

There was some recent season where all teams were above .400 and below .600, remember? I think only the 2nd time in history?

― Dr Morbius (Dr Morbius), Thursday, 31 March 2005 13:10 (nineteen years ago) link

So do the wins/games balance out, Morbius?

― Alex in SF (Alex in SF), Thursday, 31 March 2005 15:27 (nineteen years ago) link

You must be logged in to post. Please either login here, or if you are not registered, you may register here.