data/programming/updating model question

Rufus · 07-22-2008, 01:41 AM

Hi all,

I have been working on modeling MLB for most of a year and have a (primarily econometric) model that I can confidently say should be a long-term winner. Problem is, I'm not much of a programmer. I did take an Intro Programming course (IN JAVA) in my last semester of school (last spring) and I can program pretty well in Stata (the statistical program I use), but the real problem is updating the model every day with new data. I've written code to generate a prediction (with the two teams and pitchers as inputs) but I'll need to add data new game data every day.

My question is this: Do any of you/have any of you faced this issue? What is your solution? Do I pretty much need to learn to write a webcrawler using Perl to scrape data offline?

Any help/advice would be greatly appreciated.

Sinister Cat · 07-22-2008, 02:10 AM

Yes, scrape the data. I use Tcl to do it-- a lot easier than Perl. Python or Ruby would be other choices. Perl is popular for this kind of thing too but probably more difficult for a novice programmer.

Ganchrow · 07-22-2008, 04:33 AM

I'm personally partial to Perl, in which I probably do close to 90% of my programming. If you have experience with Java you should have absolutely no problem with Perl.

You might also want to look into hiring programming help off of rentacoder.com or a similar site.

Rufus · 07-22-2008, 12:02 PM

Any good book you would recommend to learn Perl?

Ganchrow · 07-22-2008, 12:05 PM

Quote:

Originally Posted by modelman

Any good book you would recommend to learn Perl?

The O'Reilly Learning Perl and Programming Perl books are very user-friendly.

durito · 07-22-2008, 12:06 PM

I have a programmer I hired through rentacoder.com

He's pretty cheap, but the work isn't quite what i want. If i can ever get my brain to work again, I'm going to try and learn again myself.

durito · 07-22-2008, 12:25 PM

Quote:

Originally Posted by Ganchrow

The O'Reilly Learning Perl and Programming Perl books are very user-friendly.

Ordered. Finding out that amazon can deliver to Colombia has not been good for my spending habits.

Data · 07-22-2008, 01:09 PM

Quote:

Originally Posted by durito

Ordered. Finding out that amazon can deliver to Colombia has not been good for my spending habits.

This is a cheaper way:
http://proquest.safaribooksonline.com/

You read the books online (or save them on your computer as PDFs). A great time saving benefit, you get all the sample code in downloadable files.

MrX · 07-22-2008, 01:18 PM

If you have the time, I'd definitely recommend learning enough to write your own scrapers.

Occasionally the site you're scraping from will make a slight change to their format, or some aspect of a report will be different enough from the norm to throw off your scraper and it's sure nice to be able to make changes on the fly instead of waiting for your programmer.

As a side note, I scrape most of my data from MLB.com and they have remained blessedly consistent for a couple of years.

Rufus · 07-22-2008, 01:52 PM

Thanks everyone! I really appreciate the help.

Justin7 · 07-22-2008, 01:55 PM

I paid a programmer to write a scraper in Perl. It would automatically download stats from USAToday every day.

Rufus · 07-22-2008, 02:56 PM

Quote:

Originally Posted by Justin7

I paid a programmer to write a scraper in Perl. It would automatically download stats from USAToday every day.

How much would that sort of thing cost?

rsigley · 07-22-2008, 04:38 PM

i use php, works pretty well - never had a problem

and i use windows scheduler to run it once a day at 7am and input it into mysql db

also for mlb you can just use dougstats, he updates once a day though i notice he's missing a couple players (like e. gonzalez from the padres)

Rufus · 07-22-2008, 06:02 PM

I just looked at dougstats. It seems pretty good, except I need the game-by-game stats since I don't use uniform weights. I normally get it from baseball-reference (I have a subscription so I can use the Play Index) but it's pain in the ass to copy and paste it all.

Rufus · 07-22-2008, 06:52 PM

What database/statistical software do other people use? Being an econ major in college, I learned Stata, which works well for me once I get data into it. I can do all the regressions, statistical analysis, and data management. Anybody else have other database preferences?

MrX · 07-22-2008, 07:02 PM

Quote:

Originally Posted by modelman

What database/statistical software do other people use? Being an econ major in college, I learned Stata, which works well for me once I get data into it. I can do all the regressions, statistical analysis, and data management. Anybody else have other database preferences?

I find that the statistical functions in Excel 2007 meet most of my needs. I've dabbled in a couple other programs for regression analysis, but not lately.

Mysql for database needs.