1. #246
    mikmak
    mikmak's Avatar Become A Pro!
    Join Date: 05-03-13
    Posts: 29
    Betpoints: 239

    I would be interested in any literature that will help me get better and specifically with data scraping and model building. I'm guessing there are many others that would as well.

  2. #247
    wizcodlifa
    wizcodlifa's Avatar Become A Pro!
    Join Date: 01-10-12
    Posts: 921
    Betpoints: 1173

    Hello all

    I am willing to learn and master how to program. I am going to first admit I really have no idea even the littlest bit on how to program. Is there any info or classes to introduce this??

    All Help is greatly appreciated

  3. #248
    matthew919
    Update your status
    matthew919's Avatar Become A Pro!
    Join Date: 11-21-12
    Posts: 421
    Betpoints: 5869

    Not trying to be a jerk, but did you even read this thread? Start at page 1 and make an attempt. The links that were posted (by Inspirited, I believe) in a previous thread you started will also be very helpful. Ask specific questions when you run into problems.

    Or, if you're looking for classroom instruction, take (or audit) a course at a community college. Good luck.

  4. #249
    easyliving
    easyliving's Avatar Become A Pro!
    Join Date: 06-25-12
    Posts: 8,876
    Betpoints: 11

    thanks for the info will read this in my spare time.

  5. #250
    ExodusNZ
    ExodusNZ's Avatar Become A Pro!
    Join Date: 09-02-11
    Posts: 605
    Betpoints: 1314

    Hi

    Im after Rugby Union or Rugby League data ?

  6. #251
    Cyyyyk
    Cyyyyk's Avatar Become A Pro!
    Join Date: 08-30-13
    Posts: 11
    Betpoints: 78

    Interesting thread. I have been very successful building systems in excel but getting the raw data has always been the primary sticking point. I am going to have to look into some of these scraping methods. Thanks to everyone who posted!

  7. #252
    Cyyyyk
    Cyyyyk's Avatar Become A Pro!
    Join Date: 08-30-13
    Posts: 11
    Betpoints: 78

    Quote Originally Posted by bobbydrake View Post
    Simple Perl Web Parser Script
    I don't mean any disrespect toward the original poster's preferred programming language. This is just another method. We all need options in life.
    ---------------------------------------------------------------
    #! /usr/bin/perl
    print "Content-type: text/html\n\n";

    use LWP::Simple;

    use HTML::TreeBuilder;

    use HTML::FormatText;

    $URL = get("http://www.websiteyouwanttoparse.com");

    $Format = HTML::FormatText->new;

    $TreeBuilder = HTML::TreeBuilder->new;

    $TreeBuilder->parse($URL);

    $Parsed = $Format->format($TreeBuilder);

    print "$Parsed";
    open(FILE, ">file.txt");
    print FILE "$Parsed";
    close(File);

    exit;
    --------------------------------------------------------------
    Things you need to edit:
    $URL = get("http://www.websiteyouwanttoparse.com");
    Change the website name to a website you want to parse. If you know a little perl, those quotations have to be included.

    and also edit
    open(FILE, ">file.txt");
    Change file.txt to a file name of your choice. Remember to change this every time you parse a new website or you'll have this program delete all the info on that file from a previous save.

    This is my opinion. I take no responsibility for your actions. This is only for educational purposes.
    Forgive my ignorance...... but how would I actually utilize this script? Do I upload to a server cgibin and then go to the page? I mean how do I get this script to run?

  8. #253
    dolyms
    dolyms's Avatar Become A Pro!
    Join Date: 02-07-12
    Posts: 23
    Betpoints: 39

    thank you for doing this.

  9. #254
    bozeman
    bozeman's Avatar Become A Pro!
    Join Date: 11-11-09
    Posts: 2,161
    Betpoints: 5145

    don't waste your time research something else, but not stats

  10. #255
    V4Value
    V4Value's Avatar Become A Pro!
    Join Date: 09-01-13
    Posts: 368
    Betpoints: 400

    Just came across this..can anyone point out any other valuable threads like this one.

  11. #256
    HUY
    HUY's Avatar Become A Pro!
    Join Date: 04-29-09
    Posts: 253
    Betpoints: 3257

    Quote Originally Posted by mikmak View Post
    I would be interested in any literature that will help me get better and specifically with data scraping and model building. I'm guessing there are many others that would as well.
    Since you're asking, here's a plug for my site, www.statsfair.com . You might pick up some ideas from there.

  12. #257
    elgreco
    elgreco's Avatar Become A Pro!
    Join Date: 12-16-09
    Posts: 988
    Betpoints: 1612

    Wrote my first python script thanks to this thread

  13. #258
    bobbydrake
    bobbydrake's Avatar Become A Pro!
    Join Date: 02-16-09
    Posts: 38
    Betpoints: 375

    Quote Originally Posted by Cyyyyk View Post
    Forgive my ignorance...... but how would I actually utilize this script? Do I upload to a server cgibin and then go to the page? I mean how do I get this script to run?
    First, find out if you have >= perl 5.0 installed
    Install perl modules (www.cpan.org) LWP::Simple, HTML::TreeBuilder, HTML::FormatText
    Copy and Paste script in text editor. Save and make file executable (chmod +x file.pl)
    On command line ~$ perl file.pl

    Forgive me for the late response. Don't come here that often.

    My original post, it seems, was a digression to the OP's intention for this thread. I apologize.
    I would have to yield to the OP's programming language (python) in this thread since he can educate better than I can. Python is a great programming language. It's MIT's language of choice.

  14. #259
    drawster
    drawster's Avatar Become A Pro!
    Join Date: 11-03-13
    Posts: 23
    Betpoints: 72

    Great Post!

  15. #260
    kaalhode
    kaalhode's Avatar Become A Pro!
    Join Date: 01-11-14
    Posts: 15
    Betpoints: 341

    Really informative and great post. Got a tiny bit of programming experience, but have never really used it for something useful

  16. #261
    jtluongo
    jtluongo's Avatar Become A Pro!
    Join Date: 01-06-14
    Posts: 104
    Betpoints: 60

    before learning programming I would like to learn more about predictive analytics and modeling. is there any good books out there I can read up on or maybe a youtube series etc

  17. #262
    Inspirited
    Inspirited's Avatar SBR PRO
    Join Date: 06-26-10
    Posts: 1,783
    Betpoints: 17732

    edx.com has a sabermetrics course coming up
    mathletics is a good book to start with

  18. #263
    metaldeth
    metaldeth's Avatar Become A Pro!
    Join Date: 02-28-14
    Posts: 43
    Betpoints: 644

    I've a basic knowledge of programming in java and excel. I've read the first 4 chapters of the Python book, nothing new, just a new language. But I find the transition from those 4 chapters to scraping webpages extremely difficult. I've never worked with websites before.

    Are there any examples, books, exercises out there that can help me with this transition?


    Thanks!

  19. #264
    Maniac
    Maniac's Avatar Become A Pro!
    Join Date: 04-12-11
    Posts: 667
    Betpoints: 8815

    Quote Originally Posted by Inspirited View Post
    edx.com has a sabermetrics course coming up
    mathletics is a good book to start with
    That Sabermetrics course starts today for anyone still interested.

  20. #265
    horja1
    Grrrrrr
    horja1's Avatar Become A Pro!
    Join Date: 01-13-11
    Posts: 5,646
    Betpoints: 12


  21. #266
    dodo_molnar
    dodo_molnar's Avatar Become A Pro!
    Join Date: 06-02-14
    Posts: 3
    Betpoints: 24

    Hi,
    maybe on-line software (betviz) can help you.

  22. #267
    lamichaeljames
    lamichaeljames's Avatar Become A Pro!
    Join Date: 06-02-14
    Posts: 40
    Betpoints: 109

    great stuff here. Hopefully, this continues.

  23. #268
    donkeyshark
    donkeyshark's Avatar Become A Pro!
    Join Date: 08-09-14
    Posts: 6
    Betpoints: 162

    Quote Originally Posted by jtluongo View Post
    before learning programming I would like to learn more about predictive analytics and modeling. is there any good books out there I can read up on or maybe a youtube series etc
    Coursera has a machine learning course which is offered by Stanford. It's a good place to start.

  24. #269
    poetbil
    poetbil's Avatar Become A Pro!
    Join Date: 07-11-13
    Posts: 4
    Betpoints: 528

    people like this ,who share knowledge are Blessed

  25. #270
    chipper
    chipper's Avatar Become A Pro!
    Join Date: 01-07-10
    Posts: 1,994
    Betpoints: 2174

    Great info... I need to read & reread it a few times for sure.

  26. #271
    markorozo
    markorozo's Avatar Become A Pro!
    Join Date: 01-24-15
    Posts: 13
    Betpoints: 366

    Quote Originally Posted by ljump12 View Post
    Section D) How to scrape the internet for data

    One of the most important aspects of research is the data that you have. Without data, there can't be any model. Fortunately, most data is free -- Unfortunately, most data isn't immediately in the best computer parsable formats [like .csv, or .xml]. To get the data into formats we can use we will need to "scrape" websites for it.

    A couple "packages" have been created that will greatly improve our ability to scrape webpages. It can certaintly be done in python without them -- but they will make your life a whole lot easier:

    Mechanize - This will allow us to open webpages easily (http://wwwsearch.sourceforge.net/mechanize/)
    Beautiful Soup - This will allow us to parse apart the webpages (http://www.crummy.com/software/BeautifulSoup/)

    Installing Beautiful Soup is pretty easy, you can just put the http://www.crummy.com/software/Beaut...lSoup-3.0.0.py Beautiful soup python file in the same directory you are running your code from.

    Installing Mechanize is a little tougher, on a *nix machine, cd to the directory of where you downloaded it and extract it (tar -xzvf [filename]). Then cd into the extracted directory and install it by typing "sudo python setup.py install" It should install, you can post here if you have any problems. As far as windows goes, you may be on your own -- I can't imagine it's very tough, and there's probably a tutorial somewhere online.

    Now that the installation is out of the way, it's time to get down to business. I'll give you the basics here, and you should be able to refer to the documentation for more complicated examples. I'm going to assume you have a basic familiarity of html -- if you don't, you may want to search for a quick tutorial. Let's make our first example getting a list of today's injuries from statfox for MLB baseball:

    PHP Code:


    from BeautifulSoup import BeautifulSoup
    SoupStrainer ## This tells python to use Beautiful Soup
    from mechanize import Browser   ## This tells python we want to use a browser (which is defined in mechanize)
    import re   ## This tells python that we will be using some regular expressions.
                ## .. Regular expression allow us to search for a sequence of characters
                ## .. within a larger string
    import time
    import datetime

    ## The first step is to create our browser..
    br Browser()

    ## Now let's open the injuries page on statfox. This one line will open and retreive the html.
    response br.open("http://www.sbrodds.com/StoryArchivesForm.aspx?ShortNameLeague=mlb&ArticleType=injury&l=3").read()

    ## Now we need to tell Beautiful Soup that we would like to search through the response.
    ## .. This next line will tell beautiful soup to only return links to the individual inuries.
    ## .. We know that all the links to the injuries have "ShortNameLeague=mlb&ArticleType=injury" 
    ## .. in their url, so we search for these links. Each of these links has a title that describes
    ## .. the injury which we will use in the next line.
    linksToInjuries SoupStrainer('a'href=re.compile('ShortNameLeague=mlb&ArticleType=injury'))

    ## This will put the title of all links in the "linksToInjuries" into an array.
    ## We then call Set on our array to change the array to a "set" which by definition has no duplicates.
    injuryTitles set([injuryPage['title'] for injuryPage in BeautifulSoup(responseparseOnlyThese=linksToInjuries)])


    ## Finally let's print all the injuries out that are for today's date.
    today datetime.date.today()
    # the function strftime() (string-format time) produces nice formatting
    # All codes are detailed at http://www.python.org/doc/current/lib/module-time.html
    date =  today.strftime("%m/%d"

    ## Now let's print out the injuries that we have.
    for title in injuryTitles:
        
    ## See if the date is in the title, if it is: print it.
        
    if re.search(datetitle):
            print 
    title 
    It might seem like a lot at first, but it's not much code. Take it slow and use google when you dont know what a function does. Googling "python [some piece of code you dont understand]" will work magic. Ask here and i can further break down any slice of code.

    Sorry I haven't had much time -- If anyone can post an example of what kind of data they would like to be scraped, I will create one more example using both BeautifulSoup and Mechanize.
    Thanks for sharing all this

    How would you go to scrap a Javascript website? Lets say whoscored.com?

  27. #272
    stevenash
    stevenash's Avatar Moderator
    Join Date: 01-17-11
    Posts: 62,368
    Betpoints: 32811

    Jump12?
    What do you do for a living?

    And so far all your posts are spot on.
    Well done.

    I'm an I/T Operations Analyst myself.
    The business end of sports is pretty much all analytics.

    I'm big on SQL.

    Good work here, I'll drop by later.

  28. #273
    stevenash
    stevenash's Avatar Moderator
    Join Date: 01-17-11
    Posts: 62,368
    Betpoints: 32811

    Quote Originally Posted by jtluongo View Post
    before learning programming I would like to learn more about predictive analytics and modeling. is there any good books out there I can read up on or maybe a youtube series etc
    Read this twice.
    Buy it, it's not cheap, but it's the best.
    You can get it in the Google store

    https://play.google.com/store/books/...Q&gclsrc=aw.ds

  29. #274
    Big Bear
    Love your neighbor
    Big Bear's Avatar Become A Pro!
    Join Date: 11-01-11
    Posts: 43,253
    Betpoints: 14

    good stuff here is food for thought

    Decision =power
    Progress = happiness

    Learning is the relationship of a "Known" to an "unknown" If you are having trouble learning something it's b/c you are unable to relate it to something you already know.

    example Fishing is like sales... If you know how to fish then you know how to sell.

    A strategy is specific way of organizing resources in order to consistently generate the results you want.

    The formula for happiness is to be able to meet or exceed your expectations.

    think about this when designing your model for this baseball season.

  30. #275
    oilcountry99
    oilcountry99's Avatar Become A Pro!
    Join Date: 08-29-10
    Posts: 707
    Betpoints: 1094

    This was a great thread until post #26 and the OP stopped after section 1d. What happened?

  31. #276
    Elko
    Elko's Avatar Become A Pro!
    Join Date: 10-09-15
    Posts: 4
    Betpoints: 30

    this thread has been around so long I think 1/2 of the original posters have died.

  32. #277
    ljump12
    ljump12's Avatar Become A Pro!
    Join Date: 12-08-09
    Posts: 108
    Betpoints: 258

    Quote Originally Posted by Elko View Post
    this thread has been around so long I think 1/2 of the original posters have died.
    Count me in the half that's alive

    For what it's worth (because I doubt I'm ever going to finish this guide):
    http://www.amazon.com/Web-Scraping-P...s=web+scraping

    &

    http://www.amazon.com/Python-Data-An...0_SR121%2C160_

    Should get you far.
    Last edited by ljump12; 10-12-15 at 01:40 PM.
    Nomination(s):
    This post was nominated 1 time . To view the nominated thread please click here. People who nominated: vavoulas

  33. #278
    elgary
    elgary's Avatar Become A Pro!
    Join Date: 10-20-15
    Posts: 4
    Betpoints: 468

    help

    if someone can help me i will appreciate it, i read the chapters and all that stuff, wathc videos, but i dont know how to download the csv file form the link. i hope someone can help me with that, thanks.

  34. #279
    Muller Rose
    Muller Rose's Avatar Become A Pro!
    Join Date: 06-01-16
    Posts: 219
    Betpoints: 144

    Great info you have here. Thanks for taking time to post this.

  35. #280
    mtalock
    mtalock's Avatar Become A Pro!
    Join Date: 08-09-16
    Posts: 45
    Betpoints: 2141

    Wow,this is easy?

    Hey, thank you for the tech info and the links.
    this seems like a plethora of info but I am diving into the deep end.

First ... 56789 Last
Top