1. #36
    HAPPY BOY
    HAPPY BOY's Avatar Become A Pro!
    Join Date: 08-10-05
    Posts: 7,108
    Betpoints: 800

    man this place has changed

  2. #37
    smitch124
    smitch124's Avatar SBR PRO
    Join Date: 05-19-08
    Posts: 12,564
    Betpoints: 1547

    Happy I think this calls for a cartoon that depicts this change at SBR.

  3. #38
    03fan
    03fan's Avatar Become A Pro!
    Join Date: 02-01-12
    Posts: 35
    Betpoints: 468

    What does a scraper do for you? Help get lines before movements?

  4. #39
    Kelly_F
    Kelly_F's Avatar Become A Pro!
    Join Date: 12-27-11
    Posts: 43
    Betpoints: 1119

    Quote Originally Posted by 03fan View Post
    What does a scraper do for you? Help get lines before movements?
    Erm, no, because then it would be able to predict the future. So you would be better off getting next weeks lotto results than lines off SBO.

    If you mean does a scraper alert you to line movements, it is very challenging to build a scraper fast enough to do that. You would have to constantly collect data to do so an getting blocked is one of the issues you would face.

  5. #40
    Kelly_F
    Kelly_F's Avatar Become A Pro!
    Join Date: 12-27-11
    Posts: 43
    Betpoints: 1119

    Quote Originally Posted by slobib View Post
    I need it to compare with local bookies and get the best odds fastest.
    There are sites that do this but i think they lack quality and arent optimal.
    In reference to local bookies, what country are you in?

  6. #41
    Kelly_F
    Kelly_F's Avatar Become A Pro!
    Join Date: 12-27-11
    Posts: 43
    Betpoints: 1119

    Quote Originally Posted by HUY View Post
    You won't have any problems with the amount of data. You will be blocked if you request pages too often. Just download every few minutes and you should be fine.
    I would hazard a guess that he wants the data faster/more regularly than that, which is of course the challenge with writing a scraper.

  7. #42
    Maverick22
    Maverick22's Avatar Become A Pro!
    Join Date: 04-10-10
    Posts: 807
    Betpoints: 58

    How is setting a timeout/sleep time a challenge?

  8. #43
    Kelly_F
    Kelly_F's Avatar Become A Pro!
    Join Date: 12-27-11
    Posts: 43
    Betpoints: 1119

    Quote Originally Posted by Maverick22 View Post
    How is setting a timeout/sleep time a challenge?
    I contract a couple of programmers to write/maintain applications for me, and so I don't specifically understand what you mean by that.

    However SBO's odds are very dynamic, very fast. To meaningfully catch those movements a scraper would need to be collecting data as regularly as possible. I'm talking every few seconds or less. That's why the XML feed would be preferable to the OP, but he doesn't have access.

    I don't know what a timeout/sleep time is but if it enables you to scrape a site every few seconds please explain.

  9. #44
    Maverick22
    Maverick22's Avatar Become A Pro!
    Join Date: 04-10-10
    Posts: 807
    Betpoints: 58

    while(1) **

    doWorkToScrapeData();

    sleep(2000);

    **

    protected final void sleep() **

    if (SLEEP_TIME == 0) **
    return;
    **

    try **

    logger.debug("Application Sleeping For **** seconds", (SLEEP_TIME / 1000));
    Thread.sleep(getSleepTimeMinusWarning()) ;

    logger.debug("Application Resuming In **** seconds", (SLEEP_TIME_WARNING / 1000));
    Thread.sleep(SLEEP_TIME_WARNING);
    ** catch (InterruptedException ex) **
    logger.error("Exception Occurred While Trying To Sleep");
    logger.error(ex.getMessage());
    **
    **

  10. #45
    Maverick22
    Maverick22's Avatar Become A Pro!
    Join Date: 04-10-10
    Posts: 807
    Betpoints: 58

    SBR doesnt seem like curly braces. but that is what i mean. the actual scraping code is not set on a time out. it just executes some tasks. and some code that controls the scraper will designate when to sleep.

    simplifies the scraper itself makes it easier to manage how the sleeping is controlled.

  11. #46
    Kelly_F
    Kelly_F's Avatar Become A Pro!
    Join Date: 12-27-11
    Posts: 43
    Betpoints: 1119

    Sorry, I missed a few words out of my post. I meant to say I am not a programmer myself (but contract a couple to write applications for me).

    So whilst I have experience and understanding of some of the challenges that they face, I don't understand your code.

    What I am asking is why do you think that setting a timeout/sleep time will enable data to be collected frequently, hitting the site many times in a minute.

  12. #47
    Maverick22
    Maverick22's Avatar Become A Pro!
    Join Date: 04-10-10
    Posts: 807
    Betpoints: 58

    I don't really understand the question.

    By creating functionality to sleep. you can repeatedly call the same code over and over again and simply configure how much time between each call.

    So you dictact: "go to the pinnacle odds xml page, download and process the xml and then insert it into my database. Then... sleep (or wait) for 3000 milliseconds and then do it all over again, until i tell you to stop."

  13. #48
    Kelly_F
    Kelly_F's Avatar Become A Pro!
    Join Date: 12-27-11
    Posts: 43
    Betpoints: 1119

    configuring the time between each call meaning how often the scraper fetches/collects the data right?

    That's fine, if he is using the Pinnacle API, but it' different if he is scraping a site. Scrape it too often, and he will get blocked. He has to put things in place to prevent that.

    And scraping a site is not as fast as using an XML feed.

    The point I am making is that I think I an tell from the OP's posts that speed is important to him. When the odds change, he wants to know about it instantly, or at least as fast as possible.

  14. #49
    BigDaddy777
    BigDaddy777's Avatar Become A Pro!
    Join Date: 05-14-13
    Posts: 115
    Betpoints: 428


    I've written scrapers for many websites and I can scrape SBObet for you, contact me
    Do you have any pre-written for this site - > http://scoresandodds.com?

  15. #50
    BigDaddy777
    BigDaddy777's Avatar Become A Pro!
    Join Date: 05-14-13
    Posts: 115
    Betpoints: 428

    Data Scraping from Websites

    Here's some feedback I received from a programmer when I sent him the comments about scrapers from this forum -

    =====

    You mentioned someone said "I use python and LXML, and some tool for firefox "page inspector" i can get any info from a page within a few minutes."

    Well, there are different scraping techniques. There is also something called cUrl technology. The site to be crawled determines which is the most suitable technology for the job.

    You also mentioned someone wrote "Scoresandodds.com offer an XML feed API for their match ups and odds covering all sports. I consulted with a group who used it..."

    I don't think this is a true statement. The site doesn't have any api listing. Again the statement is contradictory. Either they provide XML feed or API. *XML feed API* is a wrong phrase.

    =====

    Any reaction to this?
    Last edited by BigDaddy777; 02-13-14 at 08:56 PM. Reason: grammar

  16. #51
    BigDaddy777
    BigDaddy777's Avatar Become A Pro!
    Join Date: 05-14-13
    Posts: 115
    Betpoints: 428


    Don't all chime in at once...

    baha

  17. #52
    flyingillini
    flyingillini's Avatar SBR PRO
    Join Date: 12-06-06
    Posts: 41,218
    Betpoints: 2163

    One of the best books in the business, been using them for years.

  18. #53
    SportMac
    SportMac's Avatar Become A Pro!
    Join Date: 02-25-14
    Posts: 50
    Betpoints: 25

    also there are lots of application those can track your account and if found robotic, it will be freezed.

  19. #54
    SportMac
    SportMac's Avatar Become A Pro!
    Join Date: 02-25-14
    Posts: 50
    Betpoints: 25

    Quote Originally Posted by Kelly_F View Post
    configuring the time between each call meaning how often the scraper fetches/collects the data right?

    That's fine, if he is using the Pinnacle API, but it' different if he is scraping a site. Scrape it too often, and he will get blocked. He has to put things in place to prevent that.

    And scraping a site is not as fast as using an XML feed.

    The point I am making is that I think I an tell from the OP's posts that speed is important to him. When the odds change, he wants to know about it instantly, or at least as fast as possible.
    Do someone have any sample of XML feed. I also wanna to make a progreame that can help in XML feed.

  20. #55
    lamichaeljames
    lamichaeljames's Avatar Become A Pro!
    Join Date: 06-02-14
    Posts: 40
    Betpoints: 109

    Quote Originally Posted by SportMac View Post
    also there are lots of application those can track your account and if found robotic, it will be freezed.
    agreed but there are ways around it.

  21. #56
    nmlfree
    nmlfree's Avatar Become A Pro!
    Join Date: 06-22-10
    Posts: 3
    Betpoints: 156

    Arwar, i don't have enough posts to send you a PM but i'm interested in your scraper, can you send a PM with your email or skype so we can discuss further details?

    Thanks,

    Eric

  22. #57
    bihon
    bihon's Avatar Become A Pro!
    Join Date: 11-03-09
    Posts: 731

    Quote Originally Posted by pringles View Post
    well, im using both very skilled designer and programmer, we have done the interface and the programming starts in a few days.
    im paying around 3k€ for the whole set-up
    The price looks fair, but if you are paying €3000 for that, you probably don't bet peanuts and you depend on reliability of data. Why risking when you can order paid feed in multiple formats from reliable companies?

First 12
Top