Link to home
Start Free TrialLog in
Avatar of Fred Miles
Fred Miles

asked on

Scraping a dynamic URL that changes based on time using Selenium in Python

I am attempting to scrape the following URL:

https://www.oddsportal.com/soccer/england/premier-league/liverpool-norwich-4IMoMG3q/

Using the Network tab in Chrome's developer tools, you can see there is an API that feeds the data into the website - which is in the form of a JSON - and looks as below. This data is what I am trying to scrape.

https://fb.oddsportal.com/feed/match/1-1-4IMoMG3q-5-2-yj1e3.dat?_=1562831112277

This is the code I am currently trying to scrape this with:

from selenium.webdriver.chrome.options import Options
import json
import urllib.parse
from time import time

options = Options()
options.headless = True
driver = webdriver.Chrome(options=options)

# Access the initial webpage to create the info_dict (including the match_id, and hash)
driver.get('https://www.oddsportal.com/soccer/england/premier-league/liverpool-norwich-4IMoMG3q')
page = driver.page_source
info_dict = json.loads(page.split('var page = new PageEvent(')[-1].split(');')[0])
xhash = urllib.parse.unquote(info_dict['xhash'])
match_id = info_dict['id']

# Access to the feed URL based on the values from the info_dict
driver.get('http://fb.oddsportal.com/feed/match/1-1-{}-1-2-{}.dat?_={}'.format(match_id, xhash, int(round(time()*1000)) + 1000))
print(driver.page_source)

Open in new window


The URL is built up of 3 factors - the match_id, the hash, and epoch time in milliseconds. However, when I try to access this in Selenium, I get the following response:

globals.jsonpCallback('/feed/match/1-1-4IMoMG3q-1-2-yjb3a.dat?_=1562795864899', {'e':'404'})

Open in new window


Would really appreciate any help with this, as I don't really understand where I'm going wrong!
Avatar of David Favor
David Favor
Flag of United States of America image

https://www.oddsportal.com/community/feed/ provides, what looks to be, an easier way to acquire this data.
This question needs an answer!
Become an EE member today
7 DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform.
View membership options
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.