Missing Elements from HTML File Using BeautifulSoup

Issue

I'm very new to the web-scraping world, and I'm trying to scrape the names of shoes from a website. When I use inspect on the website, there's a div tag that has basically the entire webpage inside it, but when I print out the html code, the div tag is completely empty! Here's my current code:

from bs4 import BeautifulSoup
import requests
import time

def findShoeNames():
   html_file = requests.get('https://www.goat.com/sneakers/brand/air-jordan').text
   soup = BeautifulSoup(html_file, 'lxml')
   print(soup)
 
if __name__ == "__main__":
   findShoeNames()

When I call my function and print(soup), the div tag looks like this:

<div id="root"></div>

But as previously mentioned, when I hit inspect on the website, this div tag has basically the entire webpage inside it. So I'm unable to scrape any data from the website.

Please help! Thanks

Solution

website use js to load. so you should use selenium and chromedriver. install selenium install chromedriver from here (unzip and copy your python folder)

import time
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

url = "https://www.goat.com/sneakers/brand/air-jordan"
options = Options()
options.add_argument('--headless')
options.add_argument('--disable-gpu')
driver = webdriver.Chrome(chrome_options=options)
driver.get(url)
time.sleep(1)
page = driver.page_source
driver.quit()
soup = BeautifulSoup(page, 'lxml')

print(soup.prettify)

Answered By - kağan hazal koçdemir