How to read HTML content in python from website?

Issue

I'm trying to read a table from this nasdaq website https://www.nasdaq.com/market-activity/quotes/nasdaq-ndx-index but this simple code does not get any result, so I can't understand what I'm doing wrong

import numpy as np
import pandas as pd
import requests
import time
import random as rd
from bs4 import BeautifulSoup
import requests
import csv

def main():
   URL = 'https://www.nasdaq.com/market-activity/quotes/nasdaq-ndx-index'
   df = pd.read_html(URL)
   print(df)


if __name__ == "__main__":
    main()

Solution

The website generates data in-situ. Instead, read the html from the console using the following code:

import requests

headers={"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36"}
result=requests.get("https://api.nasdaq.com/api/quote/list-type/nasdaq100",headers=headers)
print(result)

You can find your user agent from your browser. In Google Chrome, right click on the page and click 'inspect'. From here you can go to Network, select a request which contains User Agent information and simply copy and paste.

Answered By - Beverley1