Use selenium to navigate in joint selection/list elements

Issue

I'm using beautifulSoup / selenium to do some webscraping and am hitting a wall with a certain dropdown select menu. The rough HTML is as follows:

<div class="selection-box" alt="selection" title="selection" role="select" tabindex="0">
<select id="select" style="display: none;">
    <option value="1">First</option>
    <option value="2">Second</option>
    <option value="3" selected="selected">Third</option>
</select>
<div class="current">Third</div>
<ul class="options" style="display: none;">
    <li class="search--option" alt="First option" title="First option" aria-label="First option" role="option" tabindex="0">First</li>
    <li class="search--option" alt="Second option" title="Second option" aria-label="Second option" role="option" tabindex="0">Second</li>
    <li class="search--option selected" alt="Third option" title="Third option" aria-label="Third option" role="option" tabindex="0">Third</li>
</ul>

When I'm working the menu through the browsers, it changes as follows:

• the wrapper div class changes to "selection-box active"
• the ul changes to "display: block"
• once I pick a different option, those two get reversed again and the middle div and the selected li item change accordingly

I want to use selenium to select a certain option. So far, I tried the following:

    from selenium.webdriver.support.ui import Select
    drpBrand = driver.find_element(By.ID, "select");
    css = 'select#select' # css selector of the element
    js = """const data_options = Array.from(document.querySelectorAll('{css}'));
            data_options.forEach(a=>{{a.style='display:block;';}});""".format(css=css)
    driver.execute_script(js)
    drpBrand.select_by_visible_text("Third");

This is a best-of using various threads (element not visible: Element is not currently visible and may not be manipulated - Selenium webdriver, How to select a dropdown value in Selenium WebDriver using Java), but it still doesn't work. Any ideas? I assume I need to target the list as well (besides the select)?

The error is always

selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable: Element is not currently visible and may not be manipulated

Thanks

Solution

If I understood your goal correct (to select dropdown option), I suggest to emulate native user behaviour and rely on visible elements.

Dropdown element has selector .superstar-search--selection-box. You should wait for it's presence and click on it.

Dropdown options have selector .superstar-search--option . You should wait for visibility of this elements and filter them, for example, by containing text condition.

You can build more complex function than I wrote by defining dropdown open selector and dropdown option selector as function arguments.

If you have issue with Google Ads, I added example how to deal with it.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()

driver.get("https://www.wwe.com/superstars")
wait = WebDriverWait(driver, 15)
driver.maximize_window()

def select_dropdown_option_by_text(text):
    dropdown = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, '.superstar-search--selection-box')))
    dropdown.click()
    options = wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, '.superstar-search--option')))
    expected_option = [element for element in options if element.text.lower() == text][0]
    expected_option.click()
    wait.until(EC.invisibility_of_element(expected_option))

def remove_google_ads():
    return driver.execute_script("""
      function waitForElementAndRemove() {
        let element = document.querySelector('[id*=google_ads_iframe],[id*=ad_iframe]');
        if (element) {
            element.remove();
            console.log('Removed ad');
        } else {
           setTimeout(waitForElementAndRemove, 1000);
        }
    }

      waitForElementAndRemove();
    """)

remove_google_ads()
select_dropdown_option_by_text('all superstars')

Answered By - Yaroslavm