finding username field with mechanize

Issue

I am attempting to automate navigation of https://support.paloaltonetworks.com/, which requires me to first log in on https://sso.paloaltonetworks.com/ I've managed to open the site with the following code

import re
import mechanize
br = mechanize.Browser()
br.set_handle_robots(False)
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]
br.open("https://paloaltonetworks.com/")
print(br)

Anytime I try to find the username/e-mail address field to fill in, I get some error.

If I add the following to look for forms:

formcount=0
for frm in br.forms():
  if str(frm.attrs["id"])=="username":
    break
  formcount=formcount+1
br.select_form(nr=formcount)

I get this output:

Traceback (most recent call last):
  File "scraper_mechanize.py", line 20, in <module>
    br.select_form(nr=formcount)
  File "/home/tdadmin/testing/Python/PaloAlto/grabdata/lib64/python3.6/site-packages/mechanize/_mechanize.py", line 681, in select_form
    raise FormNotFoundError("no form matching " + description)
mechanize._mechanize.FormNotFoundError: no form matching nr 0

Then I try it with this code, which produces no output:

for form in br.forms():
   print("Form name:", form.name)
   print(form)

In the source of the page, I find the input field:

<input type="text" placeholder="" name="username" id="idp-discovery-username" value="" aria-label="" autocomplete="username" aria-required="true" required="" data-keeper-lock-id="k-o4yd510ha7">

That's part of a larger structure. The whole form element is as follows:

<form novalidate="novalidate" method="POST" action="/" data-se="o-form" slot="content" id="form19" class="idp-discovery-form o-form o-form-edit-mode"><div data-se="o-form-content" class="o-form-content o-form-theme clearfix"><h2 data-se="o-form-head" class="okta-form-title o-form-head">Sign In</h2><div class="o-form-error-container" data-se="o-form-error-container"></div><div class="o-form-fieldset-container" data-se="o-form-fieldset-container"><div data-se="o-form-fieldset" class="o-form-fieldset o-form-label-top margin-btm-30"><div data-se="o-form-label" class="okta-form-label o-form-label"><label for="idp-discovery-username">Email&nbsp;</label></div><div data-se="o-form-input-container" class="o-form-input"><span data-se="o-form-input-username" class="o-form-input-name-username o-form-control okta-form-input-field input-fix"><input type="text" placeholder="" name="username" id="idp-discovery-username" value="" aria-label="" autocomplete="username" aria-required="true" required="" data-keeper-lock-id="k-o4yd510ha7"><keeper-lock class="focus-visible keeper-lock-disabled" tabindex="-1" id="k-o4yd510ha7" aria-label="Open Keeper Popup" role="button" style="background-image: url(&quot;chrome-extension://bfogiafebfohielmmehodmfbbebbbpei/images/ico-field-fill-lock-disabled.svg&quot;) !important; background-size: 24px 24px !important; cursor: pointer !important; width: 24px !important; position: absolute !important; opacity: 1 !important; margin-top: auto !important; min-width: 24px !important; top: 7px; left: 281px; z-index: 1; padding: 0px; animation-name: none; height: 24px !important;"></keeper-lock></span></div></div><div data-se="o-form-fieldset" class="o-form-fieldset o-form-label-top margin-btm-0"><div data-se="o-form-input-container" class="o-form-input"><span data-se="o-form-input-remember" class="o-form-input-name-remember"><div class="custom-checkbox"><input type="checkbox" name="remember" id="input34"><label for="input34" data-se-for-name="remember" class="">Remember me</label></div></span></div></div></div></div><div class="o-form-button-bar"><input class="button button-primary" type="submit" value="Next" id="idp-discovery-submit" data-type="save"></div></form>

This would indicate there is a form called "form19"... But that doesn't seem to want to be addressed/shown.

Is there a way to find a specific HTML tag, or do I need BeautifulSoup for that?

As I am new to mechanize, I've tried multiple things I've found in my searches, but none have worked yet.

Solution

I have decided that Selenium is the way to go, and automate does not work for me. The particular page does not appear to use any forms.

Answered By - Bokkie