How to find element in a deeply nested html structure

How to find element in a deeply nested html structure - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html)
+--- Thread: How to find element in a deeply nested html structure (/thread-40854.html)

How to find element in a deeply nested html structure - Orientation_group - Oct-04-2023

Hi, I have a deeply nested element (the 11.7 value underlined) as shown below. May I know how can I find this element?

<div class="col-lg-6 ">
<div class="form-group ">
<label>Size</label>

<span data-val controltovalidate="MainPlaceHolder_Size " data-val-focusOnError="t " data-val-errormessage="(Required field*)" id="MainPlaceHolderFormCtrl_Size " data-val="true " data-val-evaluationfunction="RequiredFieldValidatorEvaluateIsValid " data-val-initialvalue="" style="color:Red;visibility:hidden;">(Required field*)</span>
<input name="MainPlaceHolderContentPlaceHolder" type="text " id="MainPlaceHolder_Size " value="N.A " class="form-control " style="display:none;" />
<input name="MainPlaceHolderContentPlaceHolderSize " type="text " id="MainPlaceHolder_ContentPlaceHolderSize " class="form-control " value="11.7 " />
</div>
</div>
</div>

I have tried finding element by ID,class name,XPath but nothing works..

RE: How to find element in a deeply nested html structure - Larz60+ - Oct-04-2023

Please show URL, need to see full context of webpage.

RE: How to find element in a deeply nested html structure - Orientation_group - Oct-06-2023

(Oct-04-2023, 10:20 AM)Larz60+ Wrote: Please show URL, need to see full context of webpage.

Sry I cant share the website as it requires a login access but I can share a snippet of the code section I extracted from Chrome developer tools.
[Image: Capture.jpg]

I have successfully logged in on Python but cant seemed to extract the element containing the value 11.7 as underlined in red as shown in the snippet above. I tried finding element by id, class, and even input name but to no avail..

RE: How to find element in a deeply nested html structure - Larz60+ - Oct-06-2023

Please, at least include the snippit as text that can be copied (need to test our suggestions)
It's too much to type from scratch as there are so many threads that we need to look at.

RE: How to find element in a deeply nested html structure - Orientation_group - Oct-09-2023

(Oct-06-2023, 08:26 AM)Larz60+ Wrote: Please, at least include the snippit as text that can be copied (need to test our suggestions)
It's too much to type from scratch as there are so many threads that we need to look at.

Here is the text version of the snippet:
<div class="col-lg-6 ">
<div class="form-group ">
<label>Size* </label>
<span data-val-controltovalidate="MainPlaceHolderSize “ data-val-focusOnError="t " data-val-errormessage="(Required field*)" id="MainPlaceHolderFormCtrl_Size " data-val="true " data-val-evaluationfunction="RequiredFieldValidatorEvaluateIsValid " data-val-initialvalue="" style="color:Red;visibility:hidden;">(Required field*)</span>
<input name="ctl00$ctl00$MainPlaceHolderContentPlaceHolder" type="text " id="MainPlaceHolder_SIze" value="N.A " class="form-control " style="display:none;" />
<input name="ctl00$ctl00$MainPlaceHolderSize " type="text " id="MainPlaceHolder_ContentPlaceHolderSize " class="form-control " value="11.7 " />
</div>
</div>

Thank you very much in advance!

RE: How to find element in a deeply nested html structure - Larz60+ - Oct-09-2023

You don't need all of this code, but there's some additional to help you understand how to parse similar html in the future:

from bs4 import BeautifulSoup

data = '''<div class="col-lg-6 ">
<div class="form-group ">
<label>Size* </label>
<span data-val-controltovalidate="MainPlaceHolderSize “ data-val-focusOnError="t " data-val-errormessage="(Required field*)" id="MainPlaceHolderFormCtrl_Size " data-val="true " data-val-evaluationfunction="RequiredFieldValidatorEvaluateIsValid " data-val-initialvalue="" style="color:Red;visibility:hidden;">(Required field*)</span>
<input name="ctl00$ctl00$MainPlaceHolderContentPlaceHolder" type="text " id="MainPlaceHolder_SIze" value="N.A " class="form-control " style="display:none;" />
<input name="ctl00$ctl00$MainPlaceHolderSize " type="text " id="MainPlaceHolder_ContentPlaceHolderSize " class="form-control " value="11.7 " />
</div>
</div>'''

def find_value(data):
    soup = BeautifulSoup(data, "html.parser")
    itags = soup.find_all("input")
    value = itags[1].get('value')
    print(f"value: {value}")

find_value(data)


# you don't need this one, but may help you to understand breakdown
def show_elements(data):
    soup = BeautifulSoup(data, "html.parser")
    print(f"\nlist of all tags:\n{[tag.name for tag in soup.find_all()]}")
    itags = soup.find_all("input")
    print(f"\nattributes of 2nd input statement:\n{list(itags[1].attrs)}")

show_elements(data)

Output:value: 11.7 

list of all tags:
['div', 'div', 'label', 'span', 'input', 'input']

attributes of 2nd input statement:
['name', 'type', 'id', 'class', 'value']