Mar-13-2020, 07:59 PM
UPDATE
Wanted to post an update to see if there are any suggestions. I'm nearly there with my code, I was able to find something helpful where the 'Next' button is used instead of the individual 'gotoNextPage()' elements. However, it only appends the last page that it runs through. How can I append each page it clicks through to a master data frame?
Wanted to post an update to see if there are any suggestions. I'm nearly there with my code, I was able to find something helpful where the 'Next' button is used instead of the individual 'gotoNextPage()' elements. However, it only appends the last page that it runs through. How can I append each page it clicks through to a master data frame?
driver = webdriver.Chrome() driver.get('website') username = driver.find_element_by_id("username") password = driver.find_element_by_id("password") username.send_keys("username") password.send_keys("password"+"\n") while True: driver.implicitly_wait(30) table = driver.find_element_by_id('preblockBody') information = [] job_elems = table.find_elements_by_xpath("//*[contains(@class,'pbListingTable')]") for value in job_elems: #print(value.text) information.append(value.text) try: driver.find_element_by_partial_link_text('Next').click() except: break driver.quit() print(information)