Failure in web scraping by Beautiful Soup

[metulburr]

You can run document.cookie in your console to read all the cookies accessible from that location.

document.write("Not all needed JavaScript methods are supported.<BR>");

<noscript>JavaScript must be enabled in order to view this page.</noscript>

Its possible they changed their site to include javascript? If so, then it would stop requests in in tracks. If it does have javascript you are going to need Selenium to accomplish your task instead. It doesnt have to be the main page, any portion of information you are getting could be obtained via javascript. I often have to change my scripts as admins change the HTML or add javascript to avoid bots.

It is also entirely possible they have detected your bot as you have a fair number of requests (making a request per entry / per page). They can rate-limit in iptables to greatly reduce the request volume per source.

However based on your first post, i would suggest either cookies or javascript is the issue.

using selenium does gt the html by the way without much hassle

from selenium import webdriver

race_day_url='https://racing.hkjc.com/racing/info/meeting/Results/English/Local/'

browser = webdriver.Firefox()
browser.get(race_day_url)
time.sleep(3)
print(browser.page_source)

Hide/Show

Output:
...

<div class="content"><a class="first" href="//www.hkjc.com/english/corporate/faq.asp" target="_blank">FAQ</a> | <a href="javascript:NewWindow('//www.hkjc.com/infomenu/en/contact/feedback.aspx','mypopup','770','550','yes','yes');">Contact Us</a> | <a href="javascript:NewWindow('//mcs.hkjc.com/','mypopup','770','550','yes','yes');">Media Communication System</a> | <a href="//www.hkjc.com/english/corporate/corp_sitemap.asp">Site Map</a> | <a href="javascript:NewWindow('//www.hkjc.com/english/betting/betting_rule.htm','mypopup','770','550','yes','yes');">Rules </a>| <a href="//www.hkjc.com/responsible-gambling/en/index.aspx" target="_blank">Responsible Gambling Policy</a> | <a href="//www.hkjc.com/english/corporate/corp_privacy.asp">Privacy</a> | <a href="//www.hkjc.com/english/corporate/corp_disclaimer.asp">Disclaimer</a> | <a href="//www.hkjc.com/home/english/corporate/security-tips.aspx">Security Tips</a><br></div>
<div class="copyright">Copyright © 2000-2019 The Hong Kong Jockey Club. All rights reserved.</div>
</div></div><script src="//common.hkjc.com/wa/wa_tracker.js" type="text/javascript"></script><script src="//common.hkjc.com/wa/wa_tracker_config.js" type="text/javascript"></script><script type="text/javascript">
		var waTaggingFired = false;

		var firedfunc = function() {
			if(waTaggingFired){
				return;
			}
						
			//Original WA tagging logic goes here
			
					try {
						WATracker.trackPageView('racing-content');
					} catch (e) {}
						
		   
			waTaggingFired = true;
		};

		if ( typeof(SSO) == 'undefined') {
		}
		else {
			SSO.OnUserProfileLoaded(firedfunc);
		}
	</script>

            
        </footer>
        
    

<iframe sandbox="allow-scripts allow-same-origin" title="Adobe ID Syncing iFrame" id="destination_publishing_iframe_hkjcweb_0" name="destination_publishing_iframe_hkjcweb_0_name" style="display: none; width: 0px; height: 0px;" src="https://hkjcweb.demdex.net/dest5.html?d_nsid=0#https%3A%2F%2Fracing.hkjc.com%2Fracing%2Finformation%2FEnglish%2Fracing%2FLocalResults.aspx" class="aamIframeLoaded"></iframe></body></html>