Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Multi-threading
#1
Can I use Multi-threading to scrape images from internet fast
If yes can anyone Explain me how it works
Can't find anything related this on Internet or Youtube


for link in tqdm(images):
    filename = link.split("/")[-1]
    urllib.request.urlretrieve(link, filename)
Reply
#2
Good Morning,

I followed a really good tutorial on Youtube.

https://www.youtube.com/watch?v=IEEhzQoKtQU
Reply
#3
(Jul-15-2020, 05:26 AM)Evil_Patrick Wrote: Can I use Multi-threading to scrape images from internet fast
There is one example in Web-scraping part-2
So take time from 1,10 minute no Multi-threading,down to 10-sec for 200-images downloads.
So there use ProcessPoolExecutor loading 20 processes so quite heavy for this task,but it's blazing fast Roll Eyes
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Concurrent futures threading running at same speed as non-threading billykid999 13 1,879 May-03-2023, 08:22 AM
Last Post: billykid999
  Tutorials on sockets, threading and multi-threading? muzikman 2 2,132 Oct-01-2021, 08:32 PM
Last Post: muzikman
  Need help multi-threading scraping spacedog 2 2,494 Apr-28-2021, 03:48 PM
Last Post: spacedog
  Embedding python cause crash when use boost::asio multi threading udvatt108 0 1,730 Oct-04-2020, 03:15 PM
Last Post: udvatt108
  object base multi threading maboobelahi93 0 1,443 Jan-29-2020, 11:21 AM
Last Post: maboobelahi93
  multi-threading error in minimal script Skaperen 2 4,681 Aug-03-2019, 07:58 PM
Last Post: Skaperen
  Problem with Python, MySQL and Multi-threading queries zagk 1 11,908 Jul-01-2017, 12:15 AM
Last Post: zagk
  SSH Multi Threading nizami22247 0 9,617 Mar-31-2017, 06:04 PM
Last Post: nizami22247

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020