Basics of multiprocess with python with the requests lib

1. Build a list of URL that you want to get.

   project_url_list_pages = ['www.potato.com', 'www.potato1.com', 'www.potato2.com']
   

2. Create your pool of workers. You can hardcode the number or get it from the CPU amount

   import multiprocessing
   pool = Pool(processes=32)
   multiprocessing.cpu_count(multiprocessing.cpu_count())
   

3. Map the function you need to speed up to a pool of workers.

   import requests
   
   def get_url_multiprocess(url):
   data = requests.get(url).json()
   return data
   
   

   #pool.map($FUNCTION, $PARAMETER_TO_ASS_TO_FUNCTION)
   pool_outputs = pool.map(get_url_multiprocess,
                       project_url_list_pages)