Apr-13-2024, 09:36 PM
I am using python to do stock backtesting for day trading.
I have a function called process_date(datedf), this is passed a pandas dataframe.
This function backtests the strategy across a single day, and returns me another pandas dataframe with the desired results.
Because this is day trading, each day is completely independent from the next so I parallelized my code to run multiple days at once using joblib, and it works great.
However, I want to take it to the next step - my main computer only has 16 cores, it's the fastest one I have but I have a menagerie of other computers strewn about. So my thought was what if I can parallelize this process_date function across multiple computers using dask distributed. Just like how joblib does on one computer, but now I'm also using the cores in multiple other computers at once.
I've never used dask before, but my understanding is that each function needs to be passed a dask dataframe. I could convert the pandas dataframe to a dask dataframe before passing it - but then would all my pandas functions inside of process_date() still work? I read that each partition of a dask dataframe is essentially a pandas dataframe, so my understanding is it should work, but I'm seeking confirmation.
Also, I do make use of some global variables within the function. This works fine with joblib, but would this work with dask across multiple machines? Or would I need to rewrite that code?
Let me know if conceptually my idea would work, or not. Thanks.
I have a function called process_date(datedf), this is passed a pandas dataframe.
This function backtests the strategy across a single day, and returns me another pandas dataframe with the desired results.
Because this is day trading, each day is completely independent from the next so I parallelized my code to run multiple days at once using joblib, and it works great.
However, I want to take it to the next step - my main computer only has 16 cores, it's the fastest one I have but I have a menagerie of other computers strewn about. So my thought was what if I can parallelize this process_date function across multiple computers using dask distributed. Just like how joblib does on one computer, but now I'm also using the cores in multiple other computers at once.
I've never used dask before, but my understanding is that each function needs to be passed a dask dataframe. I could convert the pandas dataframe to a dask dataframe before passing it - but then would all my pandas functions inside of process_date() still work? I read that each partition of a dask dataframe is essentially a pandas dataframe, so my understanding is it should work, but I'm seeking confirmation.
Also, I do make use of some global variables within the function. This works fine with joblib, but would this work with dask across multiple machines? Or would I need to rewrite that code?
Let me know if conceptually my idea would work, or not. Thanks.