I learned a little bit about what multithreading is. So I wanted to implement a multithreaded reader that gets the temperature, signal, battery, and velocity from each row and calculates the total average of each data point.

I’m also hoping to plot the time it takes to run this program with 1 thread all the way up to 15 threads. I have 10 cores on my CPU: 6 efficiency, 4 performance, so it will be interesting to see where the gains level off.

The first thing I learned in Python was that you can’t do true multithreading. Because of their GIL, or Global Interpreter Lock, it will not run threads in parallel; instead, it will switch between threads.

So to achieve true parallelism, I had to import the multiprocessing library.

The second issue I ran into was that the overhead to create a new process exceeded the time gain that having multiple processes saved.

When I created 2, 3, 4 processes, even though they were each reading less lines—and therefore should take less time—the overall time was slower the more processes I created. I only have ~485 lines in my database, so perhaps the work being done (which results in times like 0.054) is too small to really compare time gains.

To fix this, I added in a time.sleep() in the function for each process, and I made the amount of time each process slept proportional to the number of processes. This mimics the idea that more processes split up the work and therefore take less time.

And it works!

Now, when I went up to 15 processes, the time kept decreasing, even though there are only 10 cores on my CPU. I realized the problem was with my time.sleep()—it was still proportional to the number of processes, when it should actually have been proportional to the number of processes up till 10, because the CPU supports up to 10 processes in parallel.

This fixed it. The time started to go up.

Here is a graph displaying the increased speed as the number of processes increases. You can see the time start to go up after 10.

I could probably have bypassed many of these issues by using a much larger file, but since I wanted specific data on my database, I had to be creative with adding extra “work” (in the form of sleeping).