Generating A LOT of images. 1M+



  • I was asked to generate a lot of images.
    Definitely more than 1M... I'd say around 33 554 432 specifically ^^'.

    I'm running drawBot as a module that i set up in a virtualenv for this.

    I'm pretty happy with the performance of my script. It generates around 100 images/second and if i calculate correctly it will finish in about 90 hours...

    So i opened the activity monitor and noticed that Python is maxed out at 100% since it only uses one core (thread?). I'm not familiar with the topic... What would be the best way to utilise all CPU power? My CPU (i7-6820HQ) has 4 cores and each has 2 threads i guess? So that would be potentially 8x faster?

    My ideas:

    Idea #1:
    I can split the script in parts and run 4 scripts at once.

    Concern:
    Can i use the same virtualenv Python installation in 4 terminal windows?
    Or do i have to set up 4 virtualenvs?

    Idea #2:
    Use another machine

    Concern:
    I only have one.

    I'm also concerned about the increasing number of memory used by Python. Right now it's at 100MB and it's increasing proportionally the more files i generate. 33M files would take around 30GB in this case.

    What would be the best way to approach this?