Datalab

---

Updates

https://stackoverflow.com/questions/37083116/multicore-and-multithread-on-ipython-notebook
https://stackoverflow.com/questions/32081926/a-new-thread-for-running-a-cell-in-ipython-jupyter-notebook

from IPython.display import display
from ipywidgets import IntProgress

import threading

class App(object):
    def __init__(self, nloops=2000):
        self.nloops = nloops
        self.pb = IntProgress(description='Thread loops', min=0, max=self.nloops)

    def start(self):
        display(self.pb)
        while self.pb.value < self.nloops:
            self.pb.value += 1 
        self.pb.color = 'red'

app = App(nloops=20000)

t = threading.Thread(target=app.start)

t.start()

How to use threading?

from multiprocessing import Pool
from multiprocessing.dummy import Pool as ThreadPool 
pool = ThreadPool(4) 
results = pool.map(my_function, my_array)

Map is a cool little function, and the key to easily injecting parallelism into your Python code. For those unfamiliar, map is something lifted from functional languages like Lisp. It is a function which maps another function over a sequence.

Map handles the iteration over the sequence for us, applies the function, and stores all of the results in a handy list at the end.

import urllib2 
from multiprocessing.dummy import Pool as ThreadPool 

urls = [
  'http://www.python.org', 
  'http://www.python.org/about/',
  'http://www.onlamp.com/pub/a/python/2003/04/17/metaclasses.html',
  'http://www.python.org/doc/',
  'http://www.python.org/download/',
  'http://www.python.org/getit/',
  'http://www.python.org/community/',
  'https://wiki.python.org/moin/',
]

# make the Pool of workers
pool = ThreadPool(4) 

# open the urls in their own threads
# and return the results
results = pool.map(urllib2.urlopen, urls)

# close the pool and wait for the work to finish 
pool.close() 
pool.join() 

https://stackoverflow.com/questions/2846653/how-to-use-threading-in-python

Passing multiple arguments (works like this only in Python 3.3 and later):

To pass multiple arrays:

results = pool.starmap(function, zip(list_a, list_b))

or to pass a constant and an array:

results = pool.starmap(function, zip(itertools.repeat(constant), list_a))

Checking row with na value

df[df.isna().any(axis=1)]

Cloud Datalab backup

Cloud Datalab instances periodically back up user content to a Google Cloud Storage bucket in the user's project to prevent accidental loss of user content in case of a failed or deleted VM disk. By default, a Cloud Datalab instance stores all of the user’s content in an attached disk, and the backup utility works on this disk’s root. The backup job is run every ten minutes, creates a zip file of the entire disk, compares it to the last backup zip file, uploads the zip if there’s a difference between the two and if sufficient time has elapsed between the new changes and the last backup. Cloud Datalab uploads the backup files to Google Cloud Storage.

Cloud Datalab retains the last 10 hourly backups, 7 daily backups, and 20 weekly backups, and deletes older backup files to preserve space. Backups can be turned off by passing the --no-backups flag when creating a Cloud Datalab instance with the datalab create command.

https://cloud.google.com/datalab/docs/how-to/working-with-notebooks#cloud_datalab_backup

Due to datalab gs backup, it will result in high cpu usage. Datalab gs backup can be disabled by setting:

DATALAB_SETTINGS_OVERRIDES={             
  "enableAutoGCSBackups": false,
  ...
}

Multi-threading

Installation

!pip install joblib

Example,

from joblib import Parallel, delayed

filenames = sorted(glob.glob(mdFmt, recursive=True))
data = Parallel(n_jobs=-1, verbose=5)(delayed(runTask)(file) for file in
                sorted(filenames))

# unwrap the data
data = [item for sublist in data for item in sublist]