Examples

You can find the examples detailed on this page and more in the examples/ directory of SCOOP.

Please check the API Reference for any implentation detail of the proposed functions.

Introduction to the map() function

A core concept of task-based parallelism as presented in SCOOP is the map. An introductory example to map working is presented in examples/map_doc.py.

1from __future__ import print_function
2from scoop import futures
3
4def helloWorld(value):
5    return "Hello World from Future #{0}".format(value)
6
7if __name__ == "__main__":
8    returnValues = list(futures.map(helloWorld, range(16)))
9    print("\n".join(returnValues))

Line 1 allows Python 2 users to have a print function compatible with Python 3.

On line 2, SCOOP is imported.

On line 4-5, the function that will be mapped is declared.

The condition on line 7 is a safety barrier that prevents the main program to be executed on every workers. It ensures that the map is issued only by one worker, the root.

The map() function is located on line 8. It launches the helloWorld function 16 times, each time with a different argument value selected from the range(16) argument. This method is compatible with the standard Python map() function and thus can be seamlessly interchanged without modifying its arguments.

The example then prints the return values of every calls on line 9.

You can launch this program using python -m scoop. The output should look like this:

~/scoop/examples$ python -m scoop -n 8 map_doc.py
Hello World from Future #0
Hello World from Future #1
Hello World from Future #2
[...]

Note

Results of a map are always ordered even if their computation was made asynchronously on multiple computers.

Note

You can toy around with the previous example by changing the second parameter of the map() function. Is it working with string arrays, pure strings or other variable types?

Computation of \pi

A Monte-Carlo method to calculate \pi using SCOOP to parallelize its computation is found in examples/pi_calc.py. You should familiarize yourself with Monte-Carlo methods before going forth with this example.

Monte Carlo computation of Pi.

Image from Wikipedia made by CaitlinJo that shows the Monte Carlo computation of \pi.

First, we need to import the needed functions as such:

1from math import hypot
2from random import random
3from scoop import futures

The Monte-Carlo method is then defined. It spawns two pseudo-random numbers that are fed to the hypot function which calculates the hypotenuse of its parameters. This step computes the Pythagorean equation (\sqrt{x^2+y^2}) of the given parameters to find the distance from the origin (0,0) to the randomly placed point (which X and Y values were generated from the two pseudo-random values). Then, the result is compared to one to evaluate if this point is inside or outside the unit disk. If it is inside (have a distance from the origin lesser than one), a value of one is produced (red dots in the figure), otherwise the value is zero (blue dots in the figure). The experiment is repeated tries number of times with new random values.

The function returns the number times a pseudo-randomly generated point fell inside the unit disk for a given number of tries.

1def test(tries):
2    return sum(hypot(random(), random()) < 1 for _ in range(tries))

One way to obtain a more precise result with a Monte-Carlo method is to perform the method multiple times. The following function executes repeatedly the previous function to gain more precision. These calls are handled by SCOOP using it’s map() function. The results, that is the number of times a random distribution over a 1x1 square hits the unit disk over a given number of tries, are then summed and divided by the total of tries. Since we only covered the upper right quadrant of the unit disk because both parameters are positive in a cartesian map, the result must be multiplied by 4 to get the relation between area and circumference, namely \pi.

1def calcPi(nbFutures, tries):
2    expr = futures.map(test, [tries] * nbFutures)
3    return 4. * sum(expr) / float(nbFutures * tries)

As previously stated, you must wrap your code with a test for the __main__ name.

1if __name__ == "__main__":
2    print("pi = {}".format(calcPi(3000, 5000)))

You can now run your code using the command python -m scoop.

Sharing Constant

One usage of shared constants is to halt a computation when a worker has found a solution such as a brute forcing example.

 1from itertools import product
 2import string
 3from scoop import futures, shared
 4
 5# Create the hash to brute force
 6HASH_TO_FIND = hash("SCO")
 7
 8
 9def generateHashes(iterator):
10    """Compute hashes of given iterator elements"""
11    for combination in iterator:
12        # Stop as soon as a worker finds the solution
13        if shared.getConst('Done', timeout=0):
14            return False
15
16        # Compute the current combination hash
17        currentString = "".join(combination).strip()
18        if hash(currentString) == HASH_TO_FIND:
19            # Share to every other worker that the solution has been found
20            shared.setConst(Done=True)
21            return currentString
22    
23    # Report that computing has not ended
24    return False
25
26
27if __name__ == "__main__":
28    # Generate possible characters
29    possibleCharacters = []
30    possibleCharacters.extend(list(string.ascii_uppercase))
31    possibleCharacters.extend(' ')
32
33    # Generate the solution space.
34    stringIterator = product(possibleCharacters, repeat=3)
35
36    # Partition the solution space into iterators
37    # Keep in mind that it evaluates the whole solution space
38    # making it pretty memory inefficient.
39    SplittedIterator = [stringIterator for _ in range(1000)]
40
41    # Parallelize the solution space evaluation
42    results = futures.map(generateHashes, SplittedIterator)
43
44    # Loop until a solution is found
45    for result in results:
46        if result:
47            break
48
49    print(result)

Overall example

The examples/fullTree.py example holds a wrap-up of available SCOOP functionnalities. It notably shows that SCOOP is capable of handling twisted and complex hierarchical requirements.

Getting acquainted with the previous examples is fairly enough to use SCOOP, no need to dive into this complicated example.