Looping Over Data Sets

NoteQuestions

• How do I process many data files with the same code?

NoteObjectives

• Use glob.glob to build a list of filenames matching a pattern.
• Combine glob and a for loop to process batches of files.

Processing a list of files with a for loop

A filename is just a string, and strings can be stored in lists. That means you can loop over a list of filenames exactly as you would loop over a list of numbers, applying the same analysis to each file automatically:

import numpy
for filename in ["data/galileo_flat.csv", "data/galileo_ramp.csv"]:
    distance, height = numpy.loadtxt(filename, skiprows=2,
                                     comments="#", delimiter=',', unpack=True)
    print(filename, distance.min(), height.max())

data/galileo_flat.csv 800.0 1000.0
data/galileo_ramp.csv 253.0 1000.0

Finding files with glob

Writing out filenames by hand works for two files, but not for two hundred. The glob library provides a function, also called glob, that returns a list of all filenames matching a given pattern. The two most useful wildcards are * (zero or more characters) and ? (exactly one character):

import glob
print("all csv files in data/random:", glob.glob("data/random/*.csv"))

all csv files in data/random: []

If no files match the pattern, glob returns an empty list rather than an error:

print("all txt files in data/random:", glob.glob("data/random/*.txt"))

all txt files in data/random: []

Combining glob and for to process batches of files

Wrapping glob.glob in sorted gives you consistent, predictable ordering. Then it is straightforward to apply the same analysis to every matching file:

for filename in sorted(glob.glob('data/random/*.csv')):
    distance, height = numpy.loadtxt(filename, delimiter=',', unpack=True)
    print(filename, distance.mean(), height.std())

This pattern — glob a set of files, sort, loop and analyse — is one of the most useful idioms in data-analysis scripts. Name your data files systematically so that simple glob patterns can find the right set. The sort function always sorts alphabetically, so “A before”Z”, but also “10” before “2” but after “02”. Try to pad numbers so they always sort correctly.

CautionChallenge

Determining Matches

Which of these files is not matched by glob.glob('data/*as*.csv')?

1. data/gapminder_gdp_africa.csv
2. data/gapminder_gdp_americas.csv
3. data/gapminder_gdp_asia.csv

NoteSolution

File 1 is not matched. *as* requires the letters “as” to appear somewhere in the filename. “africa” does not contain “as”, while “americas” and “asia” both do.

CautionChallenge

Averaging across datasets

Write a program that calculates the average distance value across all files in data/random/, rather than printing each file separately.

NoteSolution

import glob
import numpy as np

all_distances = []
for filename in sorted(glob.glob('data/random/*.csv')):
    distance, height = numpy.loadtxt(filename, delimiter=',', unpack=True)
    all_distances.append(distance)

print("Overall mean distance:", np.mean(all_distances))

Overall mean distance: nan

/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/numpy/_core/fromnumeric.py:3824: RuntimeWarning: Mean of empty slice
  return _methods._mean(a, axis=axis, dtype=dtype,
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/numpy/_core/_methods.py:142: RuntimeWarning: invalid value encountered in scalar divide
  ret = ret.dtype.type(ret / rcount)

TipKey Points

• A filename is a string; lists of filenames can be used as loop collections.
• glob.glob(pattern) returns a list of filenames matching the pattern; * matches any sequence of characters.
• Wrapping glob.glob in sorted gives predictable ordering.
• Well-named, consistently structured files make glob patterns simple and reliable.