{ "metadata": { }, "nbformat": 4, "nbformat_minor": 5, "cells": [ { "id": "metadata", "cell_type": "markdown", "source": "
\n\n# Python - Globbing\n\nby [Helena Rasche](https://training.galaxyproject.org/hall-of-fame/hexylena/), [Donny Vrins](https://training.galaxyproject.org/hall-of-fame/dirowa/), [Bazante Sanders](https://training.galaxyproject.org/hall-of-fame/bazante1/)\n\nCC-BY licensed content from the [Galaxy Training Network](https://training.galaxyproject.org/)\n\n**Objectives**\n\n- How can I collect a list of files.\n\n**Objectives**\n\n- Use glob to collect a list of files\n- Learn about the potential pitfalls of glob\n\n**Time Estimation: 15M**\n
\n", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-0", "source": "

Globbing is the term used in computer science when we have a bunch of files and we want to list all of them matching some pattern.

\n
\n
Agenda
\n

In this tutorial, we will cover:

\n
    \n
  1. Setup
    2. \n
\n
\n

Setup

\n

We’ll start by creating some files for use in the rest of this tutorial

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-1", "source": [ "import os\n", "import subprocess\n", "\n", "dirs = ['a', 'a/b', 'c', 'c/e', 'd', '.']\n", "files = ['a.txt', 'a.csv', 'b.csv', 'b.txt', 'e.glm']\n", "\n", "for d in dirs:\n", " # Create some directories\n", " os.makedirs(d, exist_ok=True)\n", " # Create some files\n", " for f in files:\n", " subprocess.check_output(['touch', os.path.join(d, f)])" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "python" ], "id": "" } } }, { "id": "cell-2", "source": "

Now we should have a pretty full folder!

\n

Finding Files

\n

We can use the glob module to find files:

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-3", "source": [ "import glob\n", "print(glob.glob('*.csv'))\n", "print(glob.glob('*.txt'))" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "python" ], "id": "" } } }, { "id": "cell-4", "source": "

Here we use an asterisk (*) as a wildcard, it matches any bit of text (but not into folders!) to all matching files. Here we list all matching csv or txt files. This is great to find files matching a pattern.

\n

We can also use asterisks anywhere in the glob, it doesn’t just have to be the filename portion:

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-5", "source": [ "print(glob.glob('a*'))" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "python" ], "id": "" } } }, { "id": "cell-6", "source": "

Here we even see a third entry: the directory.

\n

Finding files in directories

\n

Until now we’ve found only files in a single top level directory, but what if we wanted to find files in subdirectories?

\n

Only need a single directory? Just include that!

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-7", "source": [ "print(glob.glob('a/*.csv'))" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "python" ], "id": "" } } }, { "id": "cell-8", "source": "

But if you need more levels, or want to look in all folders, then you need the double wildcard! With two asterisks ** we can search recursively through directories for files:

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-9", "source": [ "print(glob.glob('**/a.csv'))" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "python" ], "id": "" } } }, { "id": "cell-10", "source": "

Exercise

\n
\n
❓ Question: Where in the world is the CSV?
\n
    \n
  1. How would you find all .csv files?
    2. \n
  2. How would you find all .txt files?
    3. \n
  3. How would you find all files starting with the letter ‘e’?
    4. \n
\n
👁 View solution\n
👁 Solution
\n
    \n
  1. glob.glob('**/*.csv')
    2. \n
  2. glob.glob('**/*.txt')
    3. \n
  3. glob.glob('**/e*')
    4. \n
\n
\n
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-11", "source": [ "# Try things out here!" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "python" ], "id": "" } } }, { "id": "cell-12", "source": "

Pitfalls

\n

Some analyses (especially simultaions) can be dependent on data input order or data sorting. This was recently seen in Neupane et al. 2019 where the data files used were sorted one way on Windows, and another on Linux, resulting in different results for the same code and the same datasets! Yikes!

\n

If you know your analyses are dependent on file ordering, then you can use sorted() to make sure the data is provided in a uniform way every time.

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-13", "source": [ "print(sorted(glob.glob('**/a.csv')))" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "python" ], "id": "" } } }, { "id": "cell-14", "source": "

If you’re not sure if your results will be dependent, you can try sorting anyway. Or better yet, randomising the list of inputs to make sure your code behaves properly in any scenario.

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "cell_type": "markdown", "id": "final-ending-cell", "metadata": { "editable": false, "collapsed": false }, "source": [ "# Key Points\n\n", "- If your data is ordering dependent, sort your globs!\n", "\n# Congratulations on successfully completing this tutorial!\n\n", "Please [fill out the feedback on the GTN website](https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-glob/tutorial.html#feedback) and check there for further resources!\n" ] } ] }