pycnet.process package

Module contents

Defines functions to provide a higher-level interface for processing audio data using PNW-Cnet.

Functions:

batchImageData

Read image files from a folder and generate batches of the image data for classification using the PNW-Cnet model.

batchProcess

Perform a standard processing workflow on a list of folders read from a text file.

blockMessage

Format a message with borders and padding for emphasis.

buildProcQueue

Create a JoinableQueue defining .wav files to be processed and the directories where temporary spectrogram image files will be stored.

combineOutputFiles

Consolidate output files from multiple folders (e.g. as generated from a batch processing run).

generateClassScores

Generate class scores for a set of images using the PNW-Cnet model.

generateEmbeddings

Generate embeddings for a set of images using PNW-Cnet.

generateSpectrograms

Generate a set of spectrograms representing segments of the audio data.

getFullPaths

Reconstruct absolute paths to audio files listed in a DataFrame containing filenames and relative paths.

logMessage

Print a message and optionally write it to a log file.

parsePycnetArgs

Define command-line options for the ‘pycnet’ console script.

processFolder

Perform one or more processing operations on a folder.

pycnet.process.batchImageData(target_dir, batch_size=16, check_images=False)

Supply batches of image data from a folder for classification.

This function searches the target directory for image files with a .png file extension and chunks them up into batches to be supplied to the PNW-Cnet model. Images are loaded in batches and converted to NumPy arrays with pixel values rescaled to floating-point values in the range [0,1].

Parameters:

  • target_dir (str) – Path to the folder containing the images.

  • batch_size (int) – Number of images to process in each batch. Larger batches may allow faster processing at the cost of increased memory usage.

  • check_images (bool) – Use pycnet.file.image.checkImages to check for PNG files that cannot be loaded and omit them from the batched data.

Returns:

A dict containing “image_paths”, a list of the full paths of all .png images in the target directory; “image_names”, a list of the filenames of the same files; “image_batches”, a Keras DataFrameIterator; “images_checked”, a boolean indicating whether non-loadable images were omitted; and “bad_images”, a list of image paths that could not be loaded, or an empty list if check_images==False.

Return type:

dict

pycnet.process.batchProcess(target_dirs, cnet_version='v5', spectro_dir=None, n_workers=None, review_settings=None, log_to_file=False, show_prog=False, cleanup=False, check_images=True, combine_output=False, flac_mode=False)

Process several folders in sequence using the same parameters.

Essentially acts as a wrapper around processFolder with some basic exception handling. Available arguments are mostly the same as for processFolder.

Parameters:

  • target_dirs (list) – List of paths to folders containing data to be processed.

  • cnet_version (str) – Which version of the PNW-Cnet model to use when generating class scores (“v4” or “v5”).

  • spectro_dir (str) – Path to the folder where spectrograms should be stored.

  • n_workers (int) – How many worker processes to use when generating spectrograms.

  • review_settings (str) – Path to a CSV file with one column “Class” listing classes to include in the review file and one column “Threshold” listing the score threshold to use for each class -OR- a string containing class codes or groups of class codes followed by the score threshold to be used for each class or group.

  • log_to_file (bool) – Whether to copy console messages to a log file (does not include progress bars).

  • show_prog (bool) – Whether to show progress bars when generating spectrograms and classifying images with PNW-Cnet.

  • cleanup (bool) – Whether to delete spectrograms and temporary folders when processing is complete.

  • check_images (bool) – Whether to verify that all spectrogram images can be loaded before attempting to generate class scores. Skipping this step will save a bit of time but the program will crash if it encounters an unloadable image.

  • combine_output (bool) – Whether to create a set of combined output files from the output files of the individual folders that were processed.

  • flac_mode (bool) – Process FLAC files instead of WAV files.

Returns:

Nothing.

pycnet.process.blockMessage(message, pad_char='#', width=80)

Print a message with borders and padding for emphasis.

Parameters:

  • message (str) – Message to be printed emphatically.

  • pad_char (char) – Character to be used to pad the message.

  • width (int) – Width of the printed message.

  • print (bool) – Whether to print the formatted message in addition to returning it.

Returns:

The formatted message.

Return type:

str

pycnet.process.buildProcQueue(inv_df, input_dir, image_dir, n_chunks=0)

Return a queue of input file paths mapped to output folders.

If n_chunks is not supplied, the function will try to avoid generating >50,000 image files in any one folder.

Parameters:

  • inv_df (Pandas.DataFrame) – Dataframe containing basic information on audio files to be processed, as produced by pycnet.file.inventoryFolder.

  • input_dir (str) – Root of the directory tree containing audio files.

  • image_dir (str) – Path to the directory where spectrograms should be generated (in subfolders as needed).

  • n_chunks (int) – Number of subfolders to create for the spectrograms.

Returns:

A dictionary containing “queue” (the queue itself) and “spectro_dirs”, a list of folders where spectrograms will be generated.

Return type:

dict

pycnet.process.combineOutputFiles(target_dirs, cnet_version='v5', review_settings=None, keep_original_output=False, flac_mode=False)

Consolidate output files generated from a batch_process run.

This function is intended for combining output from folders representing subsets of a larger dataset, e.g. recording stations from a single site. The combined output files will be saved in the folder representing the lowest-level directory containing all the target directories, i.e. the longest common prefix that is a valid directory path. If there is no valid path shared among all the target directories, the function will print a warning message and return without combining anything.

Parameters:

  • target_dirs (list) – List of paths to folders that were processed.

  • cnet_version (str) – Version of PNW-Cnet that was used to generate class scores; either “v5” (default) or “v4”.

  • review_settings (str) – Path to a CSV file with one column “Class” listing classes to include in the review file and one column “Threshold” listing the score threshold to use for each class -OR- a string containing class codes or groups of class codes followed by the score threshold to be used for each class or group.

  • keep_original_output (bool) – Retain output files from target folders after combining their contents.

  • flac_mode (bool) – Assume audio files were .flac, not .wav.

Returns:

Nothing.

pycnet.process.generateClassScores(target_dir, model_path, show_prog=True, check_images=False)

Generate class scores for a set of spectrograms using PNW-Cnet.

The DataFrame returned contains one column Filename for the names of the image files and either 51 (PNW-Cnet v4) or 135 (PNW-Cnet v5) columns for class scores.

Parameters:

  • target_dir (str) – Directory containing spectrograms in the form of .png image files to be be classified using the PNW-Cnet model.

  • model_path (str) – Path to either the PNW-Cnet v4 or v5 trained model file.

  • show_prog (bool) – Whether to show a text-based progress bar as the model processes batches of images.

  • check_images (bool) – Use pycnet.file.image.checkImages to check for PNG files that cannot be loaded and omit them from the batched data.

Returns:

A DataFrame containing the class scores for each image file.

Return type:

Pandas.DataFrame

pycnet.process.generateEmbeddings(target_dir, cnet_version, show_prog=True, check_images=False)

Generate embeddings for a set of images using PNW-Cnet.

Embeddings are the activation of the penultimate fully-connected layer of the model. They are not directly interpretable in the same way as the class scores but might yield useful information via cluster analysis or other forms of dimensionality reduction, and can also be used to train linear classifiers.

Parameters:

  • target_dir (str) – Directory containing spectrograms in the form of .png image files to be be classified using the PNW-Cnet model.

  • cnet_version (str) – Which version of the PNW-Cnet model (“v4” or “v5”) to use for generating embeddings.

  • show_prog (bool) – Whether to show a text-based progress bar as the model processes batches of images.

  • check_images (bool) – Use pycnet.file.image.checkImages to check for PNG files that cannot be loaded and omit them from the batched data.

Returns:

A dictionary containing two Pandas DataFrames. “predictions” contains the class scores for each image file. “embeddings” contains the embeddings for each image file.

Return type:

dict

pycnet.process.generateSpectrograms(proc_queue, n_workers, show_prog=True)

Generate spectrogram image files from a queue of .wav files.

Parameters:

  • proc_queue (Multiprocessing.JoinableQueue) – A joinable queue containing tuples mapping paths of .wav audio files to folders where the spectrograms generated from each .wav file should be saved.

  • n_workers (int) – Number of worker processes to use for spectrogram generation.

  • show_prog (bool) – Whether to display a text-based progress bar as spectrograms are generated.

Returns:

Nothing.

pycnet.process.getFullPaths(inventory_df, top_dir)

Reconstruct full paths for wav files listed in the inventory.

Parameters:

  • wav_inventory (Pandas.DataFrame) – DataFrame listing the folder (relative to top_dir), filename, size, and duration of each .wav file.

  • top_dir (str) – Path of the root of the directory tree containing the .wav files listed in wav_inventory.

Returns:

A list of absolute paths for the .wav files listed in wav_inventory.

Return type:

list[str]

pycnet.process.logMessage(message, log_file_path=None)

Print a message and optionally write it to a log file.

Parameters:

  • message (str) – Message to be printed and written to file.

  • log_file_path (str) – Path to the log file, or None.

Returns:

Nothing.

pycnet.process.parsePycnetArgs()

Define command-line options for the ‘pycnet’ console script.

Parameters:

Nothing (reads arguments from stdin) –

Returns:

An argparse.Namespace object containing command-line arguments in an accessible form.

Return type:

argparse.Namespace

pycnet.process.processFolder(mode, target_dir, cnet_version='v5', spectro_dir=None, n_workers=None, review_settings=None, output_file=None, log_to_file=False, show_prog=True, cleanup=False, check_images=True, flac_mode=False)

Perform one or more processing operations on data in a folder.

Basically runs through the functions above in a logical sequence to produce a set of apparent target species detections from a set of .wav files. Narrates progress and marks the time at the beginning and end of each step to give the user a reasonable sense of processing speed.

Depending on the processing mode, this may create one or more comma-separated text output files within the target directory.

Parameters:

  • mode (str) – Processing mode, i.e., which operation(s) to perform.

  • target_dir (str) – Path to the folder containing audio data to be processed.

  • cnet_version (str) – Which version of the PNW-Cnet model to use when generating class scores (“v4” or “v5”).

  • spectro_dir (str) – Path to the folder where spectrograms should be stored.

  • n_workers (int) – How many worker processes to use when generating spectrograms.

  • review_settings (str) – Path to a CSV file with one column “Class” listing classes to include in the review file and one column “Threshold” listing the score threshold to use for each class -OR- a string containing class codes or groups of class codes followed by the score threshold to be used for each class or group.

  • output_file (str) – Name of the review file to be generated.

  • log_to_file (bool) – Whether to copy console messages to a log file (does not include progress bars).

  • show_prog (bool) – Whether to show progress bars when generating spectrograms and classifying images with PNW-Cnet.

  • cleanup (bool) – Whether to delete spectrograms and temporary folders when processing is complete.

  • check_images (bool) – Whether to verify that all spectrogram images can be loaded before attempting to generate class scores. Skipping this step will save a bit of time but the program will crash if it encounters an unloadable image.

  • flac_mode (bool) – Process .flac files instead of .wav files.

Returns:

Nothing.