preprocess run

Overview link

The preprocess run command is the workhorse of the preprocess CLI. It executes all the preprocessing operations defined in your Prepfile and transforms your raw data into cleaned, ready-to-use datasets.

Whether you’re preparing data for machine learning, analysis, or reporting, preprocess run applies all your specified transformations efficiently.

Basic Usage link

Quick Start link

Run preprocessing using your Prepfile:

  preprocess run --file Prepfile.toml
  

If you have a Prepfile in your current directory, you can simply run:

  preprocess run
  

The command will automatically look for Prepfile.toml in the current directory.

Command Reference link

  preprocess run [flags]
  

How It Works link

The run command follows this workflow:

1. Read Configuration: Loads your Prepfile to understand what operations to perform
2. Load Data: Reads your dataset using the specifications from the [data] section
3. Apply Operations: Executes all preprocessing operations in the order they’re defined
4. Postprocess: Applies any postprocessing steps (dropping columns, sorting, etc.)
5. Export Results: Saves the cleaned data to the specified output file

Practical Examples link

Example 1: Basic Execution link

You have a Prepfile in your current directory:

  preprocess run
  

This will:

• Find Prepfile.toml automatically
• Read your dataset
• Apply all preprocessing operations
• Save the cleaned data to the file specified in the [postprocess] section

Example 2: Specify Prepfile Path link

Your Prepfile is located elsewhere:

  preprocess run --file ./config/my_preprocessing.toml
  

Example 3: Quick Processing Without Prepfile link

For simple, one-off operations, you can specify everything via command line:

  preprocess run --data ./sales.csv --column revenue --op fillna:method=mean
  

This fills missing values in the revenue column with the mean.

Example 4: Apply Multiple Operations link

  preprocess run --data ./data.csv \
  --column age --op fillna:method=median \
  --column salary --op fillna:method=mean \
  --column salary --op scale:method=zscore
  

Note: Operations are applied in the order they appear on the command line.

Example 5: Process All Numeric Columns link

Apply operations to all numeric columns without specifying each one:

  preprocess run --data ./data.csv --numerics --op fillna:method=mean
  

This fills missing values in all numeric columns with their respective means.

Example 6: Show Differences link

Want to see what changed? Use the --show-diff flag:

  preprocess run --file my_prep.toml --show-diff
  

This opens a HTML page in your browser showing:

• Original vs. cleaned data comparison
• Statistics on changes made
• Visual indicators of modifications

Real-World Scenarios link

Scenario 1: Machine Learning Pipeline link

You’re preparing data for a machine learning model:

  # First, create your Prepfile
preprocess init --data ./training_data.csv --output ml_prep.toml

# Edit ml_prep.toml to include:
# - Missing value imputation
# - Feature scaling
# - Categorical encoding

# Then run the preprocessing
preprocess run --file ml_prep.toml
  

Your Prepfile might look like:

  [data]
filename = './training_data.csv'
csv_separator = ','
decimal_separator = '.'
encoding = 'utf-8'

[preprocess]
# Handle missing values in all numeric columns
[preprocess.numerics]
operations = [
    {op = "fillna", method = "median"}
]

# Scale all numeric features
[preprocess.numerics]
operations = [
    {op = "scale", method = "zscore"}
]

# Encode categorical variables
[[preprocess.columns]]
name = 'category'
type = 'string'
operations = [
    {op = "dummy"}
]

[postprocess]
format = 'csv'
filename = 'training_data_cleaned.csv'
  

Scenario 2: Data Cleaning for Analysis link

You need to clean survey data before analysis:

  preprocess init --data ./survey.csv --output survey_clean.toml
  

Edit to add:

• Text cleaning (trim whitespace, standardize case)
• Missing value handling
• Column renaming

Then run:

  preprocess run --file survey_clean.toml
  

Scenario 3: Automated Data Ingestion link

Set up a script to process new data files as they arrive:

  #!/bin/bash
# process_new_data.sh

# Generate timestamp
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

# Run preprocessing
preprocess run --file ./config/standard_prep.toml \
  --data ./new_data/incoming_$TIMESTAMP.csv \
  --output ./processed/cleaned_$TIMESTAMP.csv

# Log the operation
echo "Processed incoming_$TIMESTAMP.csv at $(date)" >> ./logs/processing.log
  

Make it executable:

  chmod +x process_new_data.sh
  

Advanced Usage link

Chaining Multiple Prepfiles link

For complex pipelines, you can chain multiple Prepfiles:

  # Step 1: Initial cleaning
preprocess run --file step1_cleaning.toml

# Step 2: Feature engineering
preprocess run --file step2_features.toml --data ./output_from_step1.csv

# Step 3: Final preparation
preprocess run --file step3_final.toml --data ./output_from_step2.csv
  

Using with Different Data Formats link

While preprocess works primarily with CSV files, you can process various formats:

  # Tab-separated values
preprocess run --data ./data.tsv --sep $'\t'

# Semicolon-separated (common in European data)
preprocess run --data ./data.csv --sep ";" --dsep ","

# Pipe-separated
preprocess run --data ./data.txt --sep "|"
  

Handling Large Datasets link

For large datasets, preprocess is optimized for performance:

  # Process a large file
preprocess run --file large_data_prep.toml
  

Tips for large datasets:

• Use --show-diff to verify changes without manually inspecting the entire dataset
• Process in batches if needed by creating multiple Prepfiles
• Ensure you have enough memory available

Understanding Operation Syntax link

Single Operation Format link

  --op operation:method=value
  

Examples:

• --op fillna:method=mean - Fill missing values with mean
• --op fillna:method=median - Fill missing values with median
• --op scale:method=zscore - Standardize (z-score normalization)
• --op scale:method=minmax - Min-max normalization
• --op clean:method=trimws - Trim whitespace
• --op clean:method=lower - Convert to lowercase

Multiple Operations on Same Column link

  --column revenue \
  --op fillna:method=mean \
  --op scale:method=zscore
  

This applies operations sequentially: first fill missing values, then scale.

Operation Methods Reference link

Fill Missing Values (fillna) link

Scaling (scale) link

Text Cleaning (clean) link

Other Operations link

Best Practices link

1. Start with a Prepfile link

While you can use command-line flags for simple operations, for anything more complex:

  # Do this:
preprocess init --data ./data.csv --output my_config.toml
# Edit my_config.toml
preprocess run --file my_config.toml

# Instead of this (for complex operations):
preprocess run --data ./data.csv --column col1 --op op1 --column col2 --op op2 ...
  

2. Use --show-diff for Verification link

Always verify your preprocessing with --show-diff:

  preprocess run --file my_prep.toml --show-diff
  

This helps you:

• Confirm operations are working as expected
• Catch configuration errors early
• Understand the impact of your preprocessing

3. Test Incrementally link

Add operations one at a time and verify each step:

  # Step 1: Add fillna for one column
# Test with --show-diff

# Step 2: Add scaling
# Test again

# Step 3: Add more operations
# Verify
  

4. Document Your Pipeline link

Add comments to your Prepfile explaining each operation:

  [preprocess]
# Data cleaning phase
[preprocess.numerics]
operations = [
    {op = "fillna", method = "median"}  # Handle missing values first
]

# Feature scaling for ML
[preprocess.numerics]
operations = [
    {op = "scale", method = "zscore"}  # Standardize for algorithm compatibility
]
  

Troubleshooting link

Common Issues link

Issue: Prepfile not found

  Error: Failed to load config file 'Prepfile.toml'
  

Solution: Specify the path explicitly:

  preprocess run --file ./path/to/your/Prepfile.toml
  

Issue: Dataset not found

  Error: Failed to read dataset
  

Solution: Verify the path in your Prepfile or use --data flag:

  preprocess run --file my_prep.toml --data ./correct/path/to/data.csv
  

Issue: Column not found

  Error: Column 'column_name' not found in dataset
  

Solution: Check for typos in your column names. Remember:

• Column names are case-sensitive
• Verify column names with preprocess skim --data ./your_data.csv

Issue: Type mismatch

  Error: Operation not applicable to column type
  

Solution: Ensure you’re applying the right operations to the right column types:

• Numeric operations (fillna:mean, scale) only work on numeric columns
• Text operations (clean, dummy) only work on text columns

Debugging Tips link

1. Verify your data first:

     preprocess skim --data ./your_data.csv
     

2. Check your Prepfile syntax:

   • Ensure proper TOML formatting
   • Verify all required sections are present

3. Test with minimal operations:

     # Test with just one simple operation
   preprocess run --data ./data.csv --column col1 --op fillna:method=mean
     

4. Use --show-diff to see what’s happening:

     preprocess run --file my_prep.toml --show-diff
     

Performance Tips link

Optimizing Processing Time link

1. Exclude unnecessary columns:

     [preprocess.numerics]
   exclude_columns = ["id", "timestamp", "unnecessary_col"]
     

2. Process in the right order:

   • Apply operations that reduce data size first (dropping columns)
   • Then apply transformations
   • Finally apply scaling

3. Use appropriate data types:

   • Ensure numeric columns are detected as int or float, not string

Memory Usage link

For very large datasets:

• Process in batches if possible
• Close other memory-intensive applications
• Consider using --show-diff instead of opening the full dataset

Next Steps link

After running preprocessing:

1. Verify with summary: Generate statistics to confirm your data is clean
2. Explore the output: Use preprocess skim on your cleaned data
3. Compare with original: Use preprocess diff to compare original and cleaned data
4. Iterate on your Prepfile: Refine your preprocessing pipeline

Quick Reference Card link

Supported Operations Summary link