¶ Custom Transformation - Documentation
¶ Overview
Custom Transformations allow you to apply complex data transformations to your CSV and JSON files using JavaScript. The feature now includes AI-powered code generation and support for multiple input and output files, making it easier to create sophisticated data processing workflows without manual coding.
¶ Key Features
¶ 🤖 AI-Powered Code Generation
Generate transformation code automatically by describing what you want to accomplish in plain English. The AI understands your requirements and creates production-ready JavaScript code.
¶ 📁 Multiple Input Files
Process multiple CSV or JSON files simultaneously in a single transformation. Each file can be referenced by a custom variable name in your code.
¶ 📂 Multiple Output Files
Create multiple output files from a single transformation. Split your data by category, create summary reports, or generate different views of your data—all in one operation.
¶ 📚 Built-in Libraries
Access powerful data manipulation libraries:
- Lodash: Comprehensive utility functions for data transformation
- Day.js: Modern date and time manipulation
¶ Getting Started
¶ Step 1: Create a New Transformation
- Navigate to the Custom Transformations section
- Click Add Transformation
- The transformation builder modal will open
¶ Step 2: Select Input Files
File Selection:
- Click the file selector dropdown
- Choose one or more CSV or JSON files from your data folder
- Multiple files can be selected for complex transformations
Variable Name Assignment: Each selected file needs a unique variable name that you'll use in your transformation code:
- Default names are automatically assigned (file1, file2, etc.)
- You can customize these names to be more meaningful
- Variable names must be alphanumeric (letters and numbers only)
- Names must be unique across all selected files
Example Setup:
- customers.csv → Variable name: "customers"
- orders.csv → Variable name: "orders"
- products.csv → Variable name: "products"
¶ Step 3: Configure Delimiter (Optional)
If your CSV files use a delimiter other than comma:
- Enter your custom delimiter in the delimiter field
- Common alternatives: semicolon (;), pipe (|), tab
- Default is comma (,) if left empty
¶ Step 4: Generate or Write Transformation Code
You have two options:
¶ Option A: AI Code Generation (Recommended)
Describe Your Transformation: In the AI prompt field, describe what you want to accomplish in plain English
AI Generates Code: Click the "Generate" button and the AI will create the transformation code for you
Review and Accept:
- Preview the generated code in a modal
- Review to ensure it meets your requirements
- Click "Replace Code" to use the generated code
AI Prompt Examples:
Single Output File:
- "Add a row number to each record"
- "Filter records where status is active"
- "Calculate total amount for each customer"
- "Combine first name and last name into full name"
Multiple Output Files:
- "Split data into active and inactive customers in separate files"
- "Create three outputs: orders by region (North, South, West)"
- "Generate a processed data file and a summary statistics file"
- "Separate records by year and create a file for each year"
¶ Option B: Manual Code Writing
Write your transformation code directly in the code editor:
- Use the provided code editor with syntax highlighting
- Reference your input files by their variable names
- Return either a single array or an object of arrays
¶ Step 5: Test Your Transformation
Before saving, always test your transformation:
- Click the Test button
- The system loads sample data from your selected files (first 50 rows)
- Executes your transformation code
- Displays results in an interactive table
Test Results Display:
Single Output:
- Shows total record count
- Displays data in a sortable table
- Preview all columns and values
Multiple Outputs:
- Shows number of output files created
- Tabs for each output file
- Record count for each file
- Individual tables for each output
¶ Step 6: Save Your Transformation
- After successful testing, click Next
- Enter a Name for your transformation (alphanumeric only)
- Enter a Description to explain what the transformation does
- Click Save
Your transformation is now saved and ready to use in your workflows!
¶ Understanding Output Formats
¶ Single Output File
When you want one processed output file:
- Your code returns a single array of objects
- Creates one CSV file with your transformed data
- The output file uses your specified name or defaults to the original filename
When to Use:
- Simple filtering or field modifications
- Adding calculated columns
- Deduplicating records
- Basic data cleanup
¶ Multiple Output Files (New Feature)
When you need to create multiple files:
- Your code returns an object where each property is an output file
- Each property name becomes the output filename
- Each property value must be an array of objects
- Creates a ZIP file containing all outputs
When to Use:
- Splitting data by category, region, or status
- Creating separate files for different time periods
- Generating data file + summary/report file
- Distributing data to different departments
Output File Naming:
- Use descriptive property names in your code
- Names are automatically sanitized for file system compatibility
- You can configure custom output names in the workflow configuration
- Default format: originalname_outputkey.csv
¶ Using in Workflows
¶ Basic Configuration
- Add a Custom Transformation task to your workflow
- Select your saved transformation from the dropdown
- Choose the input file for processing
- Configure output file name
- Run your workflow
¶ Best Practices
¶ Writing Effective AI Prompts
Be Specific:
- ❌ "Transform my data"
- ✅ "Filter records where amount is greater than 1000 and add a priority flag"
Mention Multiple Outputs Explicitly:
- ❌ "Split by region"
- ✅ "Create separate output files for each region (North, South, East, West)"
Include Business Logic:
- ❌ "Process the data"
- ✅ "Calculate days since last order and flag customers inactive for more than 90 days
¶ Conclusion
The enhanced Custom Transformation feature combines the power of AI-assisted development with flexible multi-file processing capabilities. Whether you need simple data filtering or complex multi-output transformations, the intuitive interface and AI assistance make it accessible while maintaining the flexibility for advanced use cases.