Orchestrating File Processing and Archiving Using appRules File Iteration Activities

A common requirement in IT shops is the ability to quickly process lots of files in a manageable and predictable manner.  CSV, Excel, Access, XML, JSON are some of the file types that you may be asked to process and archive.

We recently provided a project snippet to a customer that was migrating hundreds of Excel and Access files from multiple locations into a centralized MySQL database.   In this post, we will use this opportunity to discuss the project snippet and the file iteration activities in appRules.

A majority of the activities that were used in the snippet come from the appRules File System module which is shown below:

In a future post, we will discuss the activities in the appRules File System module as a group.  Note that in addition to the included activities, file and directory functions can also be performed using custom functions.  For this walkthrough, we will concentrate on the activities that were used in the project snippet.

 

Below is the project snippet:

Below is the description of the main activities used in the project to perform the iteration of the files:

  1. DisplayName: InitializeFileInfoSource

Workflow Activity: DirectoryInitializeFileInfoSource

Description: The DirectoryInitializeFileInfoSource activity creates records for each file in a folder.  The records contain information for each file and the file attributes are treated as data field values as shown below.  This feature allows you to treat the file just like other records in appRules.  In this example, the DirectoryInitializeFileInfoSource is the Source and upon initialization, it loads the first record.

Below are the properties of the DirectoryInitializeFileInfoSource activity.

Once configured, the record data fields can be accessed as shown below:

  1. DisplayName: Archive File

Workflow Activity: FileMove

Description: The FileMove activity is used for moving a file from one location to another.

MoveFrom: Instead of specifying a specific file path in this property, the value used in this case is a DataFieldValue from the current file record.

MoveTo: This is the destination for the file.  The file can be move to a specific file path or to a folder as specified.

  1. DisplayName: GetNextFileInfoRecord

Workflow Activity: DirectoryGetNextFileInfoRecord

Description: This activity gets the next file info record which contains the attributes of the specific file.  When the end of the records is reached, App.Result is set to false and iteration will end.

 

Conclusion:   

The appRules File System module includes workflow activities that can be used to iterate files for large file processing projects.  In this example, by using just three activities, we are able to iterate and archive the files without writing code.

 

To download appRules: www.appstrategy.com/appRulesTrial

Data Integration 101: Anatomy of a Simple ETL Project Using appRules

In this Data Integration 101 post, we will do a quick tour of a simple ETL project using appRules.

The project which is shown below is used to perform bulk export of data from a SQL database to a CSV file.  No code was used to accomplish the task and it utilizes only three main activities.

The main activities are described below:

# Activity Description
1 InitializeSqlServerCeSource This activity is the initialization for the SQL Source.  You can select an activity from any SQL module – SQL Server, Oracle etc.  Configure the properties of this activity from any SQL module to select the records to export.
2 InitializeDelimitedTargetFile Configure this activity from the Delimited File module to define the Target CSV file and its attributes.
3 MapAndSaveRecords Use this activity to map data fields from the Source to the Target.  The activity also saves the records.

 

Conclusion

This example showcases the no-code ETL capabilities of appRules.  The same process can be used to export data from Accounting, CRM, ERP and other modules – without writing code.

 

You can download appRules with sample data integration projects at: http://www.appstrategy.com/appRulesTrial