Add ability to include a python transformation script as part of an upload via the user interface
|Assignee:||Nick Wilkinson||% Done:|
|Category:||Import/Export||Estimated time:||24.00 hours|
|Target version:||Release 2.4.0|
|Google Code Legacy ID:||Tested version:|
This feature will further enhance the bulk import via the user interface development described in #10137.
Essentially, this feature will allow users the ability to include a locally created transformation script, written in python, as part of their upload. This is useful when exporting from a non-AtoM source system - instead of having to refine the XML or CSV outside of AtoM prior to import, users can craft a reusable transformation script, and simply indicate it should be used as a first step during the upload process. AtoM will execute the transformation script on the import files prior to performing the upload.
For security, this will require a system administrator to configure locally before an administrator can make use of this feature. The sysadmin will add the transformation script to the server, and then add the path to it via a configuration setting contained in config/app.yml.
If there is a path to the transformation script included in config/app.yml, then an administrator will be able to see and use a checkbox on the import page, which will trigger the transformation script to be run on the selected file(s) prior to import.
When there are problems with the transformation script or the resulting import, AtoM should fail gracefully and provide clear error messages to the user. These will be available in the Job details page, added to AtoM 2.4 as part of #9986.
#5 Updated by Steve Breker almost 4 years ago
To specify a transformation script and activate the feature:
- open config/app.yml
- look for the property "csv_transform_script_name"
- uncomment this line by removing the '#' symbol
- specify the path to the transform script replacing the placeholder text "/full/path/to/transformscript".
- the checkbox to use the transform script should now be visible on the CSV upload page including the path and name of the transform script.
#12 Updated by Steve Breker almost 3 years ago
- File command_line_shows_how_script_would_be_run_from_cli.png added
- File app_yml-showing_modified_script_path.png added
- File Result_of_run.png added
- File import_screen_showing_all_set_to_run_script_selected.png added
- File import_screen_showing_screen_defaults.png added
- File app_yml-uncommented-2space_indent_is_important.png added
- File app_yml-pre_modified.png added
- Status changed from QA/Review to Verified
- Assignee changed from Steve Breker to Nick Wilkinson
QA and doc notes:
QA looks good. Specified script, gui elements appear; script is run; output is picked up.
In AtoM install directory:
1) open config/app.yml (app_yml-pre_modified.png)
2) find the property "csv_transform_script_name"
3) uncomment this line by removing the '#' symbol (app_yml-uncommented-2space_indent_is_important.png)
4) specify the path to the transform script replacing the placeholder text "/full/path/to/transformscript". (app_yml-showing_modified_script_path.png)
5) clear cache (php symfony cc). Restart memcached & php-fpm
6) the checkbox to use the transform script should now be visible on the CSV import page including the path and name of the transform script.
Transformation file can be implemented in any language. My example here is php.
Transformation script must take two parameters:
1) first is path to the input file
2) path to transformed output file.
e.g. transformscript <input file path> <output file path> (command_line_shows_how_script_would_be_run_from_cli.png)
An admin must configure system permissions such that the transformation script is able to be run by the AtoM user, and AtoM is able to access the transformed output file.