Feature #10138

Add ability to include a python transformation script as part of an upload via the user interface

Added by Dan Gillean almost 4 years ago. Updated almost 3 years ago.

Status:VerifiedStart date:05/02/2016
Priority:MediumDue date:
Assignee:Nick Wilkinson% Done:

0%

Category:Import/ExportEstimated time:24.00 hours
Target version:Release 2.4.0
Google Code Legacy ID: Tested version:
Sponsored:Yes Requires documentation:Yes

Description

This feature will further enhance the bulk import via the user interface development described in #10137.

Essentially, this feature will allow users the ability to include a locally created transformation script, written in python, as part of their upload. This is useful when exporting from a non-AtoM source system - instead of having to refine the XML or CSV outside of AtoM prior to import, users can craft a reusable transformation script, and simply indicate it should be used as a first step during the upload process. AtoM will execute the transformation script on the import files prior to performing the upload.

For security, this will require a system administrator to configure locally before an administrator can make use of this feature. The sysadmin will add the transformation script to the server, and then add the path to it via a configuration setting contained in config/app.yml.

If there is a path to the transformation script included in config/app.yml, then an administrator will be able to see and use a checkbox on the import page, which will trigger the transformation script to be run on the selected file(s) prior to import.

When there are problems with the transformation script or the resulting import, AtoM should fail gracefully and provide clear error messages to the user. These will be available in the Job details page, added to AtoM 2.4 as part of #9986.

command_line_shows_how_script_would_be_run_from_cli.png (23.5 KB) Steve Breker, 08/08/2017 08:20 PM

app_yml-showing_modified_script_path.png (177 KB) Steve Breker, 08/08/2017 08:20 PM

Result_of_run.png (24.8 KB) Steve Breker, 08/08/2017 08:20 PM

import_screen_showing_all_set_to_run_script_selected.png (64.9 KB) Steve Breker, 08/08/2017 08:20 PM

import_screen_showing_screen_defaults.png (63.6 KB) Steve Breker, 08/08/2017 08:20 PM

app_yml-uncommented-2space_indent_is_important.png (159 KB) Steve Breker, 08/08/2017 08:20 PM

app_yml-pre_modified.png (159 KB) Steve Breker, 08/08/2017 08:20 PM

History

#2 Updated by Steve Breker almost 4 years ago

  • Status changed from In progress to Code Review
  • Assignee changed from Steve Breker to Nick Wilkinson

#3 Updated by Steve Breker almost 4 years ago

For QA, this issue is dependent on 10175. Testing prior to 10175 will mean that matching on keymap on import will not work correctly when importing csv / xml via the WebUI.

#4 Updated by Steve Breker almost 4 years ago

  • Status changed from Code Review to QA/Review
  • Assignee changed from Nick Wilkinson to Dan Gillean

Merged to qa/2.4.x - ready for QA.

#5 Updated by Steve Breker almost 4 years ago

To specify a transformation script and activate the feature:

- open config/app.yml
- look for the property "csv_transform_script_name"
- uncomment this line by removing the '#' symbol
- specify the path to the transform script replacing the placeholder text "/full/path/to/transformscript".
- the checkbox to use the transform script should now be visible on the CSV upload page including the path and name of the transform script.

#6 Updated by Dan Gillean almost 4 years ago

  • Target version changed from Release 2.4.0 to Release 2.5.0

#7 Updated by Dan Gillean almost 4 years ago

  • Description updated (diff)

Minor updates to issue description to match implementation.

#8 Updated by Dan Gillean almost 4 years ago

  • Target version changed from Release 2.5.0 to Release 2.4.0

#9 Updated by Dan Gillean about 3 years ago

  • Assignee deleted (Dan Gillean)

#10 Updated by Nick Wilkinson almost 3 years ago

  • Assignee set to Steve Breker

Hi Steve, can you please take a look at this to QA?

#11 Updated by Nick Wilkinson almost 3 years ago

Also Steve: please remember to take any screenshots you think would be helpful for documentation purposes (provided it doesn't contain client data / info)

#12 Updated by Steve Breker almost 3 years ago

QA and doc notes:

QA looks good. Specified script, gui elements appear; script is run; output is picked up.

Setting up:
In AtoM install directory:

1) open config/app.yml (app_yml-pre_modified.png)
2) find the property "csv_transform_script_name"
3) uncomment this line by removing the '#' symbol (app_yml-uncommented-2space_indent_is_important.png)
4) specify the path to the transform script replacing the placeholder text "/full/path/to/transformscript". (app_yml-showing_modified_script_path.png)
5) clear cache (php symfony cc). Restart memcached & php-fpm
6) the checkbox to use the transform script should now be visible on the CSV import page including the path and name of the transform script.
(import_screen_showing_screen_defaults.png)
(import_screen_showing_all_set_to_run_script_selected.png)
(Result_of_run.png)

Transformation file can be implemented in any language. My example here is php.

Transformation script must take two parameters:
1) first is path to the input file
2) path to transformed output file.

e.g. transformscript <input file path> <output file path> (command_line_shows_how_script_would_be_run_from_cli.png)

An admin must configure system permissions such that the transformation script is able to be run by the AtoM user, and AtoM is able to access the transformed output file.

Also available in: Atom PDF