Bug #4332

EAD - dates not importing

Added by Jessica Bushey about 8 years ago. Updated over 7 years ago.

Status:VerifiedStart date:
Priority:CriticalDue date:
Assignee:Jesús García Crespo% Done:

0%

Category:-
Target version:Release 1.3
Google Code Legacy ID:atom-2384 Tested version:
Sponsored: Requires documentation:

Description

Review this User forum thread: https://groups.google.com/d/topic/ica-atom-users/9Kna3Z6LCrM/discussion

User reported problem (Error Encountered) while importing archival descriptions as EAD. David suggested changing file extension .txt to .xml. User responded that there are no longer any errors; however, when looking at the records you find there are no dates in any of the descriptions, additionally all the refcodes from upper levels appear in all the lower levels.

Note: Editing imported records to add a date works fine; and so does creating a new record through the interface (with a date).

This issue could be related to an additional problem raised on the google user group about <unitdate>: https://groups.google.com/d/topic/ica-atom-users/IXt5n0HOgrg/discussion .

[g] Legacy categories: EAD

Leicester_edit1.xml Magnifier (37.5 KB) Anonymous, 12/01/2012 05:34 AM

Screen Shot 2012-08-01 at 7.45.22 AM.png (57.9 KB) Jesús García Crespo, 12/01/2012 05:34 AM

SSUA-164.xml Magnifier (4.55 KB) Tim Hutchinson, 12/01/2012 05:34 AM

YCWG [EAD].xml Magnifier (193 KB) Anonymous, 12/01/2012 05:35 AM

History

#1 Updated by David Juhasz about 8 years ago

  • Priority changed from High to Critical

[g] Labels added: Priority-Critical
[g] Labels removed: Priority-High

#2 Updated by Tim Hutchinson about 8 years ago

I haven't been able to figure out what conditions lead to the date problem, but it seems to come down to the part of the setDates method that tries to link a date to an existing creator. Starting at line 1389 of lib/model/QubitInformationObject.php

// assign the dates to the same event as the creator for this information object
// if there is more than one creator, assign it to the first one that is returned
if (count($creationEvents = $this->getCreationEvents()) > 0) {
$event = $creationEvents[0];
$event->setIndexOnSave(false);
$event->setStartDate($normalizedDate['start']);
$event->setEndDate($normalizedDate['end']);
$event->setTypeId($eventTypeId);
$event->setDate($date);
$event->save();
}

- I was encountering cases where the criteria was met (at least one creation event), but the date wasn't being added. Changing $event->save() to $this->events[] = $event;
seems to solve that.
- but for the most part, I couldn't replicate the criteria being met, unless I hardcoded information_object_id in getCreationEvents. In one case a different actor actually got linked up, but now I can't replicate that. Perhaps the information_object_id has not been created at this point during the routine? I don't understand the object oriented stuff enough to troubleshoot this aspect.
- in earlier import work that I've done, the linking never seems to have worked; there is always a date event separate from the creator (I remember other reports of this too). So this may be the source of the problem but under normal circumstances no creation events are found?
- so, a workaround would be to comment out the attempted linking, and force the routine to add a new event.

#3 Updated by Tim Hutchinson about 8 years ago

Re refcodes, this appears to be an import issue - the inheritance is hard-coded in the EAD files, so duplication ensues if ICA-AtoM's inheritance is enabled.

#4 Updated by Jesús García Crespo about 8 years ago

  • Status changed from New to New

Could someone please provide a document example that I can use to reproduce the issue? Thanks in advance.

[g] New owner: Jesús García Crespo

#5 Updated by Anonymous about 8 years ago

Attached is an example of a file from which the dates have not imported.

#6 Updated by Jesús García Crespo about 8 years ago

For some reason it worked in our development tree. I'll have to try against 1.2.1 later.
I've attached a screenshot, the dates have been imported.

#7 Updated by Tim Hutchinson about 8 years ago

Unfortunately the issue is more complicated than that - I haven't been able to figure out what combination of existing records triggers the error. But here's how you can replicate the steps I took to do the analysis I reported in commment #2

- import the attached file (note that Simon's file doesn't include correctly encoded creator information, i.e. the way qubit is looking for it, but we need that in order to test).
- it should have worked fine
- now go to the backend database; scroll to the last few records in the event table
- check the most recent record where both actor_id and information_object_id is populated; make note of information_object_id
- edit the getCreationEvents function to hard-code the ID lookup to that information_object_id - to force a match on the creation events lookup. E.g.:

public function getCreationEvents()
{
$criteria = new Criteria;
// $criteria->add(QubitEvent::INFORMATION_OBJECT_ID, $this->id);
$criteria->add(QubitEvent::INFORMATION_OBJECT_ID, "817");

- now re-import the same file
- this time, the dates should not have imported
- now tweak the setDates function to change the $save call:

if (count($creationEvents = $this->getCreationEvents()) > 0)
{
$event = $creationEvents[0];
$event->setIndexOnSave(false);
$event->setStartDate($normalizedDate['start']);
$event->setEndDate($normalizedDate['end']);
$event->setTypeId($eventTypeId);
$event->setDate($date);
// $event->save();
$this->events[] = $event;
}

- and re-import the same file
- this time the date should have imported, but linked to the creator record

So two things seem to be happening:
- under normal circumstances, the dates don't get linked to the creator record, at least if the creator was added during the same import routine
- if there is a match on the creation event, the date linking routine doesn't seem to be working

#8 Updated by Tim Hutchinson about 8 years ago

Further to this, the first component might be considered a separate issue - I gather that the expected behaviour is to link the dates to the creator. From my perspective, however (and I think this consistent with CCAD's advice), I'm not sure that it's a problem that the linking doesn't happen - dates of creation and creator are separate elements - so as I mentioned before, at least a workaround would be to force a separate date event to be added. I can't figure out under what circumstances the linking is actually attempted, and up until now I hadn't realized that was the intended behaviour.

#9 Updated by Anonymous about 8 years ago

Not sure if this adds anything, but the import routine must be 'reading' the dates, as earlier date ranges exported from the Hub were separated by a hyphen rather than a slash and if not changed then the import fails. Ironically, when changed the routine works - minus the dates!

#10 Updated by Tim Hutchinson about 8 years ago

Yes, it's definitely reading the dates, in that the setDates method is called. But the database doesn't get updated.

#11 Updated by Jesús García Crespo about 8 years ago

  • Status changed from New to QA/Review

Thank you Tim and Simon, your effort is much appreciated.

I couldn't find neither the situation where the problem can be reproduced. In the other hand, I think that getCreationEvents() is not working as expected during the import because as the time that it is called the id property has not been populated yet (not until the object is persisted in the database).

I'm unclear on #c7 because you are basically taking an event from another information object and linking it to the new object. $event->save() wasn't working because the information_object_id value is not changed, but by adding the object to the events array ($this->events[] = $event) you are basically re-linking it to the new information object. It would disappear from your previous imported object. In the same file, take a look at the save method, you'll find that after the object is saved every object in the events array is iterated, re-linked and saved.

As of r12025, a separate date event is added as Tim suggested.
Simon, let us know if this solution worked for you.

Thank you again!

#12 Updated by Jessica Bushey about 8 years ago

Is there a procedure or series of steps that I should take to test this issue?

#13 Updated by Tim Hutchinson about 8 years ago

I could see if I can get back to the point where I can reproduce the error - but that may be a moot point given the change you've made; and even then I would have to send you a data dump. So this will be hard to actually test. Is a puzzlement!

Simon, can you see if your host can apply this change - it's essentially the same as the workaround I'd suggested before. They could also cheat and change > 0 to < 0 around line 1391 (this may be a slightly different line number in 1.2.1). It sounds like you don't have command line access?

#14 Updated by Anonymous about 8 years ago

Tim, yes you are correct about me not having command line access.

I have requested that our host apply the change and I shall report back with the results (though i am on holiday for a week from today, so it will be the end of next week before I get chance to test it).

Jessica,

To reproduce this error: ========================
1)Import an EAD file that contains data wrapped in the EAD <unitdate> tag
2)Try to find the data in the imported information object

Resulting error: ================
Data contained in an <unitdate> tag is not imported into the information object (the archival description).

Expected result: ================
<unitdate> tag is recognized and mapped to appropriate ICA-AtoM field

#15 Updated by Anonymous about 8 years ago

I have tried importing several files and the fix worked just great, thanks for your help with this.

Simon

#16 Updated by Jesús García Crespo about 8 years ago

This would not have been possible without your help. Thank you all!

#17 Updated by Jessica Bushey about 8 years ago

  • Status changed from QA/Review to Verified

Also available in: Atom PDF