Missing images and sound files in lessons and assessments
Incident Report for Education Perfect
Postmortem

An issue occurred with Education Perfect recently which caused problems accessing user-uploaded media files, such as images, animations, videos and sound files. For this, we’re extremely sorry. We understand that Education Perfect is a core part of many schools, and that disruption wastes your time.

The root cause was a side-effect of our new process which anonymises the user accounts of students and teachers who are no longer at school, in accordance with our privacy policy. Without going into all the details, this includes deleting files uploaded by students and teachers, whose files are stored in folders based on their user ID. Unfortunately, due to a missing forward-slash at the end of the search pattern, when we deleted user ID 9, it also deleted the files of every user whose ID began with a 9.

The positive news is that because the issue was identified rapidly, the deleted files were still available in backups, and we ran a process yesterday afternoon to restore all inadvertently-deleted files. This completed at about 8pm NZST on Tuesday 16th October, and all files are now restored and available. Naturally, we’ve also fixed the root bug that caused this to happen in the first place.

A disruption like this occurring is not up to the high standards that customers expect of us, particularly in the context of assessments. You can be confident that the same fault will not occur again.

While it’s difficult to guarantee that there will never be any issues within any rapidly-evolving and improving platform, our key focus is always on maintaining reliable access to the platform, especially to the assessments area. Student answers for assessments are auto-saved every 30 seconds, as well as every time they navigate between questions. Once a student has begun an assessment they can complete it successfully even if our primary databases were offline for some reason. Even in the event of an internet outage at your school, or with our cloud provider in Sydney, answers are saved to the student’s device to be synched once they come back online.

On our roadmap we have a project scheduled for the assessments area that will introduce further measures to insulate assessments from the parts of the platform that are being regularly updated, while maintaining the integration that exists between different parts of the system.

I hope that the above information answers your questions about what happened yesterday and reassures you that we take issues like this seriously. If you have any other questions about this or anything else, please don't hesitate to contact me.

Sincerely,

Ben Crowl -- Development Team Manager
Education Perfect

Posted 11 months ago. Oct 17, 2018 - 12:48 NZDT

Resolved
Some users experienced that images, animations, sounds and student uploads were missing from lessons on Monday and Tuesday. This issue is now resolved and all missing media files should now be restored.
Posted 11 months ago. Oct 15, 2018 - 00:00 NZDT