The frustrated son – a story of data loss and backup concepts for Atlassian products

A app wreaks havoc

Its guy's night. Theo Schroeder has picked up some pizza and pasta, the table's set and after calling for the second time 14-year-old Jonas comes scuffling out of his room – with such a scowling face that you instinctively want to take a step back.

That's strange, Theo thinks to himself, that's not like him at all. His son is usually in a good mood when his mom and older sister are out of the house, and the "two men" have an evening to themselves.

Eventually, once they've been sitting there a little while and Theo – despite a great deal of effort – has only managed to get a few words out of him, Jonas spits it out:

"My saved games are gone!"

"Your saved games?"

"Yes, my entire game progress. All of it! I'm on level 32. I spent more than 70 hours playing that game. And today I go to play as usual and all of a sudden: No saved games! I'll have to start all over again!"

Now Theo gets it. They hadn't been able to get Jonas off the computer for weeks, he was completely lost in this new game. But on the other hand: It probably involves no less a task than saving the world. Such an undertaking isn't to be taken lightly.

"None at all?" he asks, a little skeptical. "They can't have just disappeared.

"But they have!" Jonas is close to tears.

"Hmm. Did you do something? Change some setting on your computer?"

"Nothing!" Jonas stares into space. "Oh yeah," he mumbles, furrowing his brow, "I added a new mod. But it wasn't any good, and I uninstalled it again."

"A mod?"

"Oh, Dad!" Jonas rolls his eyes. Then he signs and starts to explain: "Mods are alterations to the game developed by the modding community. It can be all different kinds of things. Bug fixes that the game maker didn't bother with. Graphic improvements. Changes to the game mechanics. Outfit or weapon skins. It could be anything really."

"Hmm." Theo nods slowly, starting to feel quite old all of a sudden. Perhaps he hadn't been paying enough attention to his son's hobbies lately.

"So it's like a kind of app. And... what if this mod... Well, what if it breaks something? Is that possible?" Theo dips his toe into new waters.

Jonas chews his lip and starts to think

"I mean, it's developed by the community, and it's nothing official, right? So it's not been checked or verified by the game maker," Theo asks.

"No, it's not official. By fans for fans, by the community for the community. The game makers don't have anything to do with the mods. They tolerate them and sometimes even program the game so that it's 'easy' to modify," says Jonas using air quotes. "But they don't check them or anything."

He grumpily chews his food and looking a little guilty adds, "yeah, perhaps that stupid mod broke something. I bet that was it. Oh, this sucks!"

He drops his fork onto his plate, crosses his arms and slumps back into his chair. Theo can see that he's really angry, probably with himself.

At the same time, Theo asks himself how this kind of thing works at the office. Do we make backups? Is the data in our Confluence or Jira programs safe? What if something unforeseeable happens – a new app that wreaks havoc, a bug or a serious mistake by a user? Would we be able to retrieve the lost data?

What if all of the drafts for our blog articles and newsletters that we collaborate on in Confluence were to disappear?  Or all the Jira updates from the last backlog grooming the marketing team spent half a day working on. That would easily total 70 hours of work. Unimaginable! He decides to ask his colleague Pamela Winter about it at work tomorrow.

After they clear the table (Jonas hasn't eaten much of course), Theo puts his arm around his son and says "Come on, let's see if Netflix has any new shows, ok?"

It's lunchtime the next day. Theo looks around the cantine and finally finds Pamela Winter, who's sitting at a table for six with two other colleagues. Theo heads over with his tray of food and takes a seat next to Pamela. They've known each other for years and often sit together at lunchtime. After a bit of chitchat, Theo gets to the point and asks his IT specialist colleague about backups and what happens when data is lost.

"No need to worry," says Pamela. She then explains that the company has an extensive operation package from an experienced service provider for all Atlassian systems, which also includes a backup concept for each of them.

An effective backup concept

This backup strategy for their Atlassian tools is in two stages. Firstly, they back up the customer's system directly – snapshots of the application data and database at the block level are generated hourly.

By default, these snapshots are kept for the following periods:

  • 24-hour snapshots (one snapshot is made every hour, and the last 24 snapshots are saved.)
  • Seven-day snapshots
  • Two-week snapshots
  • One-month snapshots

"Local snapshots like this don't help if there is a total system failure," explains Pamela, "but they do protect against short-term data corruption, which could – for example – happen as a result of human error."

The second stage of the backup concept ensures that the user data from the customer's system is saved to a backup server belonging to the service provider in another location. That applies to both the application data and the database. The data source is a local ZFS snapshot of the customer's system, to ensure that the backup made is consistent: The database and complete user data always have matching timestamps.

By default, these backups are saved for the following periods:

  • Seven-day backups (one backup is made each day and keep for seven days.)
  • Four-week backups
  • Three-month backups

Pamela goes on: "This is the data that is important in the case of a total system failure. Should this happen, we can restore these backups to our system so that we don't lose any information even if the errors in the system were severe."

Theo nods. That sounds consistent.

"The interesting thing," Pamela adds, "is the backup server itself, that is the place where we store our second-stage backup data. Encryption and reduancany are key factors here."

All data is securely transferred from the customer's system to the backup server in encrypted form and then stored securely there in encrypted form too. The backup server usually has three hard disks. That creates redundancy. Should one hard drive go down, we can switch it out without losing data.

"Furthermore, the system supports several backup servers as targets, and we usually use round-robin backups. Today's backup is saved on server A, tomorrow's on server B; and then it's server A's turn again. By spreading the data in this way, we increase the level of security even more," says Pamela, rounding up.

Theo Schroeder is feeling good about the whole thing. With the backup concept in this operation package, the company seems to have a good, solid plan up their sleeve. Although he'd never worried about it before, at least he now had some background information on the topic. It can't hurt to be at least a little familiar with it, even though he doesn't usually have anything to do with such technical stuff.

A saved game for Jonas

"How come you ask?" says Pamela. "Did something happen in your team? Are you having problems with the system?"

"Ahh," says Theo with a sigh and tells his colleagues about his son's mishap – how he invested 70 hours in this game and then installed a mod which deleted his game scores, and now all of his progress is gone.

"You don't say! I've been playing that game too!" says Pamela laughing.

"What?" Theo looks at her, a little surprised. "You?" Oh wow, it's obviously about time that he stops thinking in stereotypes.

"Absolutely," says Pamela with a wink. "I might be seen as something of a lone fighter amongst my team, but I've been a dedicated player for years now."

On the way out of the cantine, Pamela stops in her tracks. "Wait a minute, tell Jonas that he should get in touch with me. You can give him my number." She lifts a hand, holding her smartphone. "I still have all of my saved games on the computer. I can probably send him a saved game that would do the job so that he can carry on more or less from where he left off."

"Oh, that'd be great!" This sudden turn of events really cheers Theo up – and it'll probably do the same for Jonas too.

Your partner for all your Atlassian software operations

//SEIBERT/MEDIA offers attractive operation packages to run your Atlassian applications securely and reliably. Our packages can be tailored to your specific needs with SLAs, continuous monitoring, and robust systematic backup concepts. We are an Atlassian Platinum Solution Partner: Our experienced technical teams are experts in Atlassian systems and have created operation and hosting packages based on the specific requirements for these systems.

Do you want to find out more? Our Knowledge Base contains detailed information on our system administrative Atlassian services. Do you have any questions? Then feel free to get in touch! We'd be happy to help!

Lesen Sie diese Seite auf Deutsch

Forget Less and Ensure Quality with didit Checklists for Atlassian Cloud Forget Less and Ensure Quality with didit Checklists for Atlassian Cloud Forget Less and Ensure Quality with didit Checklists for Atlassian Cloud

Leave a Reply