Reminder: make backups and check backups.
On June 20, the fourth installment of the Toy Story cartoon series was released at cinemas. This is a continuation of the cult franchise, which gathered several Oscars and formed Pixar’s reputation as one of the best animation studios in the world. Despite the fact that the premiere of the first film took place more than 20 years ago, Toy Story 4 successfully collects the cashier and also receives rave reviews from critics.
After the premiere of the fourth part in social networks, they recalled the 2012 article from The Next Web, devoted to the production of “Toy Story 2”. Journalists talkedwith several employees of Pixar and told how shortly before the premiere of the cartoon he was twice removed and painted almost from scratch. The first time, due to an error on the server with a backup, and the second time due to the insufficiently high quality of the final version.
“/ Bin / rm -r -f” command
The continuation of the “Toy Story” was released in November 1999 – four years later, after the premiere of the original. The cartoon has extremely positive reviews fromcritics: many have written that this is the rare case when the sequel is not worse than the original. The film won the Golden Globe Award, was nominated for an Oscar and earned almost $ 500 million worldwide. But all this could not be if it were not for the cohesion and hard work of employees Pixar.
Oren Jacob – one of the founders of Pixar, who stood at the origins of the company, participated in the production of “Flick Adventures”, the first “Toy Story” and “Finding Nemo”. In the 1990s, while working on Toy Story 2, he was responsible for all the visual effects – it was Jacob who first noticed the loss of several gigabytes of cartoon files.
In the production of the cartoon involved more than 150 people. The source files of the almost finished version weighed 10 gigabytes. Someone from the staff, intentionally or accidentally, started the command “/ bin / rm -r -f” inside the directory – roughly speaking, meaning to clear the entire directory. Almost all of the 100,000 files have disappeared: these are images of heroes, drawing of rooms, and animation of characters.
Larry Cutler opened the catalog and started talking about installing a fix for Woody or for his hat. He looked into the directory where there were only 40 files. I looked again – there are four files left.
We saw the file sequences start to disappear. We thought, “Oh my god.” I grabbed the phone and said: “Turn off the server.”
Oren Jacob head of the visual effects department of Toy Story 2
After the command to turn off the servers, many employees have lost their groundwork. Many did not even understand what had happened, and went to dinner. A few hours later the server was launched and the management saw that it managed to save only 10% – 90% of the cartoon was destroyed. And no one knew exactly what 10% managed to save.
According to Jacob, for the next 48 hours, they restored the cartoon from a backup two months ago: they rendered some frames, found high-quality screenshots, and ran tests. But it turned out that an error had also crept into it: the source materials for the animation took about 10 gigabytes, but were copied to a streamer (tape drive) of 4 gigabytes. These cartoons now weigh several hundred gigabytes, but in 1998 they quietly fit into a pair of dual-layer DVDs. They did not notice the inaccuracy, because with each entry the new two thousand files “crowded out” the old two thousand files. The log file with the error notification was not displayed because it was not placed on the carrier.
The Next Web journalists suggested that the system inside Pixar was too open and free, so one employee made a mistake in the code, and another didn’t watch backups enough. The management would have had to divide the staff on rights and access to all the files, but then the company was not so big as to waste time on it.
Jacob recalled the exemplary dialogue that took place at the leadership meeting after the loss of the cartoon:
– Let’s do without the witch hunt. First we have to return our project. Let’s not spend a week of total time trying to judge and kill someone. Where is the movie?
“First of all, let’s just return the film and recreate Buzz and Woody.” We lost our friends.
dialogue at a meeting of the leadership of the “Toy Story 2”
Fortunately for the Toy Story 2 team, one of the employees saved a recent backup on a home computer. This backup was compared with a backup two months ago and realized that there are not many differences between them. Then the employees united on one task – the restoration of the “tree” of the files with the help of sequential checking of each. Most of them were damaged or unfinished: for example, in the animation, when Woody takes off his hat, the headdress first appeared on his head, and then immediately in his hand, without drawing a transition.
After checking two backups of 100 thousand files, 70 thousand were in order. They were either preserved in back-ups, or they were collected in pieces from the remaining branches. Over the weekend, a team of 30 people checked the remaining 30 thousand files manually – they scanned, read and searched for inaccuracies in each of them. People refused the weekend with their family and canceled all their plans, Jacob recalls: “We worked non-stop from Friday to Monday morning. Shifts from 10-12 people with short breaks for sleeping and eating. ” The nearby sandwich café even gave them free portions when they noticed staff working in the middle of the night.
Thanks to this, Toy Story 2 came to life again. Until now, the team restoring the cartoon, does not know where some of the files went missing and how it worked without them. But then none of the creators of the project knew that this was not the end.
New “Toy Story 2”
Christmas 1998, in all US cinemas, the premiere of Flick Adventures was held – the second full-length cartoon that Pixar made with Disney. He had to compete with another insect cartoon – “Ant Antz” from DreamWorks, released a month earlier. This did not prevent the tape to recoup its budget several times and collect 364 million dollars.
After the premiere of Flick Adventures, Pixar bosses gathered in the office to watch a preliminary (and recently saved) version of Toy Story 2. In the room sat: screenwriter and director Joe Renft, producer and animator Andrew Stanton, one of the founders of the studio, John Lasseter and animator Pete Docter. Together they decided that the picture was not good enough. All Christmas holidays guide, along with leading screenwriters rewrote the script from scratch.
In fact, got rid of all the animation and almost all layouts. All that remains of the original “Toy Story 2” is the main toy characters, Andy’s room and Al’s “Barn Toys”. Everything else came up with anew. This version was shown in cinemas and released on storage media.
The main obstacle for 150 people from the cartoon team is its release date. November 22, 1999. Such large premieres are planned for several years, they are tied to an advertising campaign, test shows and the release of merchandise. The authors had to almost completely remake the big-budget cartoon for nine months. As Jacob noted, Disney didn’t believe it, but Pixar’s management did.
Almost every member of the team has signed up for a 100-hour work week for the next nine months. At that time, no one could afford to relax: Pixar was an independent and not so well-known company as it is now, and the failure or prolonged transfer of the premiere would hit its market value and reputation.
According to Jacob, this is one of the main points in the history of Pixar, which determined its corporate culture and showed how employees are ready to come together for the common good. He described the team not just as a cartoon studio, but as a friendly community that was ready for the challenge.
Never before have I experienced anything like it. It was a loss of such a level that nobody needed to explain how bad everything was. People already knew. Everyone around us knew what was going on.
I will never forget that I was part of Toy Story 2. I was very lucky. I worked with people at such a level of trust that we were able to keep Buzz and Woody, and Toy Story, the franchise, and Pixar — everything that still exists.Oren Jacobhead of the visual effects department of Toy Story 2