Christian Davenport
And so, early one morning last July, they pressed send, shooting the software fix from Starfish headquarters in Seattle to a ground station in Norway to the spacecraft 335 miles above Earth — hoping it would work.
In a previous generation, the stars of the Space Age were the astronauts — John Glenn, Neil Armstrong, Alan Shepard — men of military training and “Right Stuff” bearing. Today, it’s the software engineers and computer scientists who power the space economy from behind their laptops.
GET CAUGHT UP
Summarized stories to quickly stay informed
A revolution in satellite technology has produced spacecraft that are smaller and more capable, capsules that fly themselves and autonomous rockets that reach space, make a U-turn and execute pinpoint landings so they can fly again. While ground engineers and computer experts have always played an important role in spaceflight, today their role is even more pronounced as software changes are beamed to spacecraft as routinely as iPhone updates.
“Software engineers are essential,” said Abhi Tripathi, the director of mission operations for the Space Sciences Laboratory at the University of California at Berkeley, who also served several senior roles at SpaceX. “Today’s spacecraft should be really good software with a spacecraft wrapped around it.”
When Tripathi was at SpaceX, now the acknowledged leader of the commercial space industry, the company valued software engineers so highly that they hired them continuously, even without formal job openings. “The standing rule was there was always a software position open,” Tripathi said. “Even now when I’m hiring junior engineers, mission control operators or thermal engineers, I always ask: does this person know how to code?”
Earlier this year, a nimble bit of on-the-fly software engineering saved a moon landing mission. Engineers at a company called Intuitive Machines realized that sensors on their lunar lander had never been turned on, meaning their Odysseus spacecraft was essentially flying blind, unable to scout the moon’s rocky and hilly landscape for a safe landing place.
During a briefing with reporters earlier this year, CEO Steve Altemus recalled delivering the news to Tim Crain, his chief technology officer and mission director.
“I said, ‘Tim, we’re going to have to land without laser range finders,’ ” Altemus said. “And his face got absolutely white, because it was like a punch in the stomach that we were going to lose the mission.”
The team thought they might be able to swap out the dead sensor system with a NASA-developed instrument affixed to the outside of the spacecraft as an experiment for future landings. But with the main sensors not working, it would be pressed into service. Changing such a key technology on the fly was not easy, however.
“We started looking at what it would take to basically hotwire the system,” James Blakeslee, a software architect at the company, said in an interview. To buy time, the team decided to fly the spacecraft around the moon one more time while the coders tested their software update on a simulator. “We worked out in the backroom and the developer that was in charge of it, he wrote it down on a Post-it note and ran it into the front room,” Blakeslee said.
Normally, such a fix would “have taken a month,” Crain said at the time. The math would have been checked through thousands of simulations, which typically would find errors, forcing coders to try again. Instead, he said, “Our team basically did that in an hour and a half. It was one of the finest pieces of engineering I’ve ever had the chance to be affiliated with.”
The spacecraft landed on its side after catching one of its legs, a partial success that allowed the company to claim the first lunar landing by a commercial venture — and the first by the U.S. since the last of the Apollo missions in 1972.
A similar drama played out in 2019, when Boeing’s Starliner spacecraft was in trouble. The spacecraft’s onboard computer system was 11 hours off, meaning it was executing commands for an entirely different part of the mission while burning precious fuel. Software programmers were able to send commands to the spacecraft, fixing the problem.
They also were able to troubleshoot for other potential issues — and found one. Upon separation from the crew capsule before reentering Earth’s atmosphere, the service module could cause a collision, potentially damaging the capsule. Software engineers were able to fix that, too.
While the spacecraft was flying a test flight with no one on board and did not dock with the International Space Station, it did land safely back on Earth. Boeing launched an investigation to study all 1 million lines of code in the spacecraft to make sure there weren’t other errors.
Perhaps no space company values software development more than Elon Musk’s SpaceX. Its giant rocket boosters fly back to Earth, landing on autonomous ships at sea or landing pads on land. Its Dragon spacecraft flies itself to the space station, relegating the astronauts onboard to little more than passengers.
But SpaceX has had its share of nail-biters that required a bit of improvisational creativity. In 2013, a Dragon spacecraft had a few valves stuck while it was approaching the space station. So a programmer sent a command to build up pressure ahead of the valves and then suddenly release it, delivering the kick needed to open them. At the time, Musk called it “the spacecraft equivalent of the Heimlich maneuver.”
As is often the case, that fix was not the result of writing entirely new code from scratch, Tripathi said, but rather tweaking existing code to produce new outcomes. It also came after engineers tested the software extensively before beaming it up to the spacecraft.
Some start-ups don’t have the resources to fully test their systems before launch and actually plan on making adjustments midflight.
“A lot of companies have a set launch date they need to get to, and if they don’t start generating revenue their investors won’t be happy,” Tripathi said. “So a lot of these companies launch something before they’ve done the full amount of software testing that they should do. They say, ‘We have a software test bed. We have good coders. We’ll figure it out on the fly.’”
That’s where Bennett and his Starfish team found themselves last summer, with their spacecraft tumbling. Called Otter Pup, the spacecraft was supposed to detach itself from what’s known as an “orbital transfer vehicle,” or OTV, a separate spacecraft that acts like a tugboat and brings it to a certain location in space.
If all went well, the tug would release Otter Pup, which would then fly itself back to the tug and reattach itself — a demonstration that Otter Pup could dock itself to satellites in space and move them to different orbits, or even repair and service them to extend their lives.
Instead, the OTV started spinning and released Otter Pup into a spin as well.
“It literally did a whole spin every second — all the way around,” Bennett recalled in an interview. “And somebody was like, ‘Oh, one revolution per second, that must be a typo. They must mean one degree per second.’ It was like no — this was just so far outside the norm, people thought it was a typo.”
It was spinning, but the spacecraft was alive.
To keep the spacecraft as light and simple as possible, it did not have a lot of thrusters. But it did have what are known as “torque rods,” which could be used to slow it down by shooting out a magnetic pulse. The trick would be to emit the pulse at just the right moments as the spacecraft spun so that it would push against Earth’s magnetic field
Bennett, who has a PhD in aerospace engineering sciences and previously worked on Blue Origin’s orbital rocket, worked on some calculations, eventually calling in the company’s chief engineer. “Sit with me here for half an hour. Just humor me,” he recalled telling his colleague. “I’m going do some quick math, and I want you to tell me where I went awry.”
But as he scribbled his calculations on a whiteboard, the chief engineer confirmed his math.
“Holy cow,” Bennett said. “I think we might be able to pull this off.”
They tested the plan with torque rods they had on the ground, sending as many as 10 commands a second because of the high tumble rate. They felt confident it would work. But the spacecraft was spinning so fast that communicating with it was difficult: They could only send up 20 lines of code at a time. So they laboriously edited their formula into bite-sized chunks, and ran those through the simulator.
On July 31, some six weeks after the launch, they started transmitting the code. Almost immediately, the spacecraft started slowing down. After several hours of executing the commands, it was finally stable.
The OTV was lost. But this year, Starfish identified another spacecraft that presented an opportunity for a rendezvous. In April, it got within about a half-mile of that satellite.
To demonstrate that victory, it snapped a picture.