LAS VEGAS — Yahoo YHOO -0.11% said Thursday it will join an effort by rival Google GOOGL +0.01% to create an encrypted email system by next year that could make it mathematically impossible to hand over users’ messages to a court.

If they’re successful, it would mark a big step in bringing encrypted messaging — long the province of privacy hawks and conspiracy theorists — to a consumer-friendly service. It also marks one of the starkest examples of tech giants rethinking their business plans after Edward Snowden began leaking secrets from the National Security Agency last year. Until February, Yahoo didn’t even have a c-level executive dedicated to information security.

Google in June announced plans for its own project for spy-proof email. But the addition of Yahoo is notable because the two have access to such a large chunk of world email users. In December 2013, Google had 366 million unique Gmail visitors followed by Yahoo at 273 million, according to ComScore.

Both companies say the encryption tool will be an optional feature that users will have to turn on.

It will rely on a version of PGP encryption, a long-tested form of encryption that has not yet been cracked. Unlike traditional webmail services that rely on tech companies holding passwords and usernames for consumer accounts, PGP relies on each user having their own encryption key stored on their laptops, tablets and smartphones.

In an interview, Yahoo’s chief information security officer, Alex Stamos, acknowledged there are challenges ahead for bringing such a tool to the general public.

For one, Yahoo has to explain to users how PGP works and that it is not a panacea for privacy concerns. For instance, it only encrypts the content of messages — not the data on who sends and receives the messages or the subject line.

“We have to make it to clear to people it is not secret you’re emailing your priest,” Stamos said in an interview at the Black Hat security conference here. “But the content of what you’re emailing him is secret.”