By Stella Yifan Xie

Ant Group Co., which revolutionized China's payments industry, is learning how hard it is to separate people from their mobile phones.

Three years ago, the financial-technology giant controlled by billionaire Jack Ma embarked on an ambitious and costly effort to install facial-recognition devices at retailers that would allow people to make payments by smiling at a screen without having to use their phones. In April 2019, Ant's Alipay network said it would spend as much as 3 billion yuan, or the equivalent of about $439 million, to promote usage of the machines via merchant subsidies and shopper rebates. Rival WeChat Pay, run by Tencent Holdings Ltd., has also been expanding in this area.

The payment technology has largely failed to gain popularity, analysts say, as some consumers have found the sign-up process cumbersome and had concerns about how their images and data would be used. It shows that even a major fintech innovator with a large customer base can face privacy concerns and struggle to change user habits.

Many Alipay users have become accustomed to making payments by scanning Quick Response codes on their phones, preferring that to using credit cards or cash. While the facial scans weren't meant to replace QR codes, Ant hoped they would incentivize people to transact more often at restaurants, convenience stores and supermarkets.

"Perhaps facial recognition as a payments tool saves some time, but QR codes are already convenient enough for most consumers," said Yanxia Lu, associate research director for IDC China. The coronavirus pandemic, which forced many store closures and led to an increase in mask-wearing, has also hampered wider adoption of the technology.

Hangzhou-based Ant, the world's most valuable fintech startup, is gearing up for a pair of blockbuster initial public offerings in Hong Kong and Shanghai this fall. In its listing prospectus, it said Alipay has more than one billion users and serves more than 80 million merchants. It handled 118 trillion yuan, equivalent to $17.4 trillion, in payment transactions in the 12 months to June.

Alipay and its rivals collect small transaction fees from businesses when people use their payment networks. Overall transaction volumes are still growing, but have slowed in recent years. Many retailers in China display signs for both Alipay and WeChat Pay and let customers make the choice.

Facial-recognition devices are commonplace in cities across China. People are often subjected to face scans for identity-verification purposes before they enter residential compounds and office buildings. Some public restrooms even require people to scan their faces to collect toilet paper.

Alipay piloted its "Smile to Pay" facial-recognition payment system in mid-2017 in Hangzhou, at a new KFC concept store called KPRO that sold salads and sandwiches. Yum China Holdings Inc., which operates KFCs and Pizza Huts across the country, said it was "the first commercial application of the facial recognition payment technology globally."

In December 2018, Ant began distributing face-scanning machines called Dragonfly that were roughly the size of a first generation iPad to retailers. Last year, Ant rolled out a second-generation model that was more compact. WeChat Pay followed with a similar machine called Frog.

The devices cost 1,699 to 1,999 yuan each. Both companies also rolled out subsidy plans to help retailers cover some costs. In July, WeChat said usage of its devices was growing and about 16% to 20% of Frog machine users scan their faces to make payments.

To encourage shoppers to use the machines, Alipay added enhancements, including a function that makes eyes look larger and skin appear fairer when people see their reflections. The underlying algorithm verifies people's faces without the filters, it said.

One stumbling block, according to Alipay users, has been the sign-up process. A user needs to open the Alipay app to activate the face-payment function, verify their identity at the retailer by taking a snapshot of their face and then key in digits from their phone number or enter a passcode sent to their phone. People sometimes have to perform this process at other retailers to use scanners at many venues.

William Wang, a 30-year old financial professional in Shanghai, said he scans his face when buying groceries at Freshippo, a supermarket chain owned by Alibaba Group Holding Ltd., because it saves time. But he mostly uses his phone to scan QR codes for payments at other stores, partly because he doesn't feel comfortable having his face scanned at many locations. He worries about breaches of his personal data by companies he is less familiar with.

A survey of more than 6,000 people in China last October found that nearly 80% were concerned about personal information leakage due to the use of facial-recognition technology. Another 57% were concerned about being tracked, according to data from Nandu Personal Information Protection Research Center. About 41% of those surveyed were willing to scan their faces for payments while another 39% said they weren't.

Earlier this year, an American artificial-intelligence startup called Kneron said it was able to fool various facial-recognition systems around the world -- including machines from Alipay and WeChat Pay -- using a 3-D mask of a human face. The company described what it did as a test.

Representatives from Ant and Tencent said protecting personal data and privacy is a priority. An Ant spokesman said users of its "Smile to Pay" system have to go through a two-step authentication process to activate it, then verify their password when they are physically at payment kiosks. Referring to the Kneron test, he added: "There is no evidence to suggest expensive and lengthy testing under a highly controlled lab environment can be replicated in real life."

Li Yufeng works as a cashier at BK24, a supermarket chain in Shanghai. She said about two out of every 10 customers make payments by scanning their faces, while some people spend additional time because they have to adjust their positions in order for their faces to be scanned properly by the machines.

"I think it's way more efficient to use QR codes," Ms. Li said, adding that the only benefit she can see for customers using facial payments is when they forget to bring their mobile phones or can't access them easily.

Write to Stella Yifan Xie at stella.xie@wsj.com