By Sahil Patel

People are streaming more movies and shows on internet-connected TV sets. But keeping track of who is watching what and where -- and how many times they see the same ads -- is becoming a bigger frustration for advertisers seeking to move money into the medium.

Connected-TV advertising is growing, although it remains a fraction of the roughly $70 billion that gets spent on traditional TV advertising in the U.S. every year. Ad spending on streaming TV will total almost $8 billion in the U.S. this year, up from nearly $6.4 billion in 2019, according to data from research firm eMarketer.

"In not too many years, it will be the primary way viewers receive their premium video," said David Campanelli, chief investment officer at media buyer Horizon Media Inc.

A bulk of streaming today occurs in ad-free services such as Netflix Inc., but ad-supported offerings from providers including Amazon.com Inc., Roku Inc. and TV network programmers are increasingly drawing viewers.

The medium faces a raft of advertising challenges, including a fragmented media-buying landscape that can lead to viewers getting hit repeatedly with the same ad.

Marketers buy ads in streaming TV from a galaxy of providers, including app owners like TV networks, vendors that connect ad buyers with sellers, and device makers with their own operating systems such as Roku, Amazon and Vizio Inc.

The wealth of options brings limitations: Roku, Amazon and Walt Disney Co.'s Hulu, for example, each sell ads and have their own audience data, but it is hard for advertisers to track or target viewers from app to app, or from one operating system to the next. Ad inventory bought from multiple sellers can often show up in the same app.

"There is a massive fragmentation of media in connected TV -- and it's multidimensional fragmentation," said Tal Chalozin, chief technology officer of ad tech firm Innovid Inc.

That complicates marketer priorities like limiting how much a viewer sees the same ad. It has been a continuing problem for marketers in connected TV, where a smaller pool of advertisers than those that buy traditional TV results in viewers sometimes seeing the same ad many times even in a single program.

"Connected TV usage has gone up significantly -- and everyone is dumping money into it -- but you can see how it's breaking at the seams," said one ad buyer who plans to spend $15 million this year on streaming TV advertising for marketer clients, more than double what he spent last year. "You get the same ad over and over -- it's worse than ever."

Amazon said it offers frequency controls to its advertisers. But that doesn't apply to advertisers who buy through other sellers to run on apps on Amazon's Fire TV.

Roku said it has made efforts to improve targeting and reach and frequency controls within its ad-buying tools. It also sends a device ID to measurement partners such as Innovid. But Innovid helps advertisers serve ads and measure performance, not buy ads in the first place.

Some ad executives said the problem will improve as more people stream TV and more advertisers follow. Traditional TV also suffers ad repetition if viewers watch long enough, they noted.

Others said more advertisers won't fix fragmentation.

"No one is holistically doing frequency management," said Brad Stockton, vice president of video innovation at Dentsu Inc.-owned Dentsu Aegis Network, which is doubling its upfront commitments to connected TV this year. "Everyone's solve is to give them every single one of your ad dollars."

Over the past few years, General Motors Co. has been increasing its media spend on connected TVs, executives from the company said. One limitation from spending more has been a lack of transparency on when and where ads run within streaming platforms and apps, said David Spencer, assistant manager of audience buying strategy for GM. Platform and app owners have been better about sharing such information over the past 18 months, but the issue hasn't been completely resolved, he said.

"The better that gets, the more willing we will be to invest larger dollars," Mr. Spencer said.

One solution could be a universally agreed-upon identifier for connected TV audiences -- akin to the tracking cookies that have historically powered online advertising. A universal ID could be a piece of data that represents a connected TV user or household that can allow advertisers to track viewing behavior across multiple devices, platforms and apps.

At the moment, device makers use data on their signed-in users to form audience segments for advertisers, while many ad tech vendors use data proxies including IP addresses.

A universal ID could help marketers more precisely reach intended audiences across apps and platforms, said Mr. Chalozin, the Innovid executive. "No consistent identity equals: I'm going to need to spend significantly more and can't be as efficient," he said.

Some are looking at new approaches. A Trade Desk project called Unified ID 2.0, which aims to harness encrypted email addresses from publishers that have login information, eventually could extend to connected-TV app makers, said the ad tech company, which offers access to ad inventory on many TV apps.

But any universal ID will require buy-in from the companies that own the devices and operating systems. And these ad sellers benefit from keeping advertisers within their own ecosystems, buyers said. This is among the reasons marketers and ad buyers said a universal ID is unlikely to be introduced anytime soon.

"I'd love a universal ID that goes across absolutely everything, but it's unrealistic right now because of the walled gardens," Mr. Stockton said.

Louqman Parampath, vice president of product management at Roku, likened any universal ID that isn't backed by identity-based data -- like the type Roku has because of its registered users -- to "a passport without a name or address."

Write to Sahil Patel at sahil.patel@wsj.com