DZ Reserve v. Facebook, Inc. Document 339: Exhibit(s)

Page 1

Exhibit L

Page 2

LATHAM & WATKINS LLP

Elizabeth L. Deeley (CA Bar No. 230798) elizabeth.deeley@lw.com Melanie M. Blunschi (CA Bar No. 234264) melanie.blunschi@lw.com Nicole C. Valco (CA Bar No. 258506) nicole.valco@lw.com 505 Montgomery Street, Suite 2000 San Francisco, CA 94111-6538 Tel.: +1.415.391.0600/Fax: +1.415.395.8095 Andrew B. Clubok (pro hac vice) andrew.clubok@lw.com Susan E. Engel (pro hac vice) susan.engel@lw.com 555 Eleventh Street, N.W., Suite 1000 Washington, D.C. 20004 Tel.: +1.202.637.2200/Fax: +1.202.637.2201

Hilary H. Mattis (CA Bar No. 271498) hilary.mattis@lw.com 140 Scott Drive Menlo Park, CA 94025 Tel.: +1.650.328.4600/Fax: +1.650.463.2600

Attorneys for Defendant Facebook, Inc.

UNITED STATES DISTRICT COURT

NORTHERN DISTRICT OF CALIFORNIA

SAN FRANCISCO DIVISION

DZ RESERVE, and CAIN MAXWELL (d/b/a MAX MARTIALIS), individually and on behalf of all others similarly situated,

Plaintiffs, v.

FACEBOOK, INC.,

Case No. 3:18-cv-04978 JD DECLARATION OF YIYANG SHI IN SUPPORT OF FACEBOOK’S OPPOSITION TO PLAINTIFFS’ MOTION FOR CLASS CERTIFICATION Hon. James Donato

Defendant.

ATTORNEYS AT LAW
SAN FRANCISCO

SHI DECLARATION

CASE NO. 3:18-cv-04978 JD

Page 3

I, Yiyang Shi, declare as follows:

1.

I am a Data Scientist on the Ads Core Data Analytics Team at Defendant Facebook,

Inc. (“Facebook”). I submit this declaration in support of Facebook’s Opposition to Plaintiffs’

Motion for Class Certification. I have personal knowledge of the matters stated herein and, if

called upon, I could and would competently testify thereto.

2.

I have a Master’s degree in Statistics and Operations Research and a Bachelor’s

degree in Economics from the University of North Carolina at Chapel Hill. I have worked at

Facebook since August 2018. I previously was a Data Scientist at Uber.

3.

In late 2018 to early 2019, I understand the Ads Targeting Team was exploring

whether to change one of the inputs to the Potential Reach estimate. Specifically, the Ads

Targeting Team was considering whether to change from estimating Potential Reach based on the

number of people who sent ad requests in the prior 30 days to estimating Potential Reach based on

the number of people who had seen an ad on a Facebook product in the prior 30 days. The Ads

Targeting Team expected that this methodology change would cause a net decrease in Potential

Reach estimates because not everyone who sends an ad request receives an ad impression.

4.

In late 2018, as part of our analysis of the potential effects of this change on

advertiser behaviors and revenue, I conducted a predictive analysis to estimate the potential

revenue impact of the methodology change. This analysis is summarized in a document called

“Potential Reach Change – Central External Metrics Review,” which has the Bates number FB-

SINGER-00183561, and another document called “R&F Revenue Model of Audience Size,”

which has the Bates number FB-SINGER-00123477.

5.

The predictive analysis focused on (i) Reach & Frequency (“R&F”) campaigns and

(ii) brand campaigns. I focused my predictive analysis on R&F and brand campaigns because we

assumed advertisers were more likely to optimize these campaigns to reach a percentage of users,

and we assumed that the primary driver of any potential behavioral changes in response to a change

to Potential Reach estimates would come from advertisers who were optimizing for reach. The

predictive analysis did not consider campaigns that were not R&F or brand campaigns, because

ATTORNEYS AT LAW
SAN FRANCISCO

SHI DECLARATION

CASE NO. 3:18-cv-04978 JD

Page 4

we did not think that the methodology change would impact advertisers’ behavior when they were

not optimizing to reach a percentage of users.

6.

For this predictive analysis, I made a number of assumptions and tested different

possible scenarios because we did not yet have actual data reflecting any impact of the

contemplated methodology change on actual advertiser behavior or revenue. The “worst case”

scenario assumed that all brand and R&F campaigns would reduce spend proportionally to the

Potential Reach methodology change. The “bad” case scenario assumed that R&F campaigns

would directly reduce spend proportional to the Potential Reach methodology change. The “more

optimistic” scenario assumed that only advertisers who specified a percentage of the target

audience they wanted to reach in the campaign setting would reduce spend proportional to the

audience size change.

7.

The predictive analysis suggested that Facebook might experience a decrease in

advertising revenue for each of these scenarios based on the assumptions that some portion of

advertisers would reduce their spend. We presented these findings to the Central Metrics XFN

group that had to approve the methodology change to the Potential Reach estimates. Central

Metrics XFN approved the methodology change, and, on March 12, 2019, Facebook implemented

the change to base Potential Reach estimates on ad impressions rather than ad requests.

8.

After the March 12, 2019 methodology change, I ran further analyses to determine

whether there was any actual revenue impact from the methodology change. These analyses found

that there was no revenue impact overall.

9.

Specifically, for the three weeks following the methodology change, I evaluated the

impact of the methodology change on advertiser behavior by conducting an A/B test. An A/B test

is a test where you show two versions of a website or information to different sets of users and

compare the different responses to the two different versions. Here, I used a control group of over

3.5 million advertisers that continued to receive Potential Reach estimates based on ad requests,

and a test group of over 3.5 million advertisers that had begun to receive Potential Reach estimates

ATTORNEYS AT LAW
SAN FRANCISCO

SHI DECLARATION

CASE NO. 3:18-cv-04978 JD

Page 5

based on ad impressions pursuant to the methodology change. I then compared behavior and

revenue across the groups. The A/B test was limited to non-R&F advertisers.

10.

The A/B test results showed that the assumptions in my pre-launch predictive

analysis about brand campaigns were wrong. The A/B test results showed that the March 12, 2019

methodology change to Potential Reach estimates caused no changes to overall revenue for non-

R&F advertisers (the types of the advertisers who were the subjects of the post-launch test),

including because revenue depends on many other factors, such as an advertiser’s bid and budget.

It also showed no changes to advertiser budgets or other behaviors, such as advertisers’ targeting

selections, though some smaller advertisers increased their target audience sizes by a small

amount. I explored breakdowns by advertiser size, sales channels, and verticals (such as retail,

professional services, entertainment and media, consumer packaged goods).

11.

I am not aware of any other studies Facebook conducted evaluating the impact of

the Potential Reach ad-request-to-impression methodology change on advertiser behavior or

revenue.

I declare under penalty of perjury that the foregoing is true and correct, and that I executed


this Declaration on May ____, 2021, in Sunnyvale, California.

By:

Yiyang Shi

ATTORNEYS AT LAW
SAN FRANCISCO

SHI DECLARATION

CASE NO. 3:18-cv-04978 JD