EXHIBITS re [296] Declaration in Support,,,,,, Exhibit [296] Exhibit L, Unredacted Version of document originally filed under seal at Dkt. 293-8, refiled pursuant to Dkts. 320, 325 and 325-5 filed byFacebook, Inc.. (Attachments: # (1) Exhibit [296] Exhibit M, Unredacted Version of document originally filed under seal at Dkt. 293-10, refiled pursuant to Dkts. 320, 325 and 325-5, # (2) Exhibit [296] Exhibit 108, Unredacted Version of document originally filed under seal at Dkt. 293-18, refiled pursuant to Dkts. 320, 325 and 325-5, # (3) Exhibit [296] Exhibit 111, Unredacted Version of document originally filed under seal at Dkt. 293-21, refiled pursuant to Dkts. 320, 325 and 325-5, # (4) Exhibit [296] Exhibit 113, Unredacted Version of document originally filed under seal at Dkt. 293-23, refiled pursuant to Dkts. 320, 325 and 325-5, # (5) Exhibit [296] Exhibit 115, Unredacted Version of document originally filed under seal at Dkt. 293-25, refiled pursuant to Dkts. 320, 325 and 325-5, # (6) Exhibit [296] Exhibit 118, Unredacted Version of document originally filed under seal at Dkt. 293-28, refiled pursuant to Dkts. 320, 325 and 325-5, # (7) Exhibit [296] Exhibit 119, Unredacted Version of document originally filed under seal at Dkt. 293-29, refiled pursuant to Dkts. 320, 325 and 325-5, # (8) Exhibit [296] Exhibit 120, Unredacted Version of document originally filed under seal at Dkt. 293-30, refiled pursuant to Dkts. 320, 325 and 325-5, # (9) Exhibit [296] Exhibit 121, Unredacted Version of document originally filed under seal at Dkt. 293-31, refiled pursuant to Dkts. 320, 325 and 325-5, # (10) Exhibit [296] Exhibit 122, Unredacted Version of document originally filed under seal at Dkt. 293-32, refiled pursuant to Dkts. 320, 325 and 325-5, # (11) Exhibit [296] Exhibit 124, Unredacted Version of document originally filed under seal at Dkt. 293-34, refiled pursuant to Dkts. 320, 325 and 325-5, # (12) Exhibit [296] Exhibit 125, Unredacted Version of document originally filed under seal at Dkt. 293-35, refiled pursuant to Dkts. 320, 325 and 325-5, # (13) Exhibit [296] Exhibit 129, Unredacted Version of document originally filed under seal at Dkt. 293-36, refiled pursuant to Dkts. 320, 325 and 325-5, # (14) Exhibit [296] Exhibit 137, Unredacted Version of document originally filed under seal at Dkt. 293-42, refiled pursuant to Dkts. 320, 325 and 325-5, # (15) Exhibit [296] Exhibit 3 to Tadelis Declaration, Unredacted Version of document originally filed under seal at Dkt. 293-46, refiled pursuant to Dkts. 320, 325 and 325-5, # (16) Exhibit [296] Exhibit 4 to Tadelis Declaration, Unredacted Version of document originally filed under seal at Dkt. 293-47, refiled pursuant to Dkts. 320, 325 and 325-5, # (17) Exhibit [296] Exhibit 5 to Tadelis Declaration, Unredacted Version of document originally filed under seal at Dkt. 293-48, refiled pursuant to Dkts. 320, 325 and 325-5, # (18) Exhibit [296] Exhibit 6 to Tadelis Declaration, Unredacted Version of document originally filed under seal at Dkt. 293-49, refiled pursuant to Dkts. 320, 325 and 325-5, # (19) Exhibit [296] Exhibit 3 to Reibstein Declaration, Unredacted Version of document originally filed under seal at Dkt. 293-50, refiled pursuant to Dkts. 320, 325 and 325-5)(Related document(s)[296]) (Blunschi, Melanie) (Filed on 7/20/2021)
There was a problem locating the requested document.
Page 1 Exhibit LPage 2 1
2
3
4
5
6
7
8
9
10
11
LATHAM & WATKINS LLP
Elizabeth L. Deeley (CA Bar No. 230798)
elizabeth.deeley@lw.com
Melanie M. Blunschi (CA Bar No. 234264)
melanie.blunschi@lw.com
Nicole C. Valco (CA Bar No. 258506)
nicole.valco@lw.com
505 Montgomery Street, Suite 2000
San Francisco, CA 94111-6538
Tel.: +1.415.391.0600/Fax: +1.415.395.8095
Andrew B. Clubok (pro hac vice)
andrew.clubok@lw.com
Susan E. Engel (pro hac vice)
susan.engel@lw.com
555 Eleventh Street, N.W., Suite 1000
Washington, D.C. 20004
Tel.: +1.202.637.2200/Fax: +1.202.637.2201
13
Hilary H. Mattis (CA Bar No. 271498)
hilary.mattis@lw.com
140 Scott Drive
Menlo Park, CA 94025
Tel.: +1.650.328.4600/Fax: +1.650.463.2600
14
Attorneys for Defendant Facebook, Inc.
12
15
UNITED STATES DISTRICT COURT
16
NORTHERN DISTRICT OF CALIFORNIA
17
SAN FRANCISCO DIVISION
18
19
20
DZ RESERVE, and CAIN MAXWELL (d/b/a
MAX MARTIALIS), individually and on
behalf of all others similarly situated,
21
22
23
24
Plaintiffs,
v.
FACEBOOK, INC.,
Case No. 3:18-cv-04978 JD
DECLARATION OF YIYANG SHI IN
SUPPORT OF FACEBOOK’S
OPPOSITION TO PLAINTIFFS’
MOTION FOR CLASS
CERTIFICATION
Hon. James Donato
Defendant.
25
26
27
28
ATTORNEYS AT LAW
SAN FRANCISCO
SHI DECLARATION
CASE NO. 3:18-cv-04978 JDPage 3 1
I, Yiyang Shi, declare as follows:
2
1.
I am a Data Scientist on the Ads Core Data Analytics Team at Defendant Facebook,
3
Inc. (“Facebook”). I submit this declaration in support of Facebook’s Opposition to Plaintiffs’
4
Motion for Class Certification. I have personal knowledge of the matters stated herein and, if
5
called upon, I could and would competently testify thereto.
6
2.
I have a Master’s degree in Statistics and Operations Research and a Bachelor’s
7
degree in Economics from the University of North Carolina at Chapel Hill. I have worked at
8
Facebook since August 2018. I previously was a Data Scientist at Uber.
9
3.
In late 2018 to early 2019, I understand the Ads Targeting Team was exploring
10
whether to change one of the inputs to the Potential Reach estimate. Specifically, the Ads
11
Targeting Team was considering whether to change from estimating Potential Reach based on the
12
number of people who sent ad requests in the prior 30 days to estimating Potential Reach based on
13
the number of people who had seen an ad on a Facebook product in the prior 30 days. The Ads
14
Targeting Team expected that this methodology change would cause a net decrease in Potential
15
Reach estimates because not everyone who sends an ad request receives an ad impression.
16
4.
In late 2018, as part of our analysis of the potential effects of this change on
17
advertiser behaviors and revenue, I conducted a predictive analysis to estimate the potential
18
revenue impact of the methodology change. This analysis is summarized in a document called
19
“Potential Reach Change – Central External Metrics Review,” which has the Bates number FB-
20
SINGER-00183561, and another document called “R&F Revenue Model of Audience Size,”
21
which has the Bates number FB-SINGER-00123477.
22
5.
The predictive analysis focused on (i) Reach & Frequency (“R&F”) campaigns and
23
(ii) brand campaigns. I focused my predictive analysis on R&F and brand campaigns because we
24
assumed advertisers were more likely to optimize these campaigns to reach a percentage of users,
25
and we assumed that the primary driver of any potential behavioral changes in response to a change
26
to Potential Reach estimates would come from advertisers who were optimizing for reach. The
27
predictive analysis did not consider campaigns that were not R&F or brand campaigns, because
28
ATTORNEYS AT LAW
SAN FRANCISCO
1
SHI DECLARATION
CASE NO. 3:18-cv-04978 JDPage 4 1
we did not think that the methodology change would impact advertisers’ behavior when they were
2
not optimizing to reach a percentage of users.
3
6.
For this predictive analysis, I made a number of assumptions and tested different
4
possible scenarios because we did not yet have actual data reflecting any impact of the
5
contemplated methodology change on actual advertiser behavior or revenue. The “worst case”
6
scenario assumed that all brand and R&F campaigns would reduce spend proportionally to the
7
Potential Reach methodology change. The “bad” case scenario assumed that R&F campaigns
8
would directly reduce spend proportional to the Potential Reach methodology change. The “more
9
optimistic” scenario assumed that only advertisers who specified a percentage of the target
10
audience they wanted to reach in the campaign setting would reduce spend proportional to the
11
audience size change.
12
7.
The predictive analysis suggested that Facebook might experience a decrease in
13
advertising revenue for each of these scenarios based on the assumptions that some portion of
14
advertisers would reduce their spend. We presented these findings to the Central Metrics XFN
15
group that had to approve the methodology change to the Potential Reach estimates. Central
16
Metrics XFN approved the methodology change, and, on March 12, 2019, Facebook implemented
17
the change to base Potential Reach estimates on ad impressions rather than ad requests.
18
8.
After the March 12, 2019 methodology change, I ran further analyses to determine
19
whether there was any actual revenue impact from the methodology change. These analyses found
20
that there was no revenue impact overall.
21
9.
Specifically, for the three weeks following the methodology change, I evaluated the
22
impact of the methodology change on advertiser behavior by conducting an A/B test. An A/B test
23
is a test where you show two versions of a website or information to different sets of users and
24
compare the different responses to the two different versions. Here, I used a control group of over
25
3.5 million advertisers that continued to receive Potential Reach estimates based on ad requests,
26
and a test group of over 3.5 million advertisers that had begun to receive Potential Reach estimates
27
28
ATTORNEYS AT LAW
SAN FRANCISCO
2
SHI DECLARATION
CASE NO. 3:18-cv-04978 JDPage 5 1
based on ad impressions pursuant to the methodology change. I then compared behavior and
2
revenue across the groups. The A/B test was limited to non-R&F advertisers.
3
10.
The A/B test results showed that the assumptions in my pre-launch predictive
4
analysis about brand campaigns were wrong. The A/B test results showed that the March 12, 2019
5
methodology change to Potential Reach estimates caused no changes to overall revenue for non-
6
R&F advertisers (the types of the advertisers who were the subjects of the post-launch test),
7
including because revenue depends on many other factors, such as an advertiser’s bid and budget.
8
It also showed no changes to advertiser budgets or other behaviors, such as advertisers’ targeting
9
selections, though some smaller advertisers increased their target audience sizes by a small
10
amount. I explored breakdowns by advertiser size, sales channels, and verticals (such as retail,
11
professional services, entertainment and media, consumer packaged goods).
12
11.
I am not aware of any other studies Facebook conducted evaluating the impact of
13
the Potential Reach ad-request-to-impression methodology change on advertiser behavior or
14
revenue.
15
16
17
I declare under penalty of perjury that the foregoing is true and correct, and that I executed
this Declaration on May ____, 2021, in Sunnyvale, California.
18
19
By:
Yiyang Shi
20
21
22
23
24
25
26
27
28
ATTORNEYS AT LAW
SAN FRANCISCO
3
SHI DECLARATION
CASE NO. 3:18-cv-04978 JD
PDF Page 1
PlainSite Cover Page
PDF Page 2
Case 3:18-cv-04978-JD Document 339 Filed 07/20/21 Page 1 of 5
Exhibit L
PDF Page 3
Case 3:18-cv-04978-JD Document 339 Filed 07/20/21 Page 2 of 5
1
2
3
4
5
6
7
8
9
10
11
LATHAM & WATKINS LLP
Elizabeth L. Deeley (CA Bar No. 230798)
elizabeth.deeley@lw.com
Melanie M. Blunschi (CA Bar No. 234264)
melanie.blunschi@lw.com
Nicole C. Valco (CA Bar No. 258506)
nicole.valco@lw.com
505 Montgomery Street, Suite 2000
San Francisco, CA 94111-6538
Tel.: +1.415.391.0600/Fax: +1.415.395.8095
Andrew B. Clubok (pro hac vice)
andrew.clubok@lw.com
Susan E. Engel (pro hac vice)
susan.engel@lw.com
555 Eleventh Street, N.W., Suite 1000
Washington, D.C. 20004
Tel.: +1.202.637.2200/Fax: +1.202.637.2201
13
Hilary H. Mattis (CA Bar No. 271498)
hilary.mattis@lw.com
140 Scott Drive
Menlo Park, CA 94025
Tel.: +1.650.328.4600/Fax: +1.650.463.2600
14
Attorneys for Defendant Facebook, Inc.
12
15
UNITED STATES DISTRICT COURT
16
NORTHERN DISTRICT OF CALIFORNIA
17
SAN FRANCISCO DIVISION
18
19
20
DZ RESERVE, and CAIN MAXWELL (d/b/a
MAX MARTIALIS), individually and on
behalf of all others similarly situated,
21
22
23
24
Plaintiffs,
v.
FACEBOOK, INC.,
Case No. 3:18-cv-04978 JD
DECLARATION OF YIYANG SHI IN
SUPPORT OF FACEBOOK’S
OPPOSITION TO PLAINTIFFS’
MOTION FOR CLASS
CERTIFICATION
Hon. James Donato
Defendant.
25
26
27
28
ATTORNEYS AT LAW
SAN FRANCISCO
SHI DECLARATION
CASE NO. 3:18-cv-04978 JD
PDF Page 4
Case 3:18-cv-04978-JD Document 339 Filed 07/20/21 Page 3 of 5
1
I, Yiyang Shi, declare as follows:
2
1.
I am a Data Scientist on the Ads Core Data Analytics Team at Defendant Facebook,
3
Inc. (“Facebook”). I submit this declaration in support of Facebook’s Opposition to Plaintiffs’
4
Motion for Class Certification. I have personal knowledge of the matters stated herein and, if
5
called upon, I could and would competently testify thereto.
6
2.
I have a Master’s degree in Statistics and Operations Research and a Bachelor’s
7
degree in Economics from the University of North Carolina at Chapel Hill. I have worked at
8
Facebook since August 2018. I previously was a Data Scientist at Uber.
9
3.
In late 2018 to early 2019, I understand the Ads Targeting Team was exploring
10
whether to change one of the inputs to the Potential Reach estimate. Specifically, the Ads
11
Targeting Team was considering whether to change from estimating Potential Reach based on the
12
number of people who sent ad requests in the prior 30 days to estimating Potential Reach based on
13
the number of people who had seen an ad on a Facebook product in the prior 30 days. The Ads
14
Targeting Team expected that this methodology change would cause a net decrease in Potential
15
Reach estimates because not everyone who sends an ad request receives an ad impression.
16
4.
In late 2018, as part of our analysis of the potential effects of this change on
17
advertiser behaviors and revenue, I conducted a predictive analysis to estimate the potential
18
revenue impact of the methodology change. This analysis is summarized in a document called
19
“Potential Reach Change – Central External Metrics Review,” which has the Bates number FB-
20
SINGER-00183561, and another document called “R&F Revenue Model of Audience Size,”
21
which has the Bates number FB-SINGER-00123477.
22
5.
The predictive analysis focused on (i) Reach & Frequency (“R&F”) campaigns and
23
(ii) brand campaigns. I focused my predictive analysis on R&F and brand campaigns because we
24
assumed advertisers were more likely to optimize these campaigns to reach a percentage of users,
25
and we assumed that the primary driver of any potential behavioral changes in response to a change
26
to Potential Reach estimates would come from advertisers who were optimizing for reach. The
27
predictive analysis did not consider campaigns that were not R&F or brand campaigns, because
28
ATTORNEYS AT LAW
SAN FRANCISCO
1
SHI DECLARATION
CASE NO. 3:18-cv-04978 JD
PDF Page 5
Case 3:18-cv-04978-JD Document 339 Filed 07/20/21 Page 4 of 5
1
we did not think that the methodology change would impact advertisers’ behavior when they were
2
not optimizing to reach a percentage of users.
3
6.
For this predictive analysis, I made a number of assumptions and tested different
4
possible scenarios because we did not yet have actual data reflecting any impact of the
5
contemplated methodology change on actual advertiser behavior or revenue. The “worst case”
6
scenario assumed that all brand and R&F campaigns would reduce spend proportionally to the
7
Potential Reach methodology change. The “bad” case scenario assumed that R&F campaigns
8
would directly reduce spend proportional to the Potential Reach methodology change. The “more
9
optimistic” scenario assumed that only advertisers who specified a percentage of the target
10
audience they wanted to reach in the campaign setting would reduce spend proportional to the
11
audience size change.
12
7.
The predictive analysis suggested that Facebook might experience a decrease in
13
advertising revenue for each of these scenarios based on the assumptions that some portion of
14
advertisers would reduce their spend. We presented these findings to the Central Metrics XFN
15
group that had to approve the methodology change to the Potential Reach estimates. Central
16
Metrics XFN approved the methodology change, and, on March 12, 2019, Facebook implemented
17
the change to base Potential Reach estimates on ad impressions rather than ad requests.
18
8.
After the March 12, 2019 methodology change, I ran further analyses to determine
19
whether there was any actual revenue impact from the methodology change. These analyses found
20
that there was no revenue impact overall.
21
9.
Specifically, for the three weeks following the methodology change, I evaluated the
22
impact of the methodology change on advertiser behavior by conducting an A/B test. An A/B test
23
is a test where you show two versions of a website or information to different sets of users and
24
compare the different responses to the two different versions. Here, I used a control group of over
25
3.5 million advertisers that continued to receive Potential Reach estimates based on ad requests,
26
and a test group of over 3.5 million advertisers that had begun to receive Potential Reach estimates
27
28
ATTORNEYS AT LAW
SAN FRANCISCO
2
SHI DECLARATION
CASE NO. 3:18-cv-04978 JD
PDF Page 6
Case 3:18-cv-04978-JD Document 339 Filed 07/20/21 Page 5 of 5
1
based on ad impressions pursuant to the methodology change. I then compared behavior and
2
revenue across the groups. The A/B test was limited to non-R&F advertisers.
3
10.
The A/B test results showed that the assumptions in my pre-launch predictive
4
analysis about brand campaigns were wrong. The A/B test results showed that the March 12, 2019
5
methodology change to Potential Reach estimates caused no changes to overall revenue for non-
6
R&F advertisers (the types of the advertisers who were the subjects of the post-launch test),
7
including because revenue depends on many other factors, such as an advertiser’s bid and budget.
8
It also showed no changes to advertiser budgets or other behaviors, such as advertisers’ targeting
9
selections, though some smaller advertisers increased their target audience sizes by a small
10
amount. I explored breakdowns by advertiser size, sales channels, and verticals (such as retail,
11
professional services, entertainment and media, consumer packaged goods).
12
11.
I am not aware of any other studies Facebook conducted evaluating the impact of
13
the Potential Reach ad-request-to-impression methodology change on advertiser behavior or
14
revenue.
15
16
17
I declare under penalty of perjury that the foregoing is true and correct, and that I executed
this Declaration on May ____, 2021, in Sunnyvale, California.
18
19
By:
Yiyang Shi
20
21
22
23
24
25
26
27
28
ATTORNEYS AT LAW
SAN FRANCISCO
3
SHI DECLARATION
CASE NO. 3:18-cv-04978 JD
Third party AI contribution in government documents
Possible
Document content is reproduced directly from government sources. Some filings in our database may themselves contain AI-generated content submitted by third parties to government agencies.
Back to 
