Open Source Verification

Aaron Stein 0:07

Hello, and welcome to the Arms Control Wonk podcast. My name is Aaron Stein. I'm a contributor to Arms Control Wonk, and I'm also an associate fellow at the Royal United Services Institute in London. And I'm Jeffrey Lewis. I'm the founding publisher of the Arms Control Wonk blog and a scholar at the Monterey Institute.

[Sponsor message]

Jeffrey Lewis 0:35

Today, we have a special guest. We are joined by Corey Hinderstein. She is the Vice President for International Programs at the Nuclear Threat Initiative. How are you doing, Corey?

Corey Hinderstein 0:46

I'm doing great. Thank you.

Jeffrey 0:48

The real reason I wanted to invite Corey is because before she was at NTI, although she has continued this work Corey is really the reason I think that the nonproliferation community makes good use of open source tools. Before she went to NTI, she was the deputy director at the Institute for Science and International Security and was one of the first people who really took information and cross-referenced it with satellite images and then started doing the kind of work Open source work and then now at NTI she's doing a whole bunch of stuff on social media and Aaron we were at this meeting and we're sitting next to one another and there's this report of a fire and or explosion in Iran at Parchin and we each had exactly the same reaction which was to go into the seismic catalogs and see if there was evidence of

Of an explosion. And it struck us. And, you know, Corey, what do you think? I mean, that's different, right? That's not how we used to do things.

Corey 1:55

It's not at all how we used to do things. And in fact, I don't know whether it's evidence of how things have evolved or how we've gotten reliant on these sources and kind of forgotten our own old tradecraft. But either way, it's something different.

Jeffrey 2:09

Yeah. So, Corey, The short version is, so we go into the seismic catalogs and there are reports from defectors that say there was a big explosion at Parchin, which is where it's, it's a big military base where some of us think the Iranians may have done explosives testing for a nuclear weapons program. The Iranians they don't mention Parchin by name, but they, they admit that there is a fire at a military facility in that area.

Now we've got some satellite images. We don't see anything in the seismic data. And so, you know, Corey, I don't know. Do you think it blew up?

Corey 2:51

Well, I think your first statement, I don't know, is the correct one in the sense that, you know, sometimes we're selected questions, even with all of the Information that we have available to us now, sometimes things aren't definitive and we don't like that anymore. I think we like things to be more definitive. But in this case, I think that if I were to put money on it, I would say there maybe wasn't an explosion, but there certainly was some sort of event.

There was an event that had at least some circumstantial evidence that people felt the effects further away. But As you said, the seismic data isn't there. And so we're left with either this wasn't a big enough event to create a ripple or that data is somehow not complete.

Jeffrey 3:42

Yeah, I have to say I'm kind of bummed out a little bit because when the IRGC, the Revolutionary Guards Missile Base exploded, you could see that event really clearly in the catalog and you could actually get a location on it that put it pretty close to the actual base. And so you could then go and cross-reference those things.

And so I thought this was like a really useful capability because then when there were these reports that there had been an explosion at Fordow, the underground enrichment plant you know, I was pretty quickly skeptical because unlike the missile base, there was no event in the catalog. And so this time, you know, we have this really ambiguous middle ground where you know, we, we can see in the satellite photographs that there's destruction.

And so there was clearly at least a fire like the But there's no event in the catalog. And so we're sort of left with you know, is it that there was no explosion? Is it that there was an explosion, but it was too small to get picked up in the catalog? Or most distressingly are the Iranians and other actors sort of starting to scrub events out of the catalog.

So I don't know, but it's really different, isn't it?

Corey 4:48

It's different. And I think it also is It's curious that so many people now after this event, you know, the scramble was kind of on who's going to get the satellite photos, who's going to be the first one to put out the information that either confirms or denies, or, you know, in this case, continues to leave some ambiguity, but somebody is going to have some, some, you know, information that is a little bit more detailed than just the press reports or the, or somebody, you know, speaking from within the region, you know, that kind of, You know, information arms race is also new and it's added a new level of kind of competition among the analysts and also people just waiting for something to come out so that they can weigh in on the conversation and they can add, you know, add their knowledge from whatever unique perspective that they have.

Aaron 5:38

Actually, oh, Aaron, go ahead. What I found interesting about this particular debate is that it took on a new sense of relevance because of the alleged weapons, explosive testing that was related to the weapons program. But if you look at the satellite photos that came out, it was a completely different side of the base. And what I'm curious about is are we always going to go running when there's some sort of mysterious explosion at a military base in different parts of the world?

And are we at the point of like sensory overload here? The satellite photographs certainly help. But when it became not in the area that we're all that concerned about, it almost became as if, well, this was probably just an accident, right?

Jeffrey 6:23

Yeah, well, so two things about that. I mean, one is that I had a kind of up-close experience with the race, which is my friend Sean O'Connor at Jane's. They went and they got the satellite photograph and they looked at it and they did the analysis. And because they were on a very traditional publishing schedule, they waited, you know, the 24 hours, right, for the next news cycle.

And in the interim, Corey's old shop, the – Institute for science and international security. I hate that. I can't call it ISIS anymore.

Aaron 6:54

Right. Just caveat with a footnote ISIS, but not that one.

Jeffrey 6:59

Right. Right. The good ISIS they beat them. Right. So they, they emailed it out like 12 hours or something something before, but yeah, I mean, I think yeah, I think, well, and this is Corey's point, which, which I'll, I'll let her describe, but Yeah, well, okay, so the North Koreans, there's apparently just a fire that burned down the fictional birthplace of Kim Jong-il.

I'm desperate for those satellite photographs. Even though I totally believe the story is true, I still want to see the images.

Corey 7:32

Right. Well, and that's, I think, what we've started to talk about in the community is, you know, if there's not a picture, did it happen anymore? If there's not, you know, if people aren't tweeting from the center of, Of whatever big social change is happening, whether it's in, you know, the Arab Spring or recently in Hong Kong, you know, how are we going to get information that we trust?

Which is kind of remarkable given the fact that not too long ago, none of this information would have been considered information that we trust. And we would have said, well, yeah, yeah, yeah, but I'll believe it when I read it in the New York Times. And instead, I feel like the model has completely switched where we want, you know, we want the everyday person's Their word is more trustworthy than the traditional press in some cases and governments in many cases.

And I think maybe we need to modulate back to the center and really see all of this open source information as additive and illustrative, but not by itself definitive. And, you know, because I also think we're losing some of the filters, you know, that actually are helpful. I mean, your mention of the traditional press cycle is You know, there's value to that due diligence sometimes.

Aaron 8:48

Yep. No, I agree. Go ahead, Jeffrey.

Jeffrey 8:52

No, I was going to say, Corey, coin the term.

Corey 8:54

Oh, the CSI effect.

Jeffrey 8:57

Yeah.

Corey 8:59

So this is what, this is what I, you know, this is not a term that I've coined, you know, out of whole cloth. It exists apparently in the courts in the United States where this program for people who maybe aren't in the United States, aren't familiar with, This program is about the forensic investigators that go into crime scenes and collect evidence to put the bad guy away in the end.

And what we have seen in the U.S. courts, at least anecdotally, is that juries now expect that kind of evidence. And without it, they are hesitant to convict criminals because they think that this is the level of expectation they have as to what proof is anymore. And I feel like we're kind of getting there on open source, which is you can have a really well-studied, researched, good interview, all of the traditional sorts of scholarships.

But unless you have some of the open source information, maybe we're not quite believing it. And so I think the CSI effect is hurting us potentially. But if nothing else, I think people are just losing our traditional information collection and analysis skills.

Jeffrey 10:12

Yeah, I have to say, I wrote a column about this for Foreign Policy where it struck me that one of the reasons I think that the rollout of the evidence linking the Syrian government to the chemical weapons attack in the suburbs of Damascus was so unsuccessful was that it relied so much on old ways of telling, you know, so it had all of this kind of language of Well, we assess and our sources indicate, and it was very much this kind of, to pick a date completely at random, pre-2003 rhetoric.

And even if there was value in the way that we used to do things, I think a lot of people had the experience of looking at all of the stuff that was available on Twitter, all of the social media explanations and so much of the data, whether it was videos of people who were suffering or images of the munitions themselves, that they had this expectation for what an intelligence briefing might look like.

And then when you kind of show them the The declassified version, they're like, well, this is crap, right? Like, where's the real stuff? And I actually find this even with my students. When I give them declassified documents, they look at them and they're like, well, there's more, right? I mean, there's another document that's maybe not declassified. And often it's like, no, that's the NIE.

Like, that's all they had.

Corey 11:45

Well, and I found another interesting thing about the Syrian case is that CNN reported Lauded, they were the first to obtain the video seen by members of the Senate Intelligence Committee that the videos had depicted the scene of the chemical attacks on August 21st. And the other piece of what they reported is that the administration told the senators that the authenticity was verified by the intelligence community, but that the videos were the ones posted on YouTube.

So CNN's touting that they have the videos that the Senate Intelligence Committee saw that convinced them of the attacks But the videos were YouTube. So this wasn't some secret black program that had the smoking gun in this case, the smoking canister. This was YouTube. And the only thing that was different was that the IC was able to validate the video.

Aaron 12:36

Yeah. I mean, just to keep it in Syria, I mean, there's a double-edged sword to this as well. The prevalence of YouTube in smartphones and the ability of people to put things on the internet so quickly and then People like myself and others who then follow these accounts that you know put out these videos, you can uncover black programs that we weren't supposed to know about.

It puts more pressure, A, to keep things – programs that are supposed to be kept secret are more difficult to do so. And then the videos come out of them using it. So that becomes a reaffirmation of the administration's policy. So I'm thinking about the using of certain American weapons by certain groups in Syria against government targets.

You know, that program is supposed to be classified, but you can watch it unfold on YouTube. And then the administration uses it and says, well, see, we are doing more than what people are telling us about. It's a strange nexus with social media and open source.

[Sponsor message]

Jeffrey 14:39

Hey, did either of you see the YouTube videos that one of the Syrian opposition groups put up when they overran a Syrian government listening post near the Golan Heights? And it turns out it was staffed by Russians. Yep.

Corey 14:52

I did not see that.

Jeffrey 14:53

Oh, it's pretty amazing because the videos contain both enough geographic information and external shots that you can find the listening post. And, you know, it's on top of a hill right next to the Golan Heights. I mean, it's... It looks like a listening post. It's totally plausible. But they go inside, and there's all of these Russian maps and pictures of Russian troops and descriptions of Russian activities.

And so it turns out, yeah, it was a Russian listening post monitoring the Golan Heights for Moscow and Damascus. And it's like, yeah, that was supposed to be secret. Right. Yeah. Yeah. Oops. Exactly. Anyways. Well, there was a, there's another kind of set of Russia stuff that is, is worth talking about because when, when Corey and I were doing the we're calling it the fire drill, right?

Trying to figure out if it was a fire or an explosion in, in Iran. I was, I was actually giving a talk about the open source stuff on Russia, which we did a podcast about. And I, what I misidentified a scooter as a motorcycle. Is that what I did, Corey?

Corey 15:58

Yes, you did. I remember this.

Jeffrey 16:00

Yeah, she was not having any of this.

Corey 16:03

As a motorcycle rider, I think that this sort of – the integrity of this sort of analysis is really – it's important.

Aaron 16:12

It's called into question.

Corey 16:14

It really makes me question really all of your methods. But I'll –

Jeffrey 16:19

My professionalism, my expertise. Well, the thing is I didn't figure that out until like I went and looked on Facebook and then realized that, yeah, you're, you like the motorcycles. So but to pick up your earlier point about changing expectations and you know, the, the revelation of information. I think we, we saw like a ton of that in terms of You know, the MH17, the Malaysian airliners, that Malaysian airliner that the Russians shot down, right?

Because the administration was making the case that you could see Russian involvement based on all the social media. And... You know, this is the point that I made in the talk is we act like social media is somehow different from intelligence. But in fact, it's just evidence, right? And I don't think it makes a difference whether the video gets posted on YouTube or whether the video is burned on a diskette and given to a CIA case officer, right?

Either way, it's somebody there takes a video. And the way that you establish its provenance partly involves the person, but also involves All kinds of technical things that you can do on an open source side just as well as you can do on a classified side.

Corey 17:38

That's right. I mean, I think we hear a lot about kind of the shift towards the credibility of open source information. And I'm not sure that the intelligence community is as bought in as the public is yet on the equivalency that you've just noted. But I think they're getting there. And in some cases, people have... You know, come up to me and the work that I've done and said, well, but can't this information be fooled?

How do you, you know, aren't you afraid that you could be manipulated by false information, et cetera. And I say, well, sure. Both of those things are possible and many more kind of ways to maliciously use or manipulate data, but that's always been the case. And it is as important now as it, as it has ever been to make sure that you're, you have the ability to Vouch for the credibility of the of the data either the source or the data itself that's always been true it's always going to be true there's nothing different about it just because it's open source and there's nothing that's more scary about it because it's open source i would even add it it's it's harder now for a source to be not very credible because you can start digging into his or her past and find say very strange posts on like old

Aaron 18:50

Like social media platforms. And so you can instantly say, well, is this person somebody I should be relying on or not if you want to do your homework?

Jeffrey 18:56

Yeah, I would say like first – and this is why Corey is so much more successful than I am is because she moderates how she says these things. But whenever I hear this kind of spoofing problem, it's like, yeah, you're right. The intelligence community has never had a problem like curveball, right? And so it seems to me that if you're worried about people who are fabricators, your point, Aaron, is exactly right.

You're way better off with an open source situation where you can go back and check a lot of this stuff and compare it. And so it's a lot harder to fabricate the lie. And the example that I use in the MH17 – Which is our old friend DJ Petros. DJ Petros, yeah. Who has some very difficult to accept notions about Russia and conspiracy theories.

Yeah. And I think he would not be reliable except, you know, when he got that friend to go out and take the picture of the place where the surface-to-air missile was probably launched. You know, the houses match. So you don't have to believe him. He doesn't have to have credibility. You can actually verify the assertion based on things you can see.

And so I always say that this is the difference between showing and telling.

Corey 20:16

But I do think it's a little bit, you know, we're at a disadvantage in discussing this, discussing open source information in the open, because I do think that as much as we all have seen evidence that these intelligence communities Hasn't always gotten it right, to say the least. It's hard for me to know what the relative value of all this information is compared to the information that either is or could be collected through sources and methods, as they say.

And it may be that there isn't a lot different. Now, I'm not convinced of this. I actually think there is more on the open source. But it could be that there's not much more on the open source That it's just different sources of similar kinds of information with similar degrees of confidence. But what is different is the public conversation about it.

And that adds something. So you don't have to convince me that the social media information or the imagery or the commercial trade data or the scientific literature, you don't have to convince me that all that is better than what they have inside. I only have to be convinced that because of what we can do with it, Because it's open, that we are now at an advantage or that we are at least, we've augmented our ability to be successful analysts outside of the classified world.

Aaron 21:41

Yeah. One thing, just to sort of broaden this out and expand a little bit, one thing that I find it to be extremely useful for is the instantaneous manner with which you can get your hands on information that would Before the internet and all these things was very difficult to do, to explain leader decision making. So like you were saying in the beginning, like, are we waiting for pictures rather than doing analysis?

I think you can do both. You can use pictures to then verify assumptions you make. And I think you can make those assumptions about leader decision making, like say, find his or her PhD dissertation.

Jeffrey 22:16

To pick a purely hypothetical example, right?

Aaron 22:18

Yeah. I mean, maybe I've done this, maybe I haven't, you know, but you find somebody's PhD dissertation and you can find What their academic research tells them about, say, international affairs. And then you can compare that to the political decisions that they make. And then you can draw up a very clear picture rather than just a moment of time ripped from, say, the front pages of the New York Times.

And so I think the discussion should actually be much broader rather than just limited to just imagery and, say, sound wave data and stuff like that.

Jeffrey 22:48

Well, you know, what I was thinking of when Corey was making her point was – We focus so much on overhead images, and probably too much, but it's fun to talk about. I'm almost certain that the intelligence community gets to do much cooler things because I've watched the different things we can do as the commercial resolution gets better.

And now that there is commercial thermal data and there is even now commercial satellite video. So I think they still have some really important comparative advantages. But what I detect is a kind of evolution where it used to be that The intelligence community could do all of the work, and they could maybe tell us about it, and we could sort of use our brains to figure out which of those things was right or wrong.

But now they're really, I think, in a position where they have some comparative advantages that they have to emphasize. And the example of where I think that this gets totally screwed up is the whole tapping of Merkel's, the German chancellor, Angela Merkel's cell phone. Because it seems to me like that's an example where they have a special technical capability and they're really excited about it.

And they're doing this incredibly politically fraught thing. And what are they really getting out of that? What are they learning about her decision-making style that they couldn't get from a conference of academics who – Interact with German officials and know how the German government makes decisions and doing things like reading dissertations and talking to people who know her.

I mean, they get a little bit, right? And maybe if there's a particularly pressing event. But it seems to me that you could learn virtually everything you needed to know about her as a person and a decision maker without snooping on her cell phones. And so it gets to this question of is that really their comparative advantage or should they be focusing scarce intelligence resources on other narrow problems?

Corey 24:51

Well, and I think Aaron's point of timeliness is an important one because it's not just You know, is this information different? Is this communication? Is our ability to use them and most importantly, integrate different sources of information? I mean, in 1998, I did an analysis of the Pakistani nuclear test site. And this was using KVR-1000 Russian images.

And it was, you know, it took weeks to do everything. And I had to, you know, overlay maps that were printed on transparency. I mean, this is like, it feels like it's the Stone Ages. But eventually I was able to find the Roscoe Range site and relocate what had been publicly accepted in the Shigai Hills to a much more specific area.

But the key to it, the key to convincing me that I had the right site was Pakistani television images from the ground that I was able to compare to what I was seeing overhead. And in order to get that, we saw news reports from PTV that they had cameras on the ground when President Nawaz Sharif visited the site.

I contacted the D.C. office. They contacted Pakistan. They agreed to send me the raw footage on, you know, on the beta tape. It literally arrived at my door at the door of ISIS. In a burlap sack with a Manila label. I wish I was making that up. And then you extracted the information and compared and yada, yada, yada.

We have the full analysis. That took so long. And is the conclusion any less confident than one that I might make today? I don't think so. I think it's a confident conclusion. But I could probably have done that within 72 hours. If it happened today, because somebody would be on the ground having taken the picture. Somebody would have driven out.

There would have been this footage available on the PTV website that you could just pull off and do your analysis from there. I mean, it's profoundly different, but it's still the same action. It's just enabled.

Jeffrey 27:05

Yeah. Whenever I have to explain this to people who haven't thought about this very much, I think that is the key point. You could have done all of this stuff 10 years ago, 20 years ago, 30 years ago, but it was just so labor-intensive. The example I always use for people are either baby pictures or vacation pictures.

It used to be that... You would take the picture with a camera and then you had to take the roll of film to a human being who would then develop it. And they would give you a physical object. And if you were really smart, you'd have them make two copies of that physical object. And then you would take that physical object and put it in a little envelope and write an address on it, right?

And put a stamp on it and like walk it to the post office. And then a human being would deliver it Then you had successfully shared the image, but then the person who had it couldn't share it any further unless they wanted to give it up. I think it's a beautiful example where it takes you so long to get this video.

It's a beta, which is just Beautiful, right? And so the act of extracting that data is so much more labor intensive than it is today when, you know, they're going to probably put the footage right up to YouTube and, you know, you're annoyed that you have to wait 24 hours, right, for it to load. So I think it's such a, right, the method, just like Corey says, and I always agree with everything Corey says, but The method is the same old method, but what's different is how much more efficient we can be.

Aaron 28:33

So how should we use it? I mean I think that's what people keep asking. How should we use this to fill in our analysis and then – and how can this add to the debate more than it already has? I mean are we self-limiting in that we're focusing so much on imagery or should – are we losing our ability to analyze things correctly?

Have we not found the happy medium? Sort of wrapping this back together to the first point that was made – that Corey first made.

Jeffrey 28:57

Hey, Aaron. Yeah. You know what would be really great?

Aaron 29:00

What?

Jeffrey 29:01

Is if somebody had done a verification study that looked at new tools. Right. Corey, do you know anybody like that?

Corey 29:10

Well, because you just mentioned it and because of everything we have available to us, I've just done the study while we're here on the phone. That's right. It's cool. Yeah. So let me tell you what we found. No. At NTI, we did this broad... A BASE STUDY CALLED INNOVATING VERIFICATION. AND WE LOOKED AT A NUMBER OF DIFFERENT, WE LOOKED REALLY AT THREE DIFFERENT SUBSETS OF THE ISSUE OF KIND OF HOW ARE WE GOING TO MEET FUTURE ARMS CONTROL VERIFICATION NEEDS AND NON-POLITICAL VERIFICATION NEEDS.

AND TWO OF THE STUDIES WERE ON BASELINE, VERIFYING BASELINE INVENTORY FOR MATERIALS AND WARHEADS AND THEN ALSO ON BUILDING GLOBAL CAPACITY, HOW DO WE INVOLVE MORE And more countries and more experts in the task. And actually, both of those topics recognized that the third topic was relevant. And the third topic was societal verification. So, you know, one thing on building global capacity, and it's something that we're talking about, about the speed and the tools is When we talk about intelligence, we're often referring to kind of the U.S. system or the other major developed systems.

But there are countries out there that don't have intelligence infrastructure like the United States does. And in fact, they are now using open source as kind of their intelligence sources. And they're playing in these conversations in ways they didn't before. But so are individuals. And so is, you know, so is civil society. And that's where our societal verification study was really looking.

And with the great participation of a working group, including Brian Lee over at the Center for Nonproliferation Studies with Jeffrey, we really tried to ask the question, what is this adding to verification? What could all of this open source add to verification? And we concluded a few things, but the key thing I would say is that governments haven't figured it out and they need to.

We had a long discussion about whether this is, you know, kind of contributive or transformative. And we all had their gut feeling that it's transformative, but we can't prove it because governments haven't figured it out and, in fact, are really nervous about introducing open source in any, you know, formal way into traditional treaty verification, which is usually a guy on the ground, an exchange of data, an allowance for satellite, you know, national satellite imagery.

And so we think that for the next generations of arms control, when we're talking about smaller items, smaller limits, that without open source, you're going to have two things. One is you're going to restrict what you can do. And the second is you're going to have a whole civil society that's going to question the conclusions that you reach about treaty compliance.

And that's something where we've seen that case study just in the few months since we published our report in July. With regard to the INF. And Jeffrey, you did it. I mean, this is an issue that you focused on.

Jeffrey 32:13

Yeah. Well, so I wanted to ask you about that because we try not to mention the things we mentioned off mic, but we didn't let you tell us what you were thinking when you said INF is a great case for this. And I can't get my head around how that would be or what that would illustrate. So tell me, and then you get my on-air surprise.

Aaron 32:34

And you mentioned that you listened to the podcast, so you instantly ingratiated yourself to us.

Corey 32:40

Well, I try to keep up on what's happening with the most important voices.

Jeffrey 32:46

How can you hear it over the roar of the motorcycle?

Corey 32:51

You'd be surprised. I do listen to podcasts in my helmet. It's called Bluetooth. It's brilliant. Anyway, so what happened is you've got this treaty, and there are kind of traditional ways of exchanging data in each country, the U.S. and Russia. Drawing their own conclusions about the other one's compliance with the treaty. And you then have, you know, kind of a broad civil society expert community that, because of the availability of open source information, is starting to reach its own conclusions about compliance with the treaty, and specifically with the Russian activity regarding potential future deployments of missiles that may or may not violate the treaty.

And so, When you have the State Department compliance report come out, I think you have to really ask yourself, would that have come out in the same way if there wasn't already a conversation going on that now the State Department is accountable to? So they're no longer just accountable to the Russians. They're accountable to this expert community.

And this is something we said in our verification report. There's a whole stream of things that are going to need to happen within official channels. But the fact that Kind of parallel activities are happening in the expert community means that not only should government take that and use it and use it for queuing and kind of see it as a resource, but now if they reach a different conclusion than that which is reached outside, they're going to have to answer to it.

And this is a perfect example of where they had to answer to it. And because of it, they now have an ongoing discussion with Russia. So I think you did what we talked about in our report, which is this shadow verification path. And because of the sources, your conclusions were credible, and therefore they needed to be accounted for.

Jeffrey 34:48

Wow. Yeah. Holy cow, Aaron. Is that a happy note?

Aaron 34:55

That is a happy note. We didn't even know we did this. I mean, you did this, then we talked about it on the podcast.

Jeffrey 35:02

That's fantastic. It was great that we told Corey we had to end on a happy note.

Aaron 35:05

All right. Well, Jeffrey, Corey, this was really interesting. Thanks again for joining me here on the Arms Control Podcast.

[Sponsor message]

Well, thanks again. Thanks, everybody, for listening.