When Open Source Goes Wrong

Aaron Stein 0:00

How do I transition out of this thing?

Jeffrey Lewis 0:02

I have no idea you f***ed this up so badly. You are listening to the Arms Control Wong Podcast. Only and therefore leading podcast on arms control, disarmament, and nonproliferation. I'm Jeffrey Lewis, a senior fellow at the Middlebury Institute of International Studies and the founding publisher of armscontrolwonk.com.

Aaron 0:28

And I'm Aaron Stein. I'm a contributor to the Arms Control Wonk blog. I'm also a doctoral fellow at the Geneva Center for Security Policy and an associate fellow at the Royal United Services Institute in London. How are you doing, Aaron? I'm doing well. It's been snowing the past three days, actually.

Jeffrey 0:42

Really?

Aaron 0:43

Yeah. Yeah. Good for the skiing, but not good for walking around Geneva, I'll tell you that much. It's been sunny and beautiful here. Looking at the pictures on Facebook. I am California dreaming at the moment. We've had two issues this week. That goes intersect. We've had two cases of open source going wrong. Very, very wrong. Yeah. So the first was a Chinese missile training exercise that was blown up into something way bigger by a Korean newspaper.

And the second is that the Iranians finally fulfilled the prognostications of the Rundfeld Commission and had an ICBM ready to fly, neither of which was true. Not one bit. Not one bit. But the photograph said that they were true. And I think you had to write two different columns, one talking about the Chinese missile test and the second talking about why perhaps the Iranian thing was not an ICBM but maybe something different.

Let's start with China. Where did the Korean newspaper go wrong?

Jeffrey 1:42

It would be a shorter conversation to talk about where they went right. OK. Let's start there.

Aaron 1:46

What did they get right?

Jeffrey 1:49

Nothing.

Aaron 1:51

OK. So they got nothing right. Moving on. Well, I think they said that the Chinese had deployed nuclear-armed missiles to the slopes of Mount Bekdu. Bekdu or Bekdu.

Jeffrey 2:01

Yeah, yeah, yeah, which the Chinese call Changbai. So that mountain, right, it's like a big deal in Korean culture. You know, Kim Jong-il was allegedly born on the slopes of snowy Mount Baekdu. So it's like a thing. And so I think in the Korean press, the idea that the Chinese would station nuclear weapons on the Chinese side of Mount Baekdu was like the kind of thing that really got people's attention.

But what really happened was that there was a – Chinese press release about a winter training exercise and someone in China tried to geolocate the training exercise. And they did a good job in the sense that based on the temperatures described in the original Chinese state media story and looking at the trees, they figured it was in Jilin province.

And that part was right. But what they then screwed up is going this extra step and asserting that it was a full-time deployment and that the deployment was at more than 1,000 meters in elevation, and therefore it must be on the slopes of this sacred mountain.

Aaron 3:20

I'm going to start at the beginning. One of the reasons I really like open source is for stuff like where they got it right, that you look at temperature data and then you're able from that temperature data to make sort of a general claim about where something is. Where open source can go seriously wrong is exactly what this person did.

If you go too far and then you make a mistake about the geolocation, but you're so confident in your geolocation that then newspapers pick it up and run with it and it can perhaps affect policy decisions.

Jeffrey 3:50

Yeah. So the thing that really – Irritated me was you have to interpret data in context. I mean, the great thing about open source data is it allows you to fuse different streams of data into one picture. So you have the ability to cross-check things. I mean, the reason I didn't like the old way that we approached these problems is because we You know, you had to sort of believe someone or rely on someone, and it was really hard to compare competing stories.

So that's like the promise of open source. And this is a case where somebody didn't do that. And they left out basically the most important fact, which is that. In Jilin province, about 100 kilometers from Mount Baekdu, there is a site. And it was originally a missile testing base. And now I think it is a second artillery training base.

And, you know, it may still have both functions, but it's near a place called Jingyu. And we were able to geolocate the training base. And it turns out it's at the same location as the old launch site, which is why we're a little confused about the relationship. But once you do that, it becomes super clear that what's happened is instead of this being a deployment, what has happened is that a unit has gone to the training center and has then conducted an exercise out of the training center, which is something...

Right. This is the context. That's something Chinese missile units do. They don't conduct training at their operational locations because it would give away the locations. And so they travel to these dedicated training sites. And that's why the Chinese government felt OK about putting out the pictures. They didn't care if we geolocated the exercise because it was just a training location and the training location is known to everyone.

Aaron 5:48

Right. I mean, what I find – It's sort of like the way that the diffusion of the capabilities to do this is both positive and negative. Positive in the fact that we can do this. Negative in the fact that somebody can do this without knowing the complete context, like you said, so you can lead to assumptions or assertions that are based on, like, I want to say half-truths.

Jeffrey 6:13

Right. Right. Well, I think that's that's exactly right. So once you don't have that context, right, once you assume that instead of being a training exercise, it must be real. And by the way, the original Chinese was totally clear that this was a training exercise and they do them all the time. But this was newsworthy because I can't believe I got to write this in the column.

Right. These guys were like freezing their balls off out in the weather. It was cold. It was really, really cold. Yeah, yeah, yeah. And that's why it was newsworthy is that it was winter training. And so when you lose that context, then you end up making these crazy claims. One of the things that we were able to do, I asked Catherine Dill who is – she's been on the podcast and she's the – Yeah, she's an old guest.

Yeah, yeah, yeah. She's also the keeper of the Center for Nonproliferation's Second Artillery Order of Battle.

Aaron 7:05

I love that the Center for Nonproliferation has a keeper for the Second Artillery's order of battle. More people should go to the site.

Jeffrey 7:12

She takes – there is a US publication that is an open source analysis that looks at all of the open source mentions of people and units. And it's not perfect. And so we then do our own analysis and we look at some other people's analysis. And so she looked up the commander who was responsible for the exercise.

And it turns out, right, he's not based out of that area. He's based out of a different location further away. And so it was super clear that they had basically come to Jingyu to conduct this exercise in the winter.

Aaron 7:51

Yeah. I mean, the story, I think, in the Korean press got a lot of attention. It raised a question for me about open source in general. It's like when it's done badly, it can actually have serious implications. If the story whips around and becomes widely accepted, it could almost reinforce conspiracy theory that you'll then never be able to disprove going forward because they'll be like, look, this was in the newspaper back when, and it said that the Chinese put Nuclear-armed missiles on the slopes of Mount Becton.

Jeffrey 8:23

Do you know what I mean? Well, I totally know. That's how my piece ends, with me complaining that I'm going to have to spend years being that irritating person at the meeting who, like, corrects the lazy guy who just repeats the claim. So, right, I'm going to end up as the jerk. What really was irritating about that piece was the...

Although the geolocation to Jilin province was correct. This person then went one step further and said, when I look at the trees, these trees show that the elevation is more than a thousand meters. And like Jingyu is at like 700 meters. And so, you know, that sounds like really competent and really persuasive. The tree line, you know, is a, A certain level, dah, dah, dah, dah, dah.

Well, so I went and, you know, it turns out there is, there are like vegetation atlases of Jilin province. Of course there are. Yeah, of course there are. And, and I, I was too lazy to go get one of the original atlases in Chinese, but there are plenty of papers, academic papers that summarize them. And it turns out the tree line is at 1100 meters, but below 1100 meters, you have these mixed You have mixed forests.

So there will be sometimes when you'll see like pine trees and evergreens, and then there'll be other times where you see trees with leaves. And so I ended up doing like what I thought was the simplest thing, which is instead of relying on this kind of textbook knowledge, Google Earth has all these panoramio photographs in them.

Aaron 9:55

Oh, yeah, of course.

Jeffrey 9:56

And I just like drove, you know, virtually drove around the Jinyu base looking at pictures of the trees. And, you know, and they match. I mean, you know, obviously I can't tell if it's the same tree, but the point is you can see for yourself that the claim that that mixture of trees only occurs above a thousand meters is just patently false because people have taken pictures of the forest and they've put them online and they've geolocated the pictures and there are enough of them.

That even if one or two are wrong, it is clear that the trees that are around these guys are consistent with the trees that are at the Jingyu Training Center. And so you couldn't exclude it.

Aaron 10:34

Yeah, I think it's a constant. I mean I could be completely wrong here, but I think it's a constant about which elevation – Over which trees don't grow. But I'm going off on a tangent here. One thing that I find – Yeah, this was – you know what this was?

Jeffrey 10:49

This was a distinction between the types of trees you find at different elevations.

Aaron 10:52

Oh, interesting.

Jeffrey 10:53

Which is classically to – right, it's overconfident. It's I have this little piece of data and I'm going to try to extrapolate these huge things out of it without having any – Any sense of context. I'm cutting you off, but I, I actually, after I wrote this darn piece we went to Tahoe for a couple of days.

Aaron 11:13

Yeah.

Jeffrey 11:14

And I couldn't resist looking for the changeover where we went from mixed forests to sub Alpine forests. The Donner pass. Right. It turns out it's lower than that. And it's not, it's not an even line. Okay. They fade out. So yeah.

Aaron 11:33

What the guy It leaves me wanting more. So when I look at the type of analysis that was done there, you can actually see how intelligence agencies would do it with more access to photographs so they could track missile patrols. One missile goes up there and then you could actually follow them back to which base they went.

Actually, it leaves me wanting more. Does that make sense? Right.

Jeffrey 11:55

Well, that's why we have a CNS second artillery order of battle. I know. I mean, that's what we do. It's actually really fun. We had this other case that is like perfectly illustrative of how fun this is. There was a guy and he appeared in a couple of images. But the videos weren't so good. And we couldn't exactly tell if it was the same guy in terms of his face.

And in one, he was named. But in the other pictures, he wasn't named. And we really wanted to make sure it was like the same Chinese guy. And so we matched the camo pattern on his shirt, right? Because he had a digital camo pattern that was unique. And so we ended up – we couldn't see his face clearly, but we could see the patterns on his collar and his sleeves.

And like unless it was like some similar-looking dude borrowing his shirt, we were able to tell it was the same guy. So yeah, it's fun.

Aaron 12:49

We had a second case coming out. I think it was by Israel Channel 1 if I'm remembering the channel correctly. I think it was Channel 2. Channel 2. Channel 2. It was Channel 2. I think I've actually done an interview with them before, but that's off the topic. A very – A high resolution satellite photograph of the CEMON space center.

And on one side of it, there was something that resembled a very large rocket or missile that was larger than their space launch vehicle.

Other voice 13:23

It's an ICBM. It's an ICBM. We're all going to die.

Aaron 13:27

The Rumsfeld commission was finally going to come true. I think it was 15 meters long. Longer than the Simurgh? My pronunciation is great there, Farsi, by the way.

Jeffrey 13:38

The tower is more than 40 meters, and the object appeared to take up most of the tower. Right.

Aaron 13:47

So for 24 hours, we were whipping around on Twitter, what the hell is this thing, right? I'll let you pick up from there. Pretty quickly, we could figure out what it was, or at least what it wasn't. Despite claims to the contrary by Channel 2.

Jeffrey 14:03

Right, it's that time Israeli TV mistook an elevator for an ICBM. Maybe it's not even an elevator. It might be a trash chute. Yeah, it might be a trash chute. You know what I'm rooting for? What? A base relief of a boosting rocket. Why do you say that? Base relief, you know, like they carved it out, like it's just a decoration.

Yeah. Yeah, so it's not a rocket, right? Right. Well, this is... This is a perfect example of the argument that you have to put things in context. And this is why you model facilities. So we don't just stare at satellite pictures. We extract three-dimensional data from them. And so, for example, we use shadows in order to figure out how tall things are.

And I find, maybe because I'm a visual learner, that the act of actually modeling these facilities... Teaches you way more about them than you would think because it's not just a picture. You actually understand how it operates and what happens when you model the facilities. It turns out the Iranian launch pad is a dead ringer for a setup that is at an Indian Space Launch Center.

Aaron 15:13

Let me stop you there. Yeah. Why? That I can't figure out and I find fascinating. Do you have any background on that or is this just one of the great mysteries of nonproliferation?

Jeffrey 15:22

Yeah, I don't know. You know, you could do it different ways. Yeah. For listeners, the difference is if you're going to launch a rocket, you have to have a flame bucket, right? Because the rocket is propelled by the combustion of the fuels. And so that creates a lot of flames. And so you want a bucket with some trenches so the flames kind of shoot out and don't like toast everything.

Yeah. And there are different ways to do it. But the basic problem you have is you've got a launch tower, and the tower has to be next to where the rocket is. And so if you can remove... The infrastructure away from the flames, then you know, it like gets damaged less. And so the two things you can do, which what we do in the United States, because we have these like really giant movers. Have you ever seen these, Aaron? Have you ever seen these things where the, the Space Shuttle is on like this giant mover? And so they assemble it, they get it already in the hangar, and then they roll it out to the launch pad. Yeah, and they roll it out. I mean, it's almost like rail tracks.

Right, right. So the easier thing to do, which is what the Indians do and what the Iranians do, is it's actually easier to assemble it on top of the flame bucket and to put the building on tracks and roll the building back.

Aaron 16:50

All right. So it's still backwards than we do.

Jeffrey 16:53

Right. I mean I'm always reluctant to call it backwards because their idea may actually make a little bit more sense.

Aaron 17:00

Their idea makes a lot more sense, especially if you under – but what I meant is the process just goes in reverse.

Jeffrey 17:06

Right. So what happens is when you model the Iranian facility and we made a – I guess they're called GIFs, although I like GIF better. We made a GIF showing this. Is it really? I always thought it was GIF. Yeah, I did, too. But apparently there was an interview with the person who came up with the file format and he was like, no, no, it's, you know, like the like the peanut butter.

OK, so you've got if you're looking at it from left to right, you've got a flame bucket and you've got an umbilical tower, which the missile is against. And then surrounding the missile is the assembly building or the gantry. And the gantry pulls away. So from left to right, it's rocket on the flame bucket, umbilical tower, and then pulled away at some distance is the gantry.

The problem with the Iranian missile claim is that the thing you're looking at is on the backside of the gantry. Right? So it can't physically, it would have to traverse through the gantry and the gantry is closed off at the back and somehow maneuver around the umbilical tower in order to get on top of the flame bucket.

So it just, it can't be a, it can't be a rocket. It's just in the wrong spot. Like when you actually understand how launch operations at the site work, it's clear that no matter what that thing is, it can't be a rocket.

Aaron 18:30

Right. Exactly. And one thing that I said to you in In an email that when this, this was all going down, is that actually if you really looked at it closely, I know that this isn't exactly the way you should do it. It looked like you were missing half the circumference of what should be the, the rocket body.

You know what I mean? It looked like it was almost growing out of the, of the structure itself. In addition to the fact that it was on the wrong side of the structure.

Jeffrey 18:57

Yeah, that's right. It was – it would be an incredibly tight fit if that were an opening. And so when you really stare at it, it just does not to me look at all like a rocket. Didn't stop people from freaking out though.

Aaron 19:15

I mean I guess that's the point is that the Korean – the Chinese-Korea brouhaha, like that was – I mean, it was a big deal, but I think it was almost less so than the ICBM. Although, because of where we are in the talks right now, I mean, I don't want to get into motives too much. But it's like, this is a case, in my opinion, where open source can seriously do some damage to something that's going on, whereas in the case we just talked about, maybe it's less so because the matters aren't that pressing.

Jeffrey 19:48

Well, this is one of the things that... I try to impress on reporters. And we can talk about why they don't necessarily do it. But in both the Korean case and in the Iranian case, The South Korean and Israeli press didn't bother to check with a real expert. So the South Korean, the Chosun Ilbo, which is the South Korean paper that published the story, and this is a recurring problem that I see in the South Korean press all the time, is they just pick up things and they reprint them and sort of figure out that, well, if experts come along and point out that it's all a pile of...

Steaming something or other, then they'll just write a second story out of it.

Aaron 20:35

Yeah, exactly. And so they don't check.

Jeffrey 20:38

And this is a case, by the way, in Iran, Israeli TV, there's a guy, Tal Inmar, Right. Who is very, very good.

Other voice 20:48

And he's an Israeli and he is right there. It is a local phone call.

Jeffrey 20:53

And he was one of the people on Twitter who was helping us, you know, because I initially posted something going like, look, we've modeled this site. That's not where the rocket goes. And he was the person who was the first to say, yeah, I think that's right. I think that's the wrong spot for the rocket. So, you know, reporters have to take that step of checking with experts because they are not so good at the geospatial analysis all by themselves.

Aaron 21:21

Yeah, no, no, I agree there. And I think that maybe that is the stopgap or that's the filter. You may not be able to keep it off something like Reddit. We've had our discussions about Reddit recently. But if before it gets picked up and sort of legitimized beyond that, I think requires an extra level of affection.

Jeffrey 21:41

Well, and I do think that's the problem with scoops. Right. Is and occasionally reporters will say this to me and I find I have to push back a little bit. You know, they don't want you to tell anybody about their scoop. And that's a problem because it means that it's I can't go check with my other colleagues because some of my colleagues who are very smart and are very capable initially looked at the picture and said, yeah, I guess that does look like a rocket.

I mean, at first glance, it does. Right. And it's really like you have to you really had to talk to somebody who had gone through the pain of modeling the space center. And so, you know, you know, as much as I say, like reporters have to call someone, like really they have to call two or three people and they have to let those two or three people look at it and talk about it for a little bit.

And I realized that that is contrary to the incentives that they face. But I don't care. Like I just – I'm sorry that you face these adverse incentives. I am here to help you. I will do everything I can to help you. But you can't report things that are false.

Aaron 22:55

Yeah, no, and it's a tough line. I think the context here, just to get into motive a little bit, because that photo was very high quality. I mean was it not?

Jeffrey 23:10

But it's pretty nice, yeah.

Aaron 23:12

It was very nice. There might have been a little intentional misdirection on that part to try and, let's say, upset ongoing negotiations with the Iranians. Would you say that?

Jeffrey 23:25

Well, it's very interesting. It is an Israeli commercial satellite. And I had not previously purchased that data, although you can purchase it. So my question is, did the news channel purchase the satellite image and just sort of do this themselves? Or did somebody see this? And decide to give it to them. Because I can certainly imagine a case where intelligence analysts saw the picture and said, well, that really looks like a missile, but then quickly figured out that it's not because it's in the wrong spot on the launch pad.

But then maybe somebody who was not so nice, who either was real dumb and thought it was a missile, or who just didn't care, dropped a dime and told their colleagues at Channel 2 to go buy it.

Aaron 24:33

I don't know. I don't know either, but the Iranians certainly – I guess I said – well, OK. I'll say it's not a missile, but there's still an outstanding issue that nobody has been able to address, the issue of the NOTAM.

Jeffrey 24:49

Yeah. Well, we're working on that. We've got a – so Melissa Hanham, who is our nuclear threat initiative-funded GURES. Yeah, OK. Yeah. Yeah. I like that. She's, she's our guru for this stuff. She oversaw the modeling of the launch pad and she's got one of our students Alex Kinnard. Actually, I should ask Alex how to say his last name.

Why? It's just Alex. He's a nice guy.

Aaron 25:24

Yeah. We have to screw up pronunciations of everything. It's what we do. Totally.

Jeffrey 25:30

It wouldn't be a podcast unless we massacred someone's name. Alex pulled the NOTAMs and has been looking at them. And it's really interesting. He's going to write something up. So there was a larger – well, there was a large NOTAM for a big period of time. And now there is an even larger area of NOTAM for a shorter period of time.

And so there's a question of whether that's related to a launch that will occur from that pad or from one of the other pads at the site. And I should say, to me, this pad does not look completed. One of the other reasons I think it's probably not a rocket is it doesn't look like they've finished building it.

But there are other pads at the – it's no longer Semnon, by the way. It's the Amman-Hulmeini Space Center.

Aaron 26:22

Yeah, I know. Yeah. Semnon had a nice ring to it.

Jeffrey 26:26

Yeah, I know, right? It does. And so I think that we're still a little bit curious about what that NOTAM is about, but I think it may be launched from a different pad.

Aaron 26:38

I'm really hoping that it's still under construction because – There's speculation that what it is on there is an elevator. I really want it to be a trash chute. That would make me very happy. I like my rocket sculpture. The rocket sculpture is also good. It does look like a rocket. But I think just in broad terms, what we're talking about is the necessity of open source going right.

There needs to be a stopgap before it gets put out there. Either journalists have to check with somebody and there are recommendations about how to do that. I think we all have to be on the lookout.

Jeffrey 27:10

Check somebody and interpret things in context. You're not just looking at a picture.

Aaron 27:15

Yeah, and I think everybody out there has to be like very, I guess, discriminatory when they just look at these things and just take open source as the analyst always gets it right. That's a great place to end. I know. I'm actually just going to ask you. New Arms Control Long Fashioned, what books are you reading this week?

Jeffrey 27:32

Yeah, it's – It's kind of a boring book reading week for me. I'm teaching my class on U.S. nuclear policy in East Asia since 1945. So I am redoing the last couple of case studies. So I have on my desk Victor Cha's book about North Korea, which includes some accounts of the six-party process that he was engaged in.

Chris Hill's book that covers the same period. And the memoirs of Hillary Clinton and Leon Panetta, because I'm trying to find a decent Obama case study. So I'm looking at... Four memoirs of senior officials who were involved in U.S. policy, largely toward North Korea in the past 10 years. And let me tell you something. I think the best of the bunch is Chris Hill's book.

And not even because of the Asia sections. His account of working for Richard Holbrook Is hands down the best and most accurate account of what Richard Holbrook as a person was like. There's a story where Holbrook is going through a hotel to have a meeting with Warren Christopher, who was a time Secretary of State, and Hill is trailing behind him and he hasn't eaten all day.

He goes through a service elevator because he doesn't want to wait for the normal elevator, and in the hallway there is a tray of Canapes which has like plastic over it because it's going to be served and he paws at it like a bear to try to get just to like grab the canapes and shove them in his mouth as he's walking by it's not even not even his food but the plastic stops him and so he leaves these like claw marks and i have to say somebody who has an enormous amount of respect for richard holbrook

Other voice 29:31

That's what he was like.

Jeffrey 29:34

And it was – it's wonderful to see that written down in a way that is funny and amusing but also sort of understands that as off-putting as some of those behaviors might have been, that's why he was the best at his job.

Aaron 29:51

Maybe I'll have to go pick that one up. I – I just ordered, and I'm actually really excited for it. It's called The Libyan Revolution and Its Aftermath. It's an edited volume by Peter Cole and Brian McQuinn with a whole bunch of different authors in there, one of which I relied heavily on for my own monograph I wrote for Roussi.

And I think it's probably the first work on this, and I think it has a lot of lessons for what's going on in Syria and armed interventions in general and sort of their efficacy of those things.

Jeffrey 30:23

And whether or not they serve U.S. policy interests.

Aaron 30:25

So I'm looking forward to starting it. I haven't started it yet, but it's on my iPad as we speak. E-book reader. Those are two good books. Go check them out. I think I'll actually pick up this Christopher Hill book so I can learn more about Richard Holbrook. I don't actually know much about him. And good talking to you this week, Jeffrey.

Jeffrey 30:48

Hey, good talking to you. And listeners, if you don't hate our podcast, leave us a review on iTunes.

Aaron 30:55

Yeah, that would help us. Absolutely. I mean, if you hate the podcast, don't bother. It's a lot of work. You shouldn't be discouraged. But if you like the podcast, totally do. Absolutely. So thanks, Jeffrey, for talking. Thanks, everybody, for listening. Until next week.