Speaker 1: Hello, and welcome to another episode of the Odd Lots Podcast.
Speaker 2: I'm Jill Wisenthal and I'm Tracy Alloway.
Speaker 1: Tracy, you know, what I felt was a really interesting
Speaker 1: thing in a recent debt ceiling episode that we did
Speaker 1: was about all the coding it would take if the
Speaker 1: Treasury were to introduce a new kind of a bill.
Speaker 2: Yes, that was interesting to me too, because I think
Speaker 2: there's a perception out there that if Treasury wakes up
Speaker 2: tomorrow and decides to sell a new type of bond,
Speaker 2: they could just do it. But actually there are all
Speaker 2: these back end changes that would need to be made.
Speaker 2: And as you and I know from multiple conversations at
Speaker 2: this point, it feels like government technology can be a
Speaker 2: little bit clunky sometimes. Is that a fair way of
Speaker 2: putting it?
Speaker 3: It is?
Speaker 1: I would say, you know, in defense, I mean get
Speaker 1: into yes, you know, corporate technology can be clunky. But
Speaker 1: that point reminded me that, like we've done, software is
Speaker 1: like a fascinating thing. And we talk a lot on
Speaker 1: the show. I think about like the sort of built economy,
Speaker 1: like the Chips Act and battery investment and semiconductor and
Speaker 1: all these things are like can the US build things again?
Speaker 1: Like as a government state capacity these ideas, but in
Speaker 1: the twenty first century also like building software in tech
Speaker 1: has to be part of that.
Speaker 2: Question, right, So it's one thing to announce that you're
Speaker 2: going to do this new thing, but almost everything that
Speaker 2: we do nowadays comes with some sort of software requirement.
Speaker 2: Like I'm sure there is a system for processing applications
Speaker 2: for chips, funding and things like that, and someone has
Speaker 2: to actually build that. So there's two questions contained in
Speaker 2: the can America build things question? It's like, can it
Speaker 2: build the actual things and then can it build the
Speaker 2: software systems that would allow it to build the things?
Speaker 1: And you know, in the pandemic period, we got sort
Speaker 1: of smacked in the face, I would say, by this
Speaker 1: realization that the saw fair component is not trivial at all.
Speaker 1: And most notably it was like every state had some
Speaker 1: sort of problem with their unemployment insurance system and there
Speaker 1: were just such a surge in claims obviously in those
Speaker 1: initial you know, really for that first year, just on
Speaker 1: the scale that none had ever seen. And it reminded
Speaker 1: people's like, oh, yeah, fifty different stage systems much with
Speaker 1: like tech that was probably built by some contractor, and
Speaker 1: I don't know the eighties or nineties and the people
Speaker 1: who knew how to run it have left, and you know,
Speaker 1: it was a I gues Eventually, I guess it got
Speaker 1: smoothed out, but it sort of exposed like, Okay, why
Speaker 1: did that happen? And what else is lurking out there
Speaker 1: that we don't really know how it works until it's broken.
Speaker 2: I'm sure this is going to end up being a
Speaker 2: classic episode where we mentioned Kobyl quite a few times,
Speaker 2: like it's inevitable, right, it's coming, you can feel it.
Speaker 1: One day, We're going to do it like actually, just
Speaker 1: like what is Kobol episode where we just like focus
Speaker 1: on that directly. But yes, we're going to talk about
Speaker 1: software and we're going to talk about it from the
Speaker 1: public sector perspective. What does it mean when the government
Speaker 1: tries to fix something, what happens when the govern tries
Speaker 1: to build something, what happens when the government tries to
Speaker 1: buy something. Some of these themes we talked about them
Speaker 1: from a private sector perspective with Patrick McKenzie earlier in
Speaker 1: the year, but more to do on this topic.
Speaker 2: Yeah, I'm excited. I have a lot This is going
Speaker 2: to be a really good chance to actually ask like
Speaker 2: the decisions that are going into some of these things
Speaker 2: being built. I'm still floored every time I log into
Speaker 2: the Treasury Direct website that you know you have to
Speaker 2: click the little the buttons like the actual letters and
Speaker 2: numbers to type your password. It won't let you just
Speaker 2: type your password.
Speaker 4: How did that come to be?
Speaker 2: I have questions?
Speaker 1: That is a great question. So we have I think
Speaker 1: literally the two perfect guests to discuss this, two guests
Speaker 1: with extensive experience answering and working on exactly these problems.
Speaker 1: We're going to be speaking with Jennifer Palka. She is
Speaker 1: the founder of Code for America. She helped found the
Speaker 1: US Digital Service. She co chaired the cal Fornia task
Speaker 1: force on fixing its unemployment insurance system during the pandemic,
Speaker 1: and she's the author of a new book called Recoding America,
Speaker 1: and we're going to be speaking with it. Dave Guarino,
Speaker 1: who's a number of things currently an independent consultant researcher
Speaker 1: focused on this stuff. Founding engineer at Code for America,
Speaker 1: helped create get cal Fresh, which was like allowed Californians
Speaker 1: to easily access snap has recently worked on the unemployment
Speaker 1: insurance modernization at the Department of Labor. Also worked at California,
Speaker 1: also worked in California. So two people's extensive experience on
Speaker 1: exactly these things. So, Jennifer and David, thank you so
Speaker 1: much for coming on odd.
Speaker 4: Lots great, thanks for having us.
Speaker 3: Absolutely great to be here.
Speaker 1: Now I want to ask Tracy, could I steal your
Speaker 1: first question?
Speaker 3: Yeah?
Speaker 5: I do it.
Speaker 4: How has just happened that.
Speaker 1: A government website doesn't let you type in something and
Speaker 1: you have to like click buttons like that like resemble
Speaker 1: mini keyboard on a website. What happened when you see that?
Speaker 1: Do you have like in your mind you're like, oh,
Speaker 1: I know exactly the meeting that caused this to happen.
Speaker 4: I think Dave has been in those meetings and has
Speaker 4: come out like with his hair on fire, fuming, So
Speaker 4: I will let him answer that.
Speaker 3: I mean, well, I don't have Yeah, I wasn't in
Speaker 3: that specific meeting. Maybe the best answer I can give is,
Speaker 3: and there's deeper issues behind it, is that the way
Speaker 3: Garment tends to build technology. The reason for that is
Speaker 3: no one wrote down specifically as a requirement upfront that
Speaker 3: people should be able to use the keypad and the
Speaker 3: numbers on their keyboard to enter the digits, and so
Speaker 3: someone's like, well, I've met the requirements. Like the requirements
Speaker 3: are that someone can do this, they can enter it,
Speaker 3: and the way they do that is they click in
Speaker 3: this somewhat I suppose insane way.
Speaker 1: This is gonna be a conversation or like bang your
Speaker 1: head on a disk a bunch. I feel like as
Speaker 1: we listen these answers, anyway.
Speaker 4: Yeah, I get ready.
Speaker 2: Okay, Well, let me ask maybe a big picture question
Speaker 2: to start, but what is the process by which government
Speaker 2: software is developed? Because you know that answer just then
Speaker 2: it makes it sound like someone gets a mandate we
Speaker 2: need a website, for instance, through which people can buy
Speaker 2: government bonds, and then someone else goes off and I
Speaker 2: guess they execute it along the lines of the mandate
Speaker 2: that's been set.
Speaker 4: I mean, I'll jump in on that one. I mean,
Speaker 4: first of all, it is changing. So just before we
Speaker 4: get everyone sort of bounding their heads on the table,
Speaker 4: there's a lot of movement in good directions. But what
Speaker 4: has been the process is exactly that there's sort of
Speaker 4: this mandated often comes from legislation or regulation that very
Speaker 4: frequently now like says there will be a website that'll
Speaker 4: even put the URL in the legislation, and then it
Speaker 4: kicks off what's a really long process that does exactly
Speaker 4: what Dave was saying. It is first centered around requirements gathering,
Speaker 4: and that process of just requirements gathering can easily take
Speaker 4: ten years. Now, when it's something like healthcare doc of
Speaker 4: you know, and they have a launch date that's three
Speaker 4: years out, they don't take ten years, but they still
Speaker 4: gather sort of every imaginable requirement and they kind of
Speaker 4: throw it in one giant bucket. And that's how the
Speaker 4: RFP is created. The request for proposal, that's what vendors
Speaker 4: bid on, and it's sort of like an undifferentiated, unprioritized
Speaker 4: mess of everything everyone could think of, plus things that
Speaker 4: have sort of pulled over from previous RFPs, plus a
Speaker 4: bunch of compliance stuff around security, compliance stuff around well
Speaker 4: various other things. Dave can maybe chime in, but what
Speaker 4: that isn't is product management. Right, there's no one saying
Speaker 4: here's the important things this should do. And as Dave said,
Speaker 4: there's never a requirement for it to just work. So
Speaker 4: then the vendor, I mean, I'll skip several steps in
Speaker 4: the middle where you know, it takes a long time
Speaker 4: to bid it out, and then there's like a protest
Speaker 4: because one vendor thinks the other vendor shouldn't have got it,
Speaker 4: and that delays it a couple more years. But then
Speaker 4: they go into development and their job, as Dave said,
Speaker 4: is just like check all these boxes, and it's thousands.
Speaker 4: I mean, when we were at California, State of California
Speaker 4: working on unemployment insurance, they were actually about to put
Speaker 4: out a Business System Modernization RFP. Actually I think about
Speaker 4: it to award it to a vendor, and I think
Speaker 4: it had seven six seven hundred requirements. We can check that,
Speaker 4: but it's in that's really a normal number of requirements.
Speaker 4: And so they do all these things that where you
Speaker 4: can check a box, but what they don't do is
Speaker 4: actually check that it works. So I mean, maybe i'll
Speaker 4: just tell a quick story from the book. And you know,
Speaker 4: there was famously this application for veterans' healthcare benefits at
Speaker 4: the VA that didn't work outside the building and they
Speaker 4: didn't couldn't see it. So basically, somewhere in the specs
Speaker 4: it had that this form needed to work on a
Speaker 4: very specific and outdated combination of Internet Explorer and Adobe
Speaker 4: Reader and The reason no one knew that it didn't
Speaker 4: work inside the building is that's how all the computers
Speaker 4: in there were set up. But if you were outside
Speaker 4: the building, had any other possible combination of those two
Speaker 4: pieces of software, it literally wouldn't load. And so they
Speaker 4: had very very few people applying for these benefits online,
Speaker 4: and you know, veterans were really, really, really frustrated. And
Speaker 4: it took a team going out there recording a veteran
Speaker 4: who had tried to do this dozens and dozens of times,
Speaker 4: bringing that video back and showing it to the Deputy
Speaker 4: secretary for them to be able to say, oh, okay,
Speaker 4: actually there is something to be fixed here. Okay, you know,
Speaker 4: you can go ahead and make a new form. But
Speaker 4: up to them, they said, sorry, it's fine, we're looking
Speaker 4: at the requirements. The requirements have been met. There's technically
Speaker 4: nothing wrong. That's amazing.
Speaker 2: It's also kind of crazy to think that people would
Speaker 2: have been looking at that and just been like, oh, well,
Speaker 2: I guess demand for veterans benefits is lower than we
Speaker 2: thought it would be. But actually it was a tech issue.
Speaker 4: I think what they said was demand for them doing
Speaker 4: it online is low. These veterans must.
Speaker 2: Not have access, which is absolutely not true, or they
Speaker 2: can't figure out the computer when in fact, yeah, I
Speaker 2: mean them couldn't figure out the software.
Speaker 4: They would tell them. And they told this guy Dominic,
Speaker 4: the veteran that they interviewed, They kept telling him it's
Speaker 4: user error. There's something wrong with you.
Speaker 1: Can I ask you a question about that bidding process?
Speaker 1: When I hear government goes out to bidders for a website,
Speaker 1: I imagine these like big sort of faceless offices in Roslin, Virginia,
Speaker 1: where you know, people get like chop salid for lunch
Speaker 1: and stuff like that, and there's like three or four
Speaker 1: of them and they sort of like rotate who wins
Speaker 1: the bid and stuff like that. How competitive, like is
Speaker 1: it would a typical rfp auction or bidding process be
Speaker 1: and how hard would it be forcea a more agile,
Speaker 1: innovative firm or someone wanting to do it differently to
Speaker 1: break into the pursuit of that contract.
Speaker 4: That's changing a lot. So now you've got a set
Speaker 4: of vendors. I think there's about thirty five of them
Speaker 4: in this thing called the Digital Services Alliance, which are
Speaker 4: smaller companies that work in a more sort of agile,
Speaker 4: user centered way, and you know, they're just a lot
Speaker 4: smaller than the Beltway, so there's still a tiny portion
Speaker 4: of the market, but you know, they're really viable businesses,
Speaker 4: and people in government can contract out to them. In fact,
Speaker 4: I think there's a mechanism by which you can put
Speaker 4: an RFP out just to those companies. But what has
Speaker 4: happened in the past is like Dave can pile onto this.
Speaker 4: Like if you were bidding out, say you know benefits
Speaker 4: system in a state, and pick your benefit. Usually they
Speaker 4: would require in the RFP that the bidder had to
Speaker 4: have done a similar system in a different state in
Speaker 4: the same category, which means that right there it limited
Speaker 4: to like three vendors, and so you were never going
Speaker 4: to get any disruptors because they were boxed out in
Speaker 4: the very first step.
Speaker 3: Just to maybe jump in. That's exactly right, and if
Speaker 3: you I mean, to some extent, it's rational. If you
Speaker 3: are thinking about this as someone who's not a technologist,
Speaker 3: you run an agency, and you're thinking, we need to
Speaker 3: do this very large project. It's going to take a
Speaker 3: very long time. It we really need it to go right.
Speaker 3: It is hard to think about, well, why wouldn't I
Speaker 3: look for a vendor who's done this five, ten, fifteen times.
Speaker 3: The problem is, as Jen said, well, I think there's
Speaker 3: two problems in that. One is you've now very very
Speaker 3: drastically narrowed your vendor pool, so there's a lot less competition,
Speaker 3: and two you also may have too big of a project,
Speaker 3: Like really, what level of attraction are you trying to
Speaker 3: work towards? And is it the case that really you
Speaker 3: need someone or a vendor who's worked on this specific
Speaker 3: type of system in another entity that it works exactly
Speaker 3: the same way and all of those details of the
Speaker 3: exact same program, the exact same structure of agencies. Probably not,
Speaker 3: because probably it's about more generically software. But if you
Speaker 3: really focus only on the track record there, you kind
Speaker 3: of get stuck, as Jema was saying, in this trap
Speaker 3: where there's only a small number of vendors who can
Speaker 3: meet those criteria, even if you think they're And it's
Speaker 3: hard because the other thing I would say is as
Speaker 3: much as there are new vendors, and maybe Jenny can
Speaker 3: speak more of this, but it's also the case that
Speaker 3: if you have a new vendor that wants to work
Speaker 3: in a more agile way, wants to work in a
Speaker 3: more use this CRD way, but they are bidding on
Speaker 3: an RFP that is structured in a very waterfall top
Speaker 3: down meet other requirements, and that's what success looks like.
Speaker 3: It's also not going to be successful because you're not
Speaker 3: giving them any space to say, oh, well, I know
Speaker 3: your requirement says a way to ender phone numbers using
Speaker 3: the key the mouse, but when we tested it with
Speaker 3: people like people, turns out really don't like using the
Speaker 3: mouse to click numbers to enter digits. They prefer to
Speaker 3: use the keyboard. Could we change that requirement? And honestly,
Speaker 3: I mean, that's an extreme example, but a lot of
Speaker 3: and there's a lot of what Jen's book is about
Speaker 3: and what resonated with me in it is that in
Speaker 3: doing these things and building software, you learn those details
Speaker 3: that you never could have gotten right up front. And
Speaker 3: so if you make success defined by implementing everything as
Speaker 3: we understand it before we ever get started, you almost
Speaker 3: ensure that you're not going to get what you really
Speaker 3: wanted because you're not allowing any information after that cut
Speaker 3: off point, right when you might be getting to the
Speaker 3: point where in doing it you're gonna learn a lot.
Speaker 2: I definitely want to ask you more about that top
Speaker 2: down waterfall nature of how a lot of these software
Speaker 2: projects get commissioned. But before I do, I'm just curious
Speaker 2: about the actual vendors because I'm sort of imagining these
Speaker 2: like big faceless offices. But what types of vendors are
Speaker 2: we talking about? And I'm trying to think how to
Speaker 2: use those kind of diplomatically. But I can imagine a
Speaker 2: company that is putting itself out bidding for government contracts
Speaker 2: on software development, Like maybe they're not as experimental or
Speaker 2: cutting edge as a software company that's, you know, somewhere
Speaker 2: in San Francisco and it's building like new exciting products
Speaker 2: for the corporate world. Although we've also done an episode
Speaker 2: on how bad internal corporate tech can be, but it
Speaker 2: seems like it might be a little bit more I
Speaker 2: hesitate to say boring, but sort of basic playing it safe.
Speaker 4: Well, you know one thing that I will say, like
Speaker 4: I hear both incredible frustration and even anger from folks
Speaker 4: that have to hire these companies. They're just like, there's
Speaker 4: no alternatives. This has gone badly, it will go badly.
Speaker 4: They kind of know that, Like I had this woman
Speaker 4: in a major state. I won't say which who. When
Speaker 4: I was said, you know your project's going to fail,
Speaker 4: she said, do you think we don't know that? The
Speaker 4: last seven projects have failed. So there's this a huge frustration.
Speaker 4: But there's also this real feeling that the company in
Speaker 4: San Francisco with its forty people who make consumer software
Speaker 4: like great, people love using that software, but they don't
Speaker 4: understand us, right, They're not going to actually understand the
Speaker 4: constraints of government. So we have to go with these
Speaker 4: big companies. And they're not wrong in the sense that
Speaker 4: the constraints of working with government are real. There's a
Speaker 4: huge compliance burden that forty person company in Sum Francisco
Speaker 4: mostly does not want these jobs, Like there's so much
Speaker 4: that goes into getting the you know, getting the bid,
Speaker 4: you know these it's sort of famously said that these
Speaker 4: companies know how to get the get the work, not
Speaker 4: how to deliver on the work. That's not probably what
Speaker 4: a fordy person company wants to do, but you know,
Speaker 4: it's true that you have to deliver on stuff that
Speaker 4: is mind bogglingly complex. Like when we're working in unemployment
Speaker 4: insurance again during the pandemic, my colleague was talking with
Speaker 4: the claims processors like week over week, and we're trying
Speaker 4: to like dissect it and figure out what's going wrong
Speaker 4: and like clear this backlog. And one of these guys
Speaker 4: keep saying, well, I'm not quite sure about that answer.
Speaker 4: I'm the new guy. I'm the new guy. And she
Speaker 4: finally says, how long have you been here? And he says,
Speaker 4: I've been here seventeen years. The guys who really know
Speaker 4: how this works have been here twenty five years or more.
Speaker 4: So think about, like, you know, going from doing some simple, cool,
Speaker 4: you know tech app, you know, easy consumer app, to
Speaker 4: trying to build or fix or improve upon a system
Speaker 4: that is so complex that it takes twenty five years
Speaker 4: to learn how to process a claim. That's that's sort
Speaker 4: of I think what needs to be on the table
Speaker 4: as part of this agenda is not just like can
Speaker 4: the tech be better? But can we go back and
Speaker 4: simplify the accumulated like ninety years of policy and process
Speaker 4: that's making that so hard to make?
Speaker 3: Well?
Speaker 1: Why do we sort of back up?
Speaker 3: Then?
Speaker 1: And I again kind of steal Tracy's question because I
Speaker 1: had the same thought, what is it about government buying
Speaker 1: that creates these massive waterfold RFPs in which you can't
Speaker 1: you know, in which a like create these constraints that
Speaker 1: the software developer would not necessarily have with a private buyer,
Speaker 1: and b puts government agencies in a position where they
Speaker 1: will say to you, of course, we know it's going
Speaker 1: to fail, which again presumably that actually probably does happen
Speaker 1: in the private sector too. Or but what is what
Speaker 1: are the to some extent, but what are the dynamics
Speaker 1: that make these things like so hard to change and
Speaker 1: so hard to you know, rip, you know, come up
Speaker 1: with a new system.
Speaker 4: Well, this is a good Dave question, but I'm gonna start.
Speaker 4: You know, I really spend a lot of time reflect
Speaker 4: on this, you know, I had. I've had sort of
Speaker 4: three years since I stepped down from Code for America.
Speaker 4: I just like sat in a room and well, I
Speaker 4: interviewed people and thought about it, and I thought about
Speaker 4: my own experiences, and I think that there's a deep
Speaker 4: seated culture in government where the policy people are the
Speaker 4: important people. They do the important stuff, and technology digital
Speaker 4: is just part of implementation, which is at the not
Speaker 4: just the bottom of like a software development waterfall, it's
Speaker 4: the bottom of a big, rigid hierarchy and which information
Speaker 4: and power and insights only flows from the top to
Speaker 4: the bottom. And so it's problematic in part because the
Speaker 4: people who are doing the tech are really just sort
Speaker 4: of downstream of everything else and the power and ability
Speaker 4: and willingness to step up and say, hey, like we
Speaker 4: probably shouldn't do those sixy seven hundred requirements, we should
Speaker 4: probably focus on these two hundred. Get that out the door,
Speaker 4: and then you know, adage cases as later like that's
Speaker 4: it's just not there's no permission really to say that,
Speaker 4: and until we you compare that with say, we'll call
Speaker 4: it metaphysical silicon valleys. That I don't mean actually like the
Speaker 4: silicon valley, Like the people I write code started the companies,
Speaker 4: they're in power, they're at the top, Like compliance like
Speaker 4: is below them. In sort of the DC government hierarchy,
Speaker 4: compliance is like way above, right, Like policy is way
Speaker 4: above and there's not this sort of build measure learn
Speaker 4: cycle the Dave was referring to, where you know, people
Speaker 4: are learning from each other as they're doing the work
Speaker 4: and that technology. You know, certainly the policy has an
Speaker 4: impact on the technology, but the building of the technology
Speaker 4: needs to have an impact on the policy, like that
Speaker 4: fundamental culture is something that needs to be like looked
Speaker 4: at and called out, and where it's not happening, where
Speaker 4: there's actual conversation, you know, dialogue beats right. Where that's happening,
Speaker 4: we should be lifting that up and saying this is
Speaker 4: possible within government because that's like the foundation I think
Speaker 4: of getting away from these mega projects that fail. But
Speaker 4: Dave you you are you're going to disagree with me
Speaker 4: on this, and I'm gonna love it.
Speaker 3: I think that's I actually do think that's right. I
Speaker 3: think a big part of it flows downstream from budgeting,
Speaker 3: meaning a lot of this the way that and again
Speaker 3: everything that I think Eyegen sate like, there's some degree
Speaker 3: of generalization here, and there's always outliers and there's always exceptions,
Speaker 3: and of course there's also a lot of people try
Speaker 3: and change these things. That said, maybe the dominant mode
Speaker 3: is we're going to budget for a large project. We
Speaker 3: get one time funding to spend to do that project,
Speaker 3: so we got to get everything right. We're only get
Speaker 3: this chance once every so often. We're going to replace everything.
Speaker 3: We're gonna have this new system. It's going to be great.
Speaker 3: And the way that this paradigm breaks down is to
Speaker 3: in what's called a design development phase, where you build
Speaker 3: the system, you design what it should do, you think
Speaker 3: about that. Again, you're kind of making all these decisions
Speaker 3: up front, which I think is very counter to a
Speaker 3: lot of the if you want to call the Silicon
Speaker 3: Valley approach. But then you're building the system, and then
Speaker 3: you enter what's called maintenance and operation, and it's supposed
Speaker 3: to be very very little. You're not going to change much.
Speaker 3: It's just kind of keeping the system going. And the
Speaker 3: problem is modern software you kind of learn the most
Speaker 3: the second you've actually deployed to real users. That is
Speaker 3: the point when you are getting people saying, oh, this
Speaker 3: doesn't work, or you're getting an error, or you're getting hey,
Speaker 3: I tried to put in this address, but I have
Speaker 3: a funky address and it won't let me do that. So,
Speaker 3: all of a sudden, you have all this information, and unfortunately,
Speaker 3: in sort of what gets called a big bang launch
Speaker 3: project where you just put it up and you're done
Speaker 3: and you're not going to touch it again, you now,
Speaker 3: almost immediately on day one, can see potentially all of
Speaker 3: these issues. And a lot of people prepare for this
Speaker 3: in these kinds of projects. They're like, the first month
Speaker 3: is going to be really hard. You all of a
Speaker 3: sudden know all the things that were assumptions you made
Speaker 3: that you got wrong. But the model that you had
Speaker 3: that you were given is, well, we're going to do
Speaker 3: this one time. Now. Contrast that with actually going live
Speaker 3: is the first step, and you're going to have funding
Speaker 3: that's continuous and kind of flat over ten, over twenty,
Speaker 3: over thirty years, and you're going to have those people
Speaker 3: on STEPH and you're going to continuously change the system
Speaker 3: as you learn more about how users interact with with it,
Speaker 3: as you learn more about like what things you didn't
Speaker 3: get right upfront? Did you You made it possible to
Speaker 3: upload a certain type of document, You accept PDFs and
Speaker 3: tiff images Tiff this is a real example, but turns
Speaker 3: out you don't support JPEGs or the I think iPhone
Speaker 3: proprietary default image format. So now all your mobile users
Speaker 3: are saying, hey, we can't upload, or a bunch of
Speaker 3: them were saying, I can't upload documents. It says I
Speaker 3: can't submit this file type right now. A lot of
Speaker 3: you know, well intentioned folks in government get to that
Speaker 3: situation and then they're stuck because they don't have the
Speaker 3: mandate or the funding the same now we're going to
Speaker 3: change that. They're waiting for the next big project to
Speaker 3: replace the system and now support those new things, Whereas instead,
Speaker 3: if you have a model where you're saying we expect
Speaker 3: to continuously change this, it's a very very different thing.
Speaker 3: And that's a little bit that is more how Silicon
Speaker 3: Valley operates. I mean, this maybe is sort of hyperbole,
Speaker 3: but Google didn't build search and then lay off ninety
Speaker 3: percent of their staff and say we're done. This is great,
Speaker 3: you know, like, look we've got this search thing. And
Speaker 3: I think that difference flowing downstream from how tech is
Speaker 3: budgeted for where you're budgeting for a one time project,
Speaker 3: and this comes from legislatures, you know, from Congress. A
Speaker 3: one time project versus we want to have ongoing fundings
Speaker 3: because this is a living, breathing system. Is a huge
Speaker 3: paradigm shift and has an uncomfortable aspect, which is it
Speaker 3: may seem like it's costing more money because it's potentially
Speaker 3: dollars ongoing because it's staff, not one time by but
Speaker 3: I think a lot of the dysfunction that we see
Speaker 3: probably flows down from that root. Cause.
Speaker 2: Okay, so here's here's a big question. If we can
Speaker 2: identify the cause of a lot of this dysfunction and
Speaker 2: if we can trace it back to this waterfall structure,
Speaker 2: the sort of top down mandates or maybe like theorists
Speaker 2: versus practitioners. So you have a politician who has a
Speaker 2: big idea and they want that big bang reveal of
Speaker 2: the software they're going to expand benefits, and it's going
Speaker 2: to come with a really cool website and everyone's going
Speaker 2: to be able to access them easily. And yet it
Speaker 2: leads to these problems that we've been discussing.
Speaker 4: Why do we keep doing it?
Speaker 2: That's a hard question, Well, what is the system that
Speaker 2: is like keeping this way of doing things in place,
Speaker 2: versus maybe you know, after a few decades of developing
Speaker 2: these types of systems, why isn't someone going well, actually,
Speaker 2: the way we've been doing things hasn't really been working well.
Speaker 4: I think a lot of people have been saying that,
Speaker 4: and that's why I say things are changing, like and
Speaker 4: I will offer as evidence COVID tests dot gov. Do
Speaker 4: you remember that one beginning of twenty twenty one, the
Speaker 4: President announced that there would be this site where you
Speaker 4: could request tests, and I think he gave a date
Speaker 4: that was I want to say six weeks out, it
Speaker 4: launched a day early, it launched in multiple languages. It
Speaker 4: took eleven seconds for me to use other people say
Speaker 4: eight seconds, fifteen seconds, right, and you're test showed up
Speaker 4: a couple days later. Like that sounds like it could
Speaker 4: have been really like okay, so it's much simpler than
Speaker 4: say signing up for healthcare ducov or whatever. But like
Speaker 4: they could have made it way more complex. They could
Speaker 4: have said, like, let's gather every possible requirement and try
Speaker 4: to fulfill every possible requirement. But you had internal capacity,
Speaker 4: Like to Dave's point, there are people in house who
Speaker 4: were saying, hey, we have been doing it away that
Speaker 4: hasn't been working. Let's try this way and it's great.
Speaker 4: Like it serves as a fantastic example I think within
Speaker 4: government and outside government. Right, So part of it is
Speaker 4: how people in government think what they believe, and part
Speaker 4: of it is what the public thinks. Right. The public
Speaker 4: can't couldn't imagine something that easy, but we have to
Speaker 4: start imagining it and holding our government accountable to doing that. So,
Speaker 4: I mean, I will say it is changing, but the
Speaker 4: belief that we aren't good at it, that we couldn't
Speaker 4: ever do like a COVID test. Dot of drives politicians, pundits,
Speaker 4: vendors to say government's bad at this, and so we
Speaker 4: should outsource everything. Now, of course we're going to have vendors,
Speaker 4: and we should have vendors, but there has to be
Speaker 4: enough internal capacity and competency to outsource well and to
Speaker 4: make those decisions like, hey, let's have this thing, only
Speaker 4: ask like name, address and like, you know, not ask
Speaker 4: for their health insurance, you know, numbers and not like
Speaker 4: do different numbers of tests per household. Like that's an
Speaker 4: internal capacity question that is really important. And we don't
Speaker 4: have enough internal capacity for these kinds of decisions in
Speaker 4: government because we believe we're not good at it, and
Speaker 4: we believe the only people who are good at out
Speaker 4: of these outside companies. Plenty of evidence that that's totally wrong.
Speaker 1: I want to go back to internal capacity in a minute,
Speaker 1: but since you mentioned the COVID test website, it's sort
Speaker 1: of a good chance to pivot a little bit here.
Speaker 1: You know, COVID specifically seem to open people's minds a
Speaker 1: little bit about how fast we can work and how
Speaker 1: much money we can spend, and obviously a historically fast
Speaker 1: pace of vaccine development that we've never seen prior to that,
Speaker 1: and so it sort of had this brief moment where
Speaker 1: it sort of opened people's minds. And I'm not sure
Speaker 1: if it's closing again, but you know, since both of
Speaker 1: you worked on the California unemployment insurance system and generally,
Speaker 1: like I think almost every state on some level was
Speaker 1: seen by some tobaccle Like what was I guess the
Speaker 1: biggest eye opening thing that going into that environment taught
Speaker 1: you or about the system. And then what was like,
Speaker 1: you know, what's the key sort of takeaway of like, Okay,
Speaker 1: here's a thing that we can learn from this that
Speaker 1: then whether it's UI and other states are more broadly, okay,
Speaker 1: this is something that you learned that then can like
Speaker 1: have extensible lessons.
Speaker 4: So I think you're right. COVID had sort of twin
Speaker 4: impacts on our thinking. One was, holy cow, we can
Speaker 4: actually do stuff, and the other one was, holy cow,
Speaker 4: we're really screwed. Right, Yeah, you're way to put it,
Speaker 4: And there's evidence of both, which is which is really fair.
Speaker 4: I mean I think for me, you know, i'd been
Speaker 4: doing this for you know, ten years when I got
Speaker 4: pulled into the the unemployment insurance I'll call it a rescue.
Speaker 4: We were clearing the backlot, and I felt like, nothing
Speaker 4: here can scare me. I have seen the VA, I've
Speaker 4: seen the operating of you know, some pretty janky, you know,
Speaker 4: government agencies. But I actually really did learn a lot
Speaker 4: and was kind of shocked at how the unemployment insurance
Speaker 4: system worked. And the metaphor that came to mind for
Speaker 4: me was like everyone kept saying, well, it's a system,
Speaker 4: obviously you can just fix it, and it isn't a system.
Speaker 4: And I think that's true of other things we think
Speaker 4: of the systems too, Like it is archaeological layers of
Speaker 4: technology that sort of map to archaeological layers of policy
Speaker 4: that didn't ever get designed. It just accrued, Like so
Speaker 4: nobody ever goes back and says, okay, how is this
Speaker 4: going to work with this? There's such little appetite for
Speaker 4: anything that's like backward looking that would actually update and
Speaker 4: make these layers work together well, that it just isn't
Speaker 4: what people think it is. When you go look at it,
Speaker 4: you're just like, WHOA, of course you can't do these things,
Speaker 4: like this thing isn't connected in any meaningful way to
Speaker 4: this thing, and like this was built in a certain
Speaker 4: era for a certain purpose, and we just glombed stuff
Speaker 4: onto it when we needed to say, add internet access,
Speaker 4: like let people apply online. I mean, the biggest thing was,
Speaker 4: like I think we haven't grappled with the fact that
Speaker 4: when these systems were quote unquote designed, I actually was
Speaker 4: to say when they were started, because they really have
Speaker 4: never been designed, and a lot of them and there's
Speaker 4: huge advantage to actually doing some design. But you went
Speaker 4: into an office and showed who you were, like you
Speaker 4: identified you, you validated your identity by going in, and
Speaker 4: then we moved online and we never figured out how
Speaker 4: to validate people's identity. And there were a number of
Speaker 4: problems in California with unemployment assurance. The fact that you
Speaker 4: know it took twenty five years to learn how to
Speaker 4: do it meant that the any claim that couldn't be
Speaker 4: processed automatically was a huge, huge bottleneck. And you couldn't
Speaker 4: add claims process to do that, Like you would have
Speaker 4: to go back in time twenty five years, start new
Speaker 4: claims browsers and have them ready to come online during
Speaker 4: a downturn, right, Like that's never going to scale. But
Speaker 4: the other big problem was that that you only went
Speaker 4: through the automatic sort of pipeline if they felt like
Speaker 4: they could verify your identity, but they weren't really doing
Speaker 4: any meaningful identity verification. And we, unlike most other industrialized countries,
Speaker 4: don't have a national system for that, Like we don't
Speaker 4: have a national technology platform for that. But also like
Speaker 4: again back to this all derives from like the legal
Speaker 4: and policy framework, but actually have a reasonable policy framework
Speaker 4: for identifying people and knowing who they are when we
Speaker 4: start to do a transaction. And that is causing problems everywhere,
Speaker 4: and there is interesting conversations about how to fix it.
Speaker 4: There's some stuff out there, but it's it's really contested,
Speaker 4: and it's not just a problem for the pandemic. It's
Speaker 4: going to continue to be a problem.
Speaker 3: Yeah, I mean, I'll just add and my vantage point
Speaker 3: was was somewhat different, right, I was coming at it
Speaker 3: from a different angle. I think the major thing that
Speaker 3: I took away, especially with so much focus on technology
Speaker 3: and so much focus on COBAL mainframes, and that was,
Speaker 3: you know.
Speaker 2: That was set we need to do a Cobyl like
Speaker 2: drinking take a shot time.
Speaker 4: How long it took us to get to that.
Speaker 3: Word oh, I can't believe, you know. And the best
Speaker 3: part is that. I mean my next point was that's
Speaker 3: misplaced causation. And yet I am the one who brought
Speaker 3: up cobal. Yeah, I mean I think I think we
Speaker 3: that is a you know, for so much focus on Oh,
Speaker 3: these tech systems are falling over they're not good. I
Speaker 3: think there's the number one lesson I learned was that
Speaker 3: and I think this is true, but we don't tend
Speaker 3: to think about it, which is the technology systems are
Speaker 3: part of a larger sociotechnical system. What I mean by
Speaker 3: that is, as Jen mentioned, you have staff who can
Speaker 3: do stuff manually, and you have a system that might
Speaker 3: be able to do stuff in an automated way. One
Speaker 3: of the problems with programs that are so complex is
Speaker 3: if you have a massive spike of volume in your
Speaker 3: technology system isn't set up to just cover all those
Speaker 3: cases in an automated way. You try to throw bodies
Speaker 3: at it, and you try to hire people. But if
Speaker 3: it takes a year and a half to train someone
Speaker 3: and like train someone to a relatively basic level, with
Speaker 3: some of the complexity of these programs, you're not able
Speaker 3: to hire fast enough and Unfortunately, you know, this is why.
Speaker 4: We were hiring really fast. We heard five thousand people
Speaker 4: in California to help process these claims, but because they
Speaker 4: couldn't do anything, they were taking up the time of
Speaker 4: the experienced claims processors. And we calculated that every single
Speaker 4: new hire the state made slowed processing down. And one
Speaker 4: of the big things we did was actually just get
Speaker 4: them to reassign staff away. I mean, some of it
Speaker 4: was just reassigned staff to other things, Like nobody was
Speaker 4: opening the mail, which was pretty critical. And yet opening
Speaker 4: the mail is something you can do if you just
Speaker 4: joined and have no background, but like, yeah, it was
Speaker 4: pretty franny. So sorry to interruptive. It's just like we
Speaker 4: were hiring. It's just that the hiring was having a
Speaker 4: perverse effect.
Speaker 3: Well, and that's I guess that's my point is that's
Speaker 3: exactly right. And I guess the number one lesson I
Speaker 3: learned was if you look at the history of a program,
Speaker 3: like un employment trends, but specifically UI, you have this
Speaker 3: thing where once every ten years or so, there's a
Speaker 3: major recession, or in this case, there's a pandemic, which
Speaker 3: was like a ten x recession, right because it was
Speaker 3: overnight as opposed to graduate was overnight. So much of
Speaker 3: the economy, so many of the working force, labor force,
Speaker 3: were just out of work immediately, so all those claims
Speaker 3: came in simultaneously. But we have this cycle of every
Speaker 3: ten years or so way of recession or I mean,
Speaker 3: I don't know, I'm non economists, I can tell you
Speaker 3: exactly what that is, but it happens in a recurring way,
Speaker 3: and then everyone gets frustrated with how difficult it is
Speaker 3: to get unemployment insurance because they're overloaded with volume. And
Speaker 3: then the claims volume goes down, recession ends, and then
Speaker 3: there's a lot there. Honestly, isn't a lot of focus
Speaker 3: on Okay, we're gonna sort of put more money into
Speaker 3: the UI system so that we're ready for next time.
Speaker 3: But unfortunately, you can't have resilient systems if you optimize
Speaker 3: for pure, pure, pure efficiency, like in the down years,
Speaker 3: in the years when you have very very low volume,
Speaker 3: if you are just going to cut staff down to
Speaker 3: very very bare bone skeleton crew. It comes back to
Speaker 3: this point of not only are you not able to
Speaker 3: do preparation if you don't have the funding to have staff,
Speaker 3: you can't even make the changes that would be good,
Speaker 3: and you can't when you get the money in a
Speaker 3: new recession hire the people because you can't train them,
Speaker 3: and so we kind of get stuck in the cycle.
Speaker 3: So that was the number one from a systems modeling
Speaker 3: perspective lesson I took away with it.
Speaker 4: From it, I got to add, though, you know, we
Speaker 4: did invest several billion across states during the Great recession
Speaker 4: or after a Great recession, so I think about half
Speaker 4: of the states technically modernized then they did take advantage
Speaker 4: of pretty significant funds. None of those states did any
Speaker 4: better than average in this downturn. So I agree with Dave,
Speaker 4: and I would argue that taking a different approach to
Speaker 4: quote unquote modernization, to policy and process simplification, to you know,
Speaker 4: having goal driven modernization like all of those things, is
Speaker 4: equally important as investing during the downturn. Investing the way
Speaker 4: we have been doing it is not working. There are
Speaker 4: so many different threads to pick out of there. But
Speaker 4: just on that last point, you know, when you were
Speaker 4: talking about how a lot of these government systems were
Speaker 4: not designed to be that way, I kept thinking of
Speaker 4: a lot of the software used by the banks, where
Speaker 4: you know, you think, you think of like JP Morgan's
Speaker 4: software system. It wasn't designed to be JP morgan software system.
Speaker 4: It's the result of many, many acquisitions of other banks,
Speaker 4: and you know, every time they take over this business,
Speaker 4: it gets sort of duct taped onto the rest of
Speaker 4: the system. But I'm curious, is there a moment that
Speaker 4: you have seen in your careers at which someone says,
Speaker 4: this system is just like irredeemable. We're going to have
Speaker 4: to start from scratch, And like, what is the trigger
Speaker 4: point for making that decision versus reaching for the band
Speaker 4: aids and the duct tape and just trying to make
Speaker 4: it work.
Speaker 3: I mean, and I forget if someone else has mentioned this.
Speaker 3: I lit the Patrick McKenzie episode, maybe he mentioned this,
Speaker 3: But in general, a good rule of thumb is never
Speaker 3: rewrite a system from scratch that's working. I mean, other
Speaker 3: people might disagree with that. But the reason I don't
Speaker 3: have a good answer for sort of a moment. Lots
Speaker 3: of people have tried to do that thing where they
Speaker 3: just rebuild it from scratch, and again it tends to
Speaker 3: go poorly because and this is the specific thing, the
Speaker 3: old system encodes so much tacit knowledge and so many
Speaker 3: of the many, many many little details, and some of
Speaker 3: those details might no longer be relevant, but many of
Speaker 3: them might encode some really hard learned truth about the world,
Speaker 3: because at the end of the day, the software is
Speaker 3: just modeling the real world. It's just saying, okay, well
Speaker 3: we have customers. They have addresses. Okay, so maybe you
Speaker 3: have two address fields in your old system and you think, oh, well,
Speaker 3: let's just streamline that down to one. But then you
Speaker 3: launch your new system has one address, and everybody calls
Speaker 3: you up and like, what are you doing? Like, we
Speaker 3: the reason we have two addresses is because our billing
Speaker 3: department is over here and the customer facing address is different.
Speaker 3: So I think the throw it away and start from
Speaker 3: scratch is an impulse everybody has, but it has a
Speaker 3: lot of risk as well, and I think the better
Speaker 3: approach tends to be can you get to a place
Speaker 3: where you can start to incrementally change your systems as
Speaker 3: you go in small ways that are safe, where it
Speaker 3: doesn't take it. You know, it's not like we have
Speaker 3: to test for six weeks any change before making it live.
Speaker 3: We can ship weekly, we can ship daily, we can
Speaker 3: ship multiple times a day. That tends to yield better things.
Speaker 3: There are plenty of projects out there that we're going
Speaker 3: to build this from scratch, from the bottom up. Most
Speaker 3: of the ones I'm aware of don't have great launch stories.
Speaker 3: Maybe JO have disagrees, that's my no.
Speaker 4: They're still in that mode of like get it all
Speaker 4: in once and then we're done, and we have sort
Speaker 4: of maintenance, and so they're pretty bad. But I will
Speaker 4: say two things. One on a practical level, just because
Speaker 4: I was talking with a colleague on the way over
Speaker 4: here about this, there are great examples now of what
Speaker 4: Dave's talking about, like not waiting for this big bang
Speaker 4: project but saying we are just going to be making
Speaker 4: this a little bit better every day. And one of
Speaker 4: them is New Jersey's Department of Labor. Their unemployment insurance
Speaker 4: response during the pandemic was quite good, but they've also
Speaker 4: learned from that and said we're not going to wait
Speaker 4: for some like big system to come that's going to
Speaker 4: fix all of our problems. We have a team now
Speaker 4: that can just incrementally fix things as we go and
Speaker 4: eventually that will involve some bigger investments as they learn X, Y,
Speaker 4: and Z. But they're just not waiting and it's just
Speaker 4: a great, great example, But I will say, Tracy, like
Speaker 4: I think about what you're asking about all the time.
Speaker 4: I don't think this will ever happen, but if it,
Speaker 4: if you do get a chance to sort of zero
Speaker 4: base it and start over, you have to zero base
Speaker 4: the policy, Like you can't. Could you do a new
Speaker 4: unemployment insurance system in California that encodes the twenty five
Speaker 4: years of knowledge that those claims processors have. Well, I
Speaker 4: mean maybe I will help you now. I don't want
Speaker 4: to talk about any but like, but like that's a problem.
Speaker 4: Like you know, I say in the book, there's this
Speaker 4: great team that's working on this Medicare problem, and it's
Speaker 4: like there's nine different definitions of even like what a
Speaker 4: medical group is. Like even the first question doctors are
Speaker 4: supposed to answer is like wildly complex. And the policy
Speaker 4: the delivery team is sort of fighting with the policy team,
Speaker 4: and she's like, I get that it's complex. It has
Speaker 4: to make sense to a person, And I like, fight, fight, fight,
Speaker 4: not over the tech, but out of like over collapsing
Speaker 4: these nine definitions into two. They want I get to one.
Speaker 4: They can't. At least they get to two. And like
Speaker 4: this idea that it has to make sense to a person.
Speaker 4: We've like lost track of that. I mean, of course
Speaker 4: you're frustrated when you apply for unemployment insurance, Like it's
Speaker 4: wildly complex in its policy and the process that accrued
Speaker 4: from those policies. Like if you were going to start over,
Speaker 4: you can't start over with the tech. You would have
Speaker 4: to say, okay, let's figure out let's like look at
Speaker 4: this ninety years of accumulated policy and say what was
Speaker 4: this actually trying to do. Can we kind of you know,
Speaker 4: easier go back and just like rationalize all the changes, right,
Speaker 4: It's not like like I say, like there's not one
Speaker 4: binder of regulations that covers youI there's ninety years of
Speaker 4: memos that change the previous memo that change the previous thing,
Speaker 4: that changed the previous thing. And you could at least
Speaker 4: go back and like condense that and like figure out
Speaker 4: all of the conflicts in it. But it would make actually,
Speaker 4: I think a lot more sense. And I will be clear,
Speaker 4: this is never going to happen if you just said,
Speaker 4: let's design the policy for an unemployment insurance system that
Speaker 4: makes sense in twenty twenty three, and then you could
Speaker 4: build tech for that.
Speaker 1: There's so many different threads and places we could go.
Speaker 1: There's such a fascinating conversation. I just have one more question,
Speaker 1: and it sort of relates to something, Jennifer that you
Speaker 1: said earlier about like in government, the tech people are
Speaker 1: like secondary likely like the policy wonks and people designing
Speaker 1: these things, and that that's a really big difference between
Speaker 1: government of public and private sector. And you know, this
Speaker 1: internal capacity, how much does that hinder And I think
Speaker 1: this is another thing that Patrick McKenzie brought up, but
Speaker 1: like hiring and so and the challenge of like, you know,
Speaker 1: public sector pay skills are not private sector pay skills.
Speaker 1: I have to imagine for a lot of these things
Speaker 1: he talked about a little bit, you know, and you
Speaker 1: know what you've worked on some of these like sort
Speaker 1: of volunteer things that sort of bridge the private and
Speaker 1: public sector. But can you talk a little bit about
Speaker 1: that can strength of like even if like just the
Speaker 1: challenge of in those in those roles getting like experienced
Speaker 1: talented people.
Speaker 4: I'm glad you brought that up because it's a little
Speaker 4: bit of a soapbox I like to get on because
Speaker 4: we have you know, like politicians, like the leaders who
Speaker 4: are so frustrated with delivery and technology, and like they're
Speaker 4: generally they're well intentioned, but the sort of things they
Speaker 4: push on are kind of unhelpful. You know, they kind
Speaker 4: of increase the risk aversion of the bureaucracy by yelling
Speaker 4: them about things and not recognizing the constraints on the bureaucracy.
Speaker 4: I call this the accountability trap. But there are things
Speaker 4: they actually could do, and fixing the civil service and
Speaker 4: civil service hiring rules is really really important. I do
Speaker 4: know what Patrick McKenzie said on your show about pay
Speaker 4: scales is not totally I don't totally agree with that.
Speaker 4: The biggest problem is not the pay, it's the time
Speaker 4: to hire. So if it takes, as is really common
Speaker 4: right now in federal government, nine months to make an offer,
Speaker 4: of course you lose everybody you've got. Actually, you know,
Speaker 4: when when I started this work, there was not like
Speaker 4: a line of people out the door, one with great
Speaker 4: tech and design skills wanting to work for government. Now
Speaker 4: there actually is, and we can't get them in the door.
Speaker 4: I mean, I know people who do wait nine months
Speaker 4: for that offer because they really want to help and
Speaker 4: the pay is fine, Like it's not fantastic. I think
Speaker 4: he was comparing to like Google whatever, but like you know,
Speaker 4: compared to a startup where yes, you have the hope
Speaker 4: of getting an exit, but you're probably not gonna make
Speaker 4: like five hundred thousand dollars right as a programmer, you're
Speaker 4: gonna make what half of that. I mean, we don't
Speaker 4: have to get into like exact numbers, but like the
Speaker 4: pay is in the right ballpark for like mid level folks.
Speaker 4: It's not going to pay at the executive level. But
Speaker 4: as he mentioned, like at the executive level, if you've
Speaker 4: been at Google for twenty years, pay is not your
Speaker 4: biggest concern. I do want to push back a little
Speaker 4: bit on the notion that that is he is in government.
Speaker 4: There are some of those people, but there are far
Speaker 4: more people who just have good tech and design skills
Speaker 4: who just want to make a difference more than anything else.
Speaker 4: And we should be working on all of the constraints
Speaker 4: that are keeping that hiring process at nine months more
Speaker 4: than we're than we're like calling up the people who
Speaker 4: manage the Department of Labor in a state and yelling
Speaker 4: at them because they had these failures, like fix the
Speaker 4: environment in which they're working. Give them that capacity, then
Speaker 4: you can hold them accountable, but they can't hire people
Speaker 4: right now.
Speaker 3: You also you need to be able to hire people,
Speaker 3: and you need them to be able to use their
Speaker 3: judgment to make higher order decisions than just do we
Speaker 3: enter the phone number with the keyboard or by clicking
Speaker 3: on a virtual pad of numbers. Like it is also
Speaker 3: the case that some of these systems are the way
Speaker 3: they are because it's just jen this might be your phrase,
Speaker 3: it's like policy vomit. It's just the rules put onto
Speaker 3: a screen, just like all the rules in statute put
Speaker 3: onto a screen. And so I think the thing I'd
Speaker 3: like to add is the whole point of product management
Speaker 3: is to arbitrate the trade offs between user experience, compliance goals,
Speaker 3: technical cost of change, and technical trade offs. And if
Speaker 3: you don't give those people if you hire them, but
Speaker 3: they don't have the ability to say, well, maybe we
Speaker 3: need to simplify this to make it make sense to
Speaker 3: a person. If instead the only response they get is
Speaker 3: what we need to put on the page exactly what's
Speaker 3: in statute then, or whatever other compliance goals there are, like, oh,
Speaker 3: we need to strictly follow this process. It's procedurally what
Speaker 3: it is. If those are counter to the outcome. You
Speaker 3: need a change in government where outcomes start to be
Speaker 3: it outcomes in this area around technology start to sort
Speaker 3: of have a higher importance than the strict process and
Speaker 3: the strict procedures. And I do think that that's a
Speaker 3: necessary corollary to bringing smart capable people in because if
Speaker 3: you bring in a superb product manager and they're just
Speaker 3: they have no ability to sort of bring these trade
Speaker 3: offs up and make those calls on the ground, you're
Speaker 3: going to get the same status quo.
Speaker 4: I think.
Speaker 2: So I can't resist asking this question. I take all
Speaker 2: the points about, like, here's what we need to do
Speaker 2: in order for this to change, But can I just ask,
Speaker 2: because both of you have that practical experience of doing this,
Speaker 2: what was the craziest thing that you saw during your
Speaker 2: respective tenures, Like, what was something that just really surprised
Speaker 2: you or shocked you. I'm back to the banging your
Speaker 2: head on the table a portion of this conversation.
Speaker 4: I mean, I guess the thing that comes to mind
Speaker 4: just because it really made an impression on me, and
Speaker 4: it's it's a negative story that I want to sort
Speaker 4: of balance out with something positive. But I was working
Speaker 4: in the White House and had decided to do what
Speaker 4: we convinced the Veterans Administration to do what we call
Speaker 4: discovery sprint on the Veterans Benefit management system. And it
Speaker 4: was going really slowly. There was like huge latency, and
Speaker 4: I got these two folks in the help out for
Speaker 4: just a couple of weeks and just you know, do
Speaker 4: an analysis of what was wrong before they go, like,
Speaker 4: you know, trying to figure you can't solve the problem
Speaker 4: till you know what it is. So like the first
Speaker 4: thing was we show up and the guy's like, oh,
Speaker 4: I'm glad they sent the White House to you know,
Speaker 4: verify that everything's fine now. And it turns out everything
Speaker 4: was fine because the leader had defined the latency as
Speaker 4: under two minutes. Okay, so you've just defined the problem away.
Speaker 4: So people who are trying to process benefit fit applications,
Speaker 4: like if they clicked and it took one minute and
Speaker 4: fifty nine seconds for the application to respond, it was fine,
Speaker 4: no problem here. Wow. But then I kept talking to
Speaker 4: the guy and I kept asking him questions about the system, like, well,
Speaker 4: why is it designed this way? Why was it designed
Speaker 4: that way? And he said, I've spent my career teaching
Speaker 4: people not to have an opinion on business requirements. The
Speaker 4: IT people should have no opinions there. And I said why,
Speaker 4: He said, well, if they ask us to build a
Speaker 4: concrete boat, will build a concrete boat, because then it's
Speaker 4: not our fault. And I was just like, I felt
Speaker 4: like I'd been gut punched, Like it was so hard
Speaker 4: to hear that, Like there were I think at the time,
Speaker 4: eighteen veterans a day committing suicide, most of whom couldn't
Speaker 4: get their healthcare benefits. And I just could understand how
Speaker 4: someone could take that approach. But honestly, you know, in
Speaker 4: the ten years since then, I've seen a lot of
Speaker 4: reasons why he felt like he could say that, And
Speaker 4: you know, I don't want to blame him, I want
Speaker 4: to blame the system. But yeah, if you tell us
Speaker 4: to build a concrete boat, we'll build a concrete boat. Like,
Speaker 4: how what way is that to run a country?
Speaker 3: I have a much more technical problem, and please yell
Speaker 3: at me if this is too technically in the weeds.
Speaker 3: So at one point, we in my old work on snap,
Speaker 3: we observed a system, one of the one of the
Speaker 3: official systems that was user facing, so external users like
Speaker 3: folks trying to apply for for essentially use it. And
Speaker 3: we noticed one day that it wasn't loading right, and
Speaker 3: it wasn't loading in the following way. You would go
Speaker 3: to the website, it would have buttons and everything loaded,
Speaker 3: but then when you try to interclick on anything, it
Speaker 3: wouldn't work, and it wouldn't work weirdly for about a
Speaker 3: minute and a half, like then all of a sudden,
Speaker 3: you can click on things and we're like, what is
Speaker 3: going on here? This is bizarre, And we looked around,
Speaker 3: and so we we did some debugging on the client side,
Speaker 3: meaning like we opened up the web browser inspector and
Speaker 3: we said, okay, what's going on? And what we found
Speaker 3: was this website was loading I forget exactly the name.
Speaker 3: It was like translations dot js or translations dot xml,
Speaker 3: and it was taking a min and a half. Everything
Speaker 3: else was loading fine, but this was taking a minut
Speaker 3: and a half, and counter to best practice, it was
Speaker 3: you know, blocking any interaction with the page until it
Speaker 3: was fully fully loaded. And it was loading very very slowly.
Speaker 3: And we looked at it and it says giant file.
Speaker 3: I was like, I don't know. It was like fifty
Speaker 3: megab items a giant file, Like what is going on here.
Speaker 3: Looked into it and it was a translation of every
Speaker 3: page on the entire system into like eight languages, and
Speaker 3: we call the thing and that was like, okay, this
Speaker 3: is not great, and there's ways to fix that. Like
Speaker 3: I said, you can let people interact with the page
Speaker 3: until before that's fully load and it's fine. Also, you
Speaker 3: definitely shouldn't put all the translations for every page in
Speaker 3: your website on eight pages into a single file that
Speaker 3: you sends to the client, Like that's just bad development practice.
Speaker 3: But the thing that maddened me is we being good
Speaker 3: hearted people who were not out to, you know, just
Speaker 3: sort of lob bombs and talk trash. We contacted the
Speaker 3: entity who ran it and we said, hey, this is happening.
Speaker 3: We're seeing consistently, this is the issue that's going on.
Speaker 3: And they said, no, no, no, no, that's not having us load.
Speaker 3: Totally fine for us. And the reason is because we
Speaker 3: diagnosed it down to one or two reasons. One was
Speaker 3: they'd already loaded the file once so it was cash
Speaker 3: and so it was on their computers every time they
Speaker 3: went to it. It was fine and like, oh yeah,
Speaker 3: it works for us. The other was and this was
Speaker 3: our better guess is their server that was serving this
Speaker 3: was on their network and actually sending it much more quickly,
Speaker 3: and they were just like, well, it works on our network.
Speaker 3: It works totally fine. And I don't know exactly what
Speaker 3: it was. But I think that is the reason that's
Speaker 3: so maddening. And it gets back to a little bit
Speaker 3: like Gem's point about accountability is the truth is we
Speaker 3: can't fix problems that we don't see or don't recognize.
Speaker 3: And I do think that they're a big part of
Speaker 3: all of this is we have some expectations that websites
Speaker 3: should work, that technology should work. We also need to
Speaker 3: give more. We have to set an expectation I think,
Speaker 3: government wide about what stuff working looks like. Like Jen
Speaker 3: gave an implicit one of a normal human should be
Speaker 3: able to understand this. That should be tested on every
Speaker 3: system in the e commerce world. Like what's the number
Speaker 3: one metric you care about? It's from start to finish?
Speaker 3: How many people start transaction and actually end up purchasing
Speaker 3: that conversion rate that I mean, it would be a
Speaker 3: really useful feedback loop to measure the success rate of
Speaker 3: users trying to do a transaction across every transaction in government,
Speaker 3: right like people applying for food, SAMs, people applying for whatever.
Speaker 3: If you saw massive drop off or massive disparities across systems,
Speaker 3: at least it would tell you, oh, there's something we
Speaker 3: need to fix here. So I think there's another layer
Speaker 3: of all this, and that's my depressing story because eventually
Speaker 3: it got fixed and I think probably they figured out
Speaker 3: we were right like a week later, but it took
Speaker 3: a week and it was a very I think that's
Speaker 3: why in my mind, there's also if we can get
Speaker 3: to better accountability and give people better on the government side,
Speaker 3: better visibility into these issues with possible routes to solving them.
Speaker 3: They want to solve them, but right now we don't
Speaker 3: have a system that necessarily looks for those issues. And
Speaker 3: I think that would be a better form of the
Speaker 3: accountability we have relative to say, did you follow every
Speaker 3: IT checklist bullet point in policy? Right? Like, if we're
Speaker 3: getting bad outcomes while following that, then maybe something else
Speaker 3: needs to change and maybe there's a different form of
Speaker 3: accountability necessary there.
Speaker 4: We should give a quick shout out to the President
Speaker 4: Biden's Customer Experience Executive Order that is trying to get agencies.
Speaker 1: Maybe in that direction, there is so much to talk
Speaker 1: about it, so many different subversions of this conversation we have,
Speaker 1: but uh, that was amazing conversation. Jennifer Palka, Dave Gorino,
Speaker 1: thank you both so much for our coming on odlocks,
Speaker 1: sharing your time and insight and direct experience and all
Speaker 1: this stuff.
Speaker 5: Yes, delight, thanks so much, Thank you, Thank you, Tracy.
Speaker 1: That was an extremely illuminating conversation I found. I mean,
Speaker 1: it's like easy enough to say like, oh, government is
Speaker 1: bureaucratic and they don't pay enough and that's why they
Speaker 1: can't build software, but they were like there was like
Speaker 1: so much richer and more complex than I like appreciated.
Speaker 5: Right.
Speaker 2: The emphasis on incentive was really interesting and sort of
Speaker 2: fits into a lot of the discussions we have here.
Speaker 2: And also Jen's point about a lot of this emanates
Speaker 2: from the policy yes side, yes, right, and you know
Speaker 2: it's people trying to tick various boxes that have been
Speaker 2: set in stone by policy.
Speaker 4: I got to say one.
Speaker 2: Thing coming out of that, though, I feel a newfound
Speaker 2: appreciation for being a journalist. And actually, you know, when
Speaker 2: you and I produce an episode or write an article,
Speaker 2: we sort of send it out in the world and
Speaker 2: then it's done, and we can move on to the
Speaker 2: next thing.
Speaker 4: We don't have to go back in ten years refine it.
Speaker 2: Yeah, in response to customer feedback and various testing and
Speaker 2: things like that.
Speaker 1: If we were if we were Wikipedia editors, we would
Speaker 1: have to be continually editing them for years. I thought
Speaker 1: that was really great. And again Jen's point that you
Speaker 1: just brought up, it's like this sort of like reflection,
Speaker 1: like the real solution like has to emanate from the
Speaker 1: policy level. I was actually getting drinks recently with a
Speaker 1: friend of mine who's a software engineer, and he said,
Speaker 1: there's like this phenomenon. He said, it's called Conway's law.
Speaker 1: Like every organization, he said, like ships, its organ structure
Speaker 1: and this then it makes like so much sense, Like
Speaker 1: the software is complicated because US politics is like complicated
Speaker 1: ninety years of policy changes and Democrats and Republicans switching
Speaker 1: who's in power, et cetera. It's like, no wonder, like
Speaker 1: the ultimate like product of that thing is going to
Speaker 1: not look like you know, Google dot Com.
Speaker 2: Well, also the story of the guy who is like
Speaker 2: it takes twenty five years to really know how to
Speaker 2: file acclaim. You can imagine if you spend twenty five
Speaker 2: years developing expertise in this one very specific skill set.
Speaker 2: You're not really incentivized to start like changing that anytime soon.
Speaker 2: And so the system just kind of perpetuates itself so
Speaker 2: many different directions, So we can go with that conversation.
Speaker 1: So many different ways. The system is a self perpetuating Yeah,
Speaker 1: I learned a lot from that.
Speaker 4: Yep.
Speaker 2: Shall we leave it there.
Speaker 1: Let's leave it there.
Speaker 2: This has been another episode of the odd Lots podcast.
Speaker 2: I'm Tracy Alloway. You can follow me on Twitter at
Speaker 2: Tracy Alloway.
Speaker 1: And I'm Joe Why Isn't Thal? You can follow me
Speaker 1: on Twitter at the Stalwart. Follow our guests on Twitter.
Speaker 1: Jennifer Palka, She's at Paulka dot and she is the
Speaker 1: author of the new book Recoding America. Definitely check it out.
Speaker 1: Follow Dave Guarino, He's at Dave Underscore Glorrino, follow our
Speaker 1: Carmen Rodriguez at Carmen Arman, and dash Oll Bennett at dashbot.
Speaker 1: And check out all of the Bloomberg podcasts under the
Speaker 1: handle at Podcasts and for more odlots content, go to
Speaker 1: Bloomberg dot com slash odd Lots, where we post transcripts.
Speaker 1: Tracy and I have a blog and a newsletter that
Speaker 1: comes out every Friday, and you can talk about all
Speaker 1: of these things twenty four to seven with fellow listeners
Speaker 1: on the Odlogs Discord. It's really fun. I hang out
Speaker 1: there a lot, go to discord dot, gg slash, odd lots,
Speaker 1: so it's a real blast and a fun place to
Speaker 1: hang out on the Internet. Thanks for listening.
Speaker 3: E