Dr. Jeffrey Wishart and the trough of disillusionment

Anthony: You’re listening to their auto be a law, the center for auto safety podcast with executive director, Michael Brooks, chief engineer, Fred Perkins, and hosted by me, Anthony seminar for over 50 years. The center for auto. Safety has worked to make cars safer.

Welcome everybody. Today we have a special guest. We have Dr. Jeffrey Wishart, who is is an expert in automated vehicles. I recently wrote a book called fundamentals of connected and automated vehicles for SAE international. He is at the Arizona commerce authority and. The sound science foundation of Arizona that is a very short description of your background.

Is that accurate?

Dr. Jeffery Wishart: It is. You got the name, right? Most people don’t get it right on the first go. So impressive.

Michael: Some of this stuff is things that, auto manufacturer have been really trying to develop for, 70 years, even we’ve seen some efforts by the industry to do and, when you look at that, how long we’ve been trying to do it, and then you have, the DARPA challenge and some of those other things mingled in, and yet we’re still seeming to really trying to get our hands around it, it it, it sheds some light on, why many people have been skeptical of the continued pronouncements from some of the industry players that, it’s going to be here next year. It’s going to be here in five years. You can’t really say that.

Fred: When I started graduate school, people were talking about fusion energy as the energy of the future.

And now it seems that people are saying that fusion energy is the energy of the future and it always will be, it seems to be a scenario that also is applying to the safe self driving vehicles. There are a few like

Dr. Jeffery Wishart: that. People talk about hydrogen the same way. I’ve got differing thoughts on all of that, but I think that around 2015, 2016, we really started to on the Gartner Hype Cycle, get to that peak of excitement.

And I think we might be in the trough of disillusionment at this point. We’ve had a lot of bankruptcies in the industry. We’ve had a lot of Claims or predictions that haven’t come true. We are by 2020, a lot of people thought this was all going to be very common on public roads. Clearly not the case so I think that there are some things that need to be changed in order to get us to where we see these vehicles all over the place and all sorts of different use cases but, I think that the progress is being made, but certainly not at the pace that we thought just a few short years ago.

Anthony: Yeah, so going through your book, you cover the history of autonomous vehicles and different approaches people have taken, and you’ve talked about some things in there that we’ve mentioned a lot on the show that I think, hey, this is a great idea. Things like V2X uh, and there was something specifically that I just mentioned before you joined us.

And it was it wasn’t adaptive cruise control. It was cooperative cruise control, which to me just sounds amazing because I always use kind of adaptive cruise control and I get mad when the people behind me don’t.

Dr. Jeffery Wishart: Yeah. Connectivity is the missing piece. I think with automation we really need connectivity to make.

Automation work well. I think that if we could be talking to each other in our cars, to the pedestrians and cyclists on the road, to the infrastructure, there are all sorts of advantages that would come with that. The problem is, I think you guys have talked about this on your show, the FCC decided a few years ago to cut half the spectrum.

But it was really, a lot of people were not happy with that decision, but it was a use it or lose it situation and people weren’t using it. 20 years, we’ve been talking about. Cars that talk to each other and then the infrastructure and very little was done beyond pilot projects. There are lots of pilot projects all over the place.

There’s still pilot projects all over the place, but we don’t have any real deployment when it comes to the OEMs. And so the FCC said we’re going to, we need this spectrum for our wifi. So we’re going to allocate it there. And, but yeah, as you say, I think there’s lots of advantages when it comes to connectivity.

If we could. If we were speaking to each other on the highway and we had adaptive cruise control, studies have shown that you could reduce your energy consumption, you could improve throughput safety would be enhanced, all sorts of advantages. But the challenges when it comes to connectivity, the cybersecurity challenges, the trusting of data that the OEMs themselves didn’t didn’t produce.

I think that they, so far we just, for whatever reason for all those reasons, we just don’t have that in our, on our roads.

Anthony: But I feel like we we have some connectivity in terms of if you use Google Maps or something like that. And, it says, hey I’m gonna reroute you this way.

Or, hey there’s an obstruction up ahead. There’s never an obstruction. It’s always a lie. Never find this obstruction. They’ve gotten really good with, there’s a speed trap up ahead. But so we’re having that connectivity happen right now.

Dr. Jeffery Wishart: Yeah, absolutely. There’s depends on how you define connectivity, right?

If you’re defining connectivity by vehicle to X, DSRC type technology or CV to X we don’t have that yet, but we certainly have been getting, people have been using the car as a hotspot for some time now. So yes, as you say, we all get. Some sort of information into our cars at this point.

Most cars are connected but not connected in, in the way that I’m saying when it, that would really help us when it comes to throughput, energy efficiency and

Anthony: safety. Do you think that’s going to happen? Sorry for that. Where basically someone like Google or Apple is essentially, they have enough of these devices and enough cars on the road where they essentially become the de facto standard.

Dr. Jeffery Wishart: I think what this industry seems to be moving towards CB to X. So using the cellular network. So rather than having what’s called roadside units periodically around the country, everywhere. I don’t, I think that’s just too expensive. They’re around 50, 000 per unit. And so you think trying to put one of those.

Every few miles or at every intersection it’s just not going to happen, I don’t think unless it comes down quite a bit in cost, but so now using the cellular networks to make it happen. We have latency issues. We have cybersecurity issues that need to be addressed. But I think that’s probably more promising, and in certain countries like China is really big on accelerator V2X.

So I think that’s where the industry seems to be heading, and there’s some USDOT grants out at the moment. It’s called the V2X Accelerator, and my research group is looking at this. So this is Looking they want they do want some RSU components, but they also are interested in these futuristic or more future forward thinking cellular applications.

Fred: Jeff, one of the objections we’ve had to the CV two X is that in comparison with DSRC, it’s hard to anonymize the information that’s going across because of course, it’s IP based. I’ve read recently about some technology that would use C two C two vx or cellular technology, which is anonymized.

Can you give me any enlightenment on that? Do you know anything about that? I’m

Anthony: sorry, before you jump into that, can you, yeah, sorry. Can you just define a couple of those? We’re using a lot of high level acronyms and I know the two of you were like, let’s jump in. Okay. So V2X we’ve talked about is vehicle to infrastructure.

And then you had a bunch of acronyms in there and I died.

Fred: All right. IP address is the numerical code that identifies whatever device you’re using to connect to the internet. And DSRC was the protocol that was being developed by the government for communication between vehicles and each other.

And and the rest of the world, the rest of the universe, which was called V2X, and that had the characteristics that it was completely anonymous. So it did not have an IP address associated with it. Now, the problem is. Or the potential problem is that if you have all the cars connected through cellular technology is always associated with the IP address of your vehicle, or your phone, or whatever, and this is how people hack into other people’s phones using technology that’s being connected.

Promulgated by bad actors around the world. So the question is really, if you’re using cellular technology to connect vehicles, doesn’t that leave you open to manipulation by people who can hack into your particular, Phone or vehicle or whatever it is connected to the cellular network. Now, what I’ve read recently is that there is some discussion of using those frequencies in that technology, but not using the IP address or somehow anonymizing or making anonymous the participants in that cellular technology.

Does that make sense?

Dr. Jeffery Wishart: Thank you. Yeah, I think I am not a cybersecurity expert by any means. And I think that it’s a absolutely crucial aspect of things that are flies under the radar a little bit. There are, there’s a big need for more cybersecurity experts in the automotive industry. I was at an event recently where they said, not just automotive industry related, but there were 15, 000 open positions in Arizona.

When it related to cyber security. So I think that’s a big missing piece of this. And, the cellular providers are working on this, as you say, we have some cellular partners in Arizona with our research group. And that’s what something that they talk about, that they look, they need to understand what those issues are.

And anonymizing is one of those Techniques, I think that they would be looking at to ensure that, that we aren’t if you can’t trust the messages coming into you, into your vehicle, you can’t make use of them. And that’s, I think, been one of the major reasons, as I said before, why the OEMs have not been open to.

The connectivity side of things. They say look, we’re just going to do things ourselves. We’re going to we’re going to perceive everything that we need to perceive and anything that comes to us. So they might be open to certain things like single phase and Tommy messages or spat messages, getting some information like that, but anything that’s safety critical.

I haven’t seen much openness to, to it, but as you say if we can make the messages robust I think that’s something that, that would help convince them. And so one of the things I’d like to do, and I’m trying to develop a research. Project around this is a trust building V2X project where we provide the perception at a location to the automated vehicles.

And they compare it against perception that they’ve done on board the vehicle. And if those things match, that’s a way of building the trust. So they can trust both the accuracy and the security of that message. I want to get back

Fred: to the book that you’ve recently published, and for our listeners, the title of the book is Fundamentals of Connected and Automated Vehicles by our guest and a lot of other distinguished people.

Jeff, let me ask you, who is the target Audience for this book.

Dr. Jeffery Wishart: So that actually stems from a course on connected and automated vehicles that I teach at Arizona State University. Each chapter is a lecture that I had. So I had all these lecture notes and I thought why not write a book about it?

And then I quickly realized that it’s a lot of work to write a book. And so I brought in some co authors. And so the audience is The first audience was my students, but now the audience is the, I tried to write it for maybe to reach both the lay audience who’s interested in this topic, but also the engineers who want a quick and quick way of understanding the basics, the fundamentals of the technology.

So each of the chapters talk about a different aspect of the technology and, each chapter could be a book in and of itself, but it’s getting you a intro fundamentals understanding of those concepts. I did

Fred: think it was a great survey of the fundamental concepts being an engineer, as we do, the first thing we do is stick a screwdriver into something and twist it to see if we can break it.

I have a couple of detailed questions if I can go ahead and throw them at you. So I’m looking at Page 189, and it says in the case of ADS, which is automatic driving systems, there are no regulations or standards, currently, that dictate how an ADS development and or subsequent V& V process, V& V means verification and validation, must be conducted, which type or amount of testing must be conducted, and what level of performance is required for the ADS.

Now, to me, that gets to the heart of the problem that we have with self driving vehicles or AVs. Where does the industry get its requirements for safety and reliability? And those are our

Dr. Jeffery Wishart: key

Fred: parameters that should be included. And everybody’s designed for a self driving vehicle, yet there seems to be an awful lot of opaquety around that.

We just can’t see what people are doing, and the government hasn’t stepped up to that. What are your thoughts

Dr. Jeffery Wishart: on that, Jeff? Yeah, that’s great friend. You pinpointed exactly the problem. Right now, if you’re developing an automated vehicle, you have to pass FM FMVSS, so Federal Motor Vehicle Safety Standards, just like any other car that’s on the road.

If you get into a collision, you have to report that to the, to NHTSA under their standing general order. But that’s. That’s basically it, then it goes state by state, each state will have different regulations when it comes to deployment on their, on the public roads. But as you said there, there’s no real process that’s required of any of these vehicles.

Some of the, there are lots, there are standards out there and there are lots of them. I was at a conference recently where someone showed just ISO standards and it was like a hundred of them. And so there’s a, there are lots of standards for which to choose, but there’s no real guidance on, oh, you have to do this one.

You could ignore this one. NHTSA in 2020 had their advanced notice of proposed rulemaking where they recommend, they started to recommend a three pretty major standards in UL 4600, ISO 26262 and ISO 21448. But it didn’t get, Pass the NPRM stage. So yeah, everyone’s doing their own thing.

Everyone is telling, saying basically you have to trust us. And there’s a lot of criticism, I think fairly against some of these companies who say, trust us and they don’t want to give us any data to backup their claims, but I guess in their defense. And I’ll do that for shortly is that they, there isn’t that process for them to go through.

They, there’s nothing for them. There’s no benchmark for them to compare against. And so that’s absolutely what’s needed. I, we, I think we all wish that. NHTSA would step in and develop some of these regulations that are required a way for us to have confidence that these vehicles are safe to be on public roads, but this hasn’t done that NTSB has been banging their drum for several years to get NHTSA to do it.

And it hasn’t happened yet. But this is just not the FAA, the aviation industry had some bad years where there were a lot of accidents, a lot of people died, but they have cleaned up their act and they really do have a pretty good safety record, the 737 Max, notwithstanding. And I think that, That’s a lot of us in the automotive industry look over at that industry, the aviation industry and say, Oh, I wish we could get there.

But this is just not the FAA people’s relationship to their cars is just not the same thing that people have a relationship with with airplanes. So I just think that it’s needed, but it’s going to be slow and it’s not going to happen at the pace that that we all want.

Fred: We actually tried to fill some of that gap with our consumer oriented AV Bill of Rights, and I know you’ve read those.

What was your

Dr. Jeffery Wishart: reaction to those? Yeah, I thought they were great. I think that, it was a while back you and I had that interaction, Fred, I can’t remember them specifically anymore, but I remember thinking, yeah, absolutely. This is something that is could be adopted in some form by NHTSA.

And, I’m glad you guys did that. I’m, my research group is doing Similar. We I mentioned that there’s no benchmark against which to compare these vehicles. My research group is trying to develop that benchmark so that we can if society and through the regulator establishes what that benchmark is my process will allow for that to happen.

We, we aren’t going to say it has to be this much X safer than the human driver, but if NHTSA makes that, that call. Then my, our process would would be usable for that purpose. And so my plan is to develop the research, toss it over the fence to myself as the V& V Task Force Chair under SAE’s On Road Automated Driving Committee, and then hopefully that influences the regulatory body.

Like you, I think that we’d like to see some of these regulations and trying to help NHTSA get there, right? I think if we just, if we’re just passive and wait for it to happen, it’s not going to happen as quickly as if we all work together and put things out there into the public domain and say, here’s something that you could use.

Anthony: Is the industry really just jumping ahead of itself? They’re jumping right into level four and level five driving. We’ve seen with, GM crews, not ever sure what happened to them. But They’re trying to get to that level, which is full autonomy, no steering wheel, no pedals and whatnot, whereas systems in level two, which is what I think a lot of cars have, is like lane keeping assist and automatic emergency braking, that stuff doesn’t work yet.

There’s no regulations around that. It works, but there’s no standards and sometimes it doesn’t work like on my vehicle. But shouldn’t they? get all of that working first before saying, Hey, let’s fly. Let’s crawl

Dr. Jeffery Wishart: first. Yeah, it’s an interesting question because I think each of the levels have serious challenges with them.

So for level two, for example, as you mentioned we’ve got, up, the level two is very interesting because it can range from just lane centering and adaptive cruise control all the way up to, this car can do Everything, I just need to always be supervising it, right? So the what’s called the the object and event detection response, OEDR is, it can vary in sophistication quite widely, but at the, it, regardless of how sophisticated it is, the driver is always responsible for the actions of the vehicle, and so the driver always has to be supervising that system.

And we know that humans are not great at supervising automation. So the HMI challenges are quite substantial when it comes to level two. And I don’t know, I haven’t seen anything that proves that a driver monitoring system can make sure that the driver is constantly engaged in the task and ready to, at a moment’s notice, intervene.

Let me

Fred: just decode the one acronym he referred to HMI, which is the human machine interface.

Dr. Jeffery Wishart: Thank you, Fred. Yeah, I do speak in acronyms. I make that mistake all the time. I

Fred: share that affliction.

Anthony: Yeah, that was adorable watching Fred do that. That was so much fun for me.

Dr. Jeffery Wishart: And then we get to level three.

Where you can, you’re, humans no longer supervising, actively supervising, but you have to be able to take over if the vehicle runs into trouble for whatever reason. And so then can you be sleeping? Can you be reading? Can you be complete? You’re completely disengaged from the drive dynamic driving task as it’s called, but you have to, within a certain time frame, be able to take over.

So if the vehicle knows it’s in trouble or it’s and tell asks you to intervene. What does that timeframe look like? Who’s liable in the transition from the, okay, you’ve been notified as the fallback driver and now you’re about to take over. What is the, where’s the liability lie? And is it safe?

Can that be done safely? Whereas you were completely disengaged from the DD dynamic driving task. And now you’re trying to engage. And try and get situational awareness such that you can be moving at 65, 70 miles an hour on the road, yet you have to be able to know what’s happening around you pretty quickly.

That’s very challenging. It’s another HMI issue there. And then as you were talking about level four we’ve got these vehicles that are, okay, so the human driver can be completely out of the loop. But can these vehicles figure out situations in which, for which they haven’t been trained? AI doesn’t extrapolate very well.

If it hasn’t seen it in training, what does it do? The idea I, you, you want it to not cause an issue, even if it hasn’t seen that that situation or scenario in training, but that’s challenging. AI is, I think there are going to be some advances in AI. But we’re not there yet. And I guess at the very least, you’d want the vehicle to pull over into, I don’t know if this is one of Fred’s favorites the minimal risk condition, but what does that look like?

And can they perform that? Safely is still a question too. Yeah I think that without the, to get back to Fred’s point without that process and the regulation, it’s hard for us to know where. What the safety impacts of these vehicles are. I think that there’s some white papers that have come out from some of the automated vehicle companies, but it doesn’t really show enough to for someone like me or the general public.

To say, yeah, I think that’s sufficient that they’ve shown that they’re safe enough to be on public roads, there’s nothing that we can’t have any confidence. I think in those claims yet. And I think that’s a situation that’s not ideal for anybody.

Fred: Let me just jump in the level three for our listeners is a self driving car that is designed for a human to take over.

In an emergency, whereas a level four car is a self driving car, which is designed such that a human does not have to take over in an emergency. The car will take care of itself. So that’s the key distinction between level three and level four. And Jeffrey mentioned a key word, which is liability, which I saw Michael perk up over there.

Michael, you want to talk about the liability consequences of these

Michael: Levels. I was actually, I may have been perking up over level three and thinking about, we see crashes all the time where there is, they’re almost instantaneous. Even human drivers aren’t able to respond to them.

And in a level three vehicle, I think those, we looked at the one, I believe a few months ago where the The truck cuts off a, some kind of boxy Kia and it, there’s literally no time for the driver to rack and the Kia flies in the air, rolls. Luckily they weren’t injured, I don’t believe, but it was a kind of a stark reminder that these things happen in milliseconds and.

I don’t foresee that there could ever be a way to design a system that’s going to allow a human driver to take over fast enough to avoid those. But at the same time, even if, humans can’t really avoid those crashes now anyway, but there, there has to be some, within those, that period of seconds, there has to be some way to avoid these crashes.

That are going to happen fast and that we are pretty sure humans aren’t going to be able to disengage from a movie or a catnap in order to address them in time, so it suggests that, level three is never going to be perfect at predicting and warning drivers and avoiding those types

Dr. Jeffery Wishart: of collisions.

And level three is really challenging because the AV developer, it’s, I guess the OEM has to be very explicit about what you can be doing, what you can’t be doing. If you are not engaged in the dynamic driving task you’re liable to potentially fall asleep, right? But is that allowed? Can you be eating?

Can you have something in your lap? Can you be looking behind you? It’s, so there is one. automaker that’s supposed to be coming out with a level three system in the U. S. And they have not been explicit about either what the driver can do or the liability in that transition period or what that transition period even looks like.

Yeah, I, for a while, I thought, no, there’s no way anyone’s going to build a level three vehicle but I may be proven wrong shortly. I think that it’s, level three is, to me, potentially the most challenging of the levels.

Anthony: Yeah, it was surprising. I think we’re seeing was, is Mercedes has a license to try out level three in Nevada, going slower than 35 miles per hour.

And we talked about liability because Mercedes made this kind of blanket claim saying, Hey, we’ll cover all of your liability if something happens. And then Michael, the lawyer is that’s never going to happen. There’s

Dr. Jeffery Wishart: no way. Yeah, the press arm is different perhaps than what the engineers were thinking, right?

So that’s always a challenge in a big company.

Anthony: But it almost sounded like what you’re talking about, that level 2 is really hard as well, that level 4 might be potentially easier. Whereas I think there’s a lot of stuff missing from level 4. Why not just put all these level 2 systems in there? Don’t make claims that they’ll drive your car, like some company does, and that they do everything for you.

Every time there’s human interaction and overriding of the system, Hey, the system, you can just learn that new data points that you can then feed into your level four system. Like, why did this person hit the break here?

Dr. Jeffery Wishart: I actually think that, I think that’s certainly potentially true.

I think that even better might be, so we’ve got what’s called active safety features. You mentioned a couple of them, Matthew, but those are actually, so if they’re intermittent. That’s considered active safety. So the lane keep assist is intermittent. Just if you start to drift and hit the lane marker, it’ll send you back towards the center.

So lane centering is sustained operations. That’s automation, but these active safety features like automatic emergency braking, for collision warning, blind spot warning, these things that are monitoring the human driver, those have proven safety impacts. So there was a study a few years ago that showed that for collision warning and automatic emergency braking reduced front to rear collisions by 50%.

And so a lot of these active safety features we should, we’ve seen that the rear cameras have proven reductions in collisions, things as you’re reversing your vehicle. Some of these technologies we know are safe. Now the Euro. Or the U. S. D. O. T. Has what’s called the new car assessment program and cap, and they test a few of these active safety features, and they’re thinking of adding some more.

I think that’s, I think that they’re getting a little bit more attention recently because of that trough of disillusionment potential for automated vehicles. And I think that’s you. definitely a promising way to go, that we, that humans are going to be in the driving loop. And yet we’re going to have these automation systems that will guard, that will act as guardrails and prevent us from potentially doing some dumb things.

Fred: There’s an interesting aspect to this in my mind, which is that no car that I’ve ever seen Informs the user as to what the margins are for safe performance. Now, every car has got probably a safe operating envelope. What does that mean? At three miles an hour, you can turn the car as hard to the left as you want.

Nothing bad is going to happen. At a hundred miles an hour, you have to be very careful. About how much you turn the car to the left because with only a few degrees of steering to the left, your car is going to go out of control. So this is what I mean by the performance envelope. One of the potential advantages of level four automation is that the car presumably would know what its performance envelope is.

And it would presumably stay within that safe performance envelope more consistently than would a human driver, because the human driver and your car and my car have no idea what the performance envelope is, right? I’m going to stop in the accelerator. We don’t get any warning that says, Hey, you idiot.

You’re going 85 miles an hour on a country road. What the hell are you doing?

Anthony: Put my wife in your passenger seat. You’ll get it. Don’t worry.

Fred: Oh, thank you. Is she generally available?

Anthony: Different

Dr. Jeffery Wishart: conversation now.

Fred: But, actually, the lane keep assist is interesting because I find that I actually hate it because when I use it, It tends to fail on curves that I wouldn’t anticipate it would fail on, and yet there I am hurtling towards the guardrails at 40, 50 miles an hour.

I generally turn it off. The only time I turn it on really is when I’m on an interstate highway and I just don’t expect anything bad to happen. Of course, then what the hell is the point? Because, you’re on a highway where you don’t expect anything bad to happen. Easy driving. Yeah, a few points

Dr. Jeffery Wishart: there.

Yeah, a few points there. First, yeah, I think it’s really important that these as the cars get more and more complicated we need more and more education for drivers, but you know what it’s like a dealership. You’re not going there and getting a full course on what the vehicle can do and what these limitations are.

And that’s a real problem. No one’s reading the manuals. And yet these, yeah. These features are getting more capable, but more complicated. But then your second point about shutting off the feature. Yeah. The features depending on the manufacturer, it’s going to vary between manufacturer each. So automatic emergency braking, as I said, has proven safety benefits, but not all automatic emergency brakings features are designed the same and have the same efficacy.

All right.

Fred: And there are no federal standards or any of the standards or what it has to include or how it’s supposed to

Dr. Jeffery Wishart: perform. Yeah, NCAP does have some of that, but it’s not a standard. It’ll give you your crash rating equivalent, but it’s not the same thing as a standard. But the, so you’ve got variants in the efficacy.

And you’ve got variance in, consumer adoption or appreciation for it. And as you say, if it becomes an annoyance to you, you’re liable to shut it off, and then you lose all those safety benefits. So there’s a real there’s a needle That needs to be threaded here that the you have to find the right efficacy and also the right user interface so that the driver is not tempted to turn it off and lose all those safety benefits.

So there’s a lot that goes into developing these features, and there’s still quite a bit of work to do.

Fred: Let me get back to your book for a second. I really enjoyed the discussion about metrics, and I know that’s something that’s close to your heart. But something I noticed about that is that it’s a couple things, actually.

First is that you don’t talk explicitly about any OEDR metrics. Object event detection and response is the process by which the vehicle sensors detect something, and then it processes the information. And then the vehicle has to respond to that information. So it’s fundamental to all these automatic processes that we’re talking about, all the way from automatic emergency braking to self driving.

And OEDR is an overall global characteristic, like miles per hour, right? That subsumes everything under it into a handy metric that people can understand. so I wonder if you could talk about that. And the other thing that’s related to that is that in the self driving vehicles, they all seem to rely on artificial intelligence, which includes neural networks and automatic self learning and all that sort of stuff.

That’s, those systems are opaque to human beings. We cannot, as human beings, understand what’s inside of those automatic processes. So the only way you can really develop metrics for those, it seems to me. Is to look at the statistics of stimulation and response, and frankly, the universe is not going to exist long enough for somebody to get all those meaningful statistical responses from every aspect of a self driving system, I think.

But anyway, your thoughts on that.

Dr. Jeffery Wishart: So when we first, so back in 2020 or so research group that I’m with, the Institute of Automated Mobility came up with a set of metrics for automated vehicle evaluation. So this is the first step in getting to evaluating how the vehicle’s performing in different scenarios.

And the way, we had some OEMs as part of our research group and talking to them about. What they’re open to providing and what they’re not, we, because of those discussions, we came up with a taxonomy of black, gray and white box metrics, black box metrics being metrics that you can measure without having any access to the automated driving system whatsoever.

Grey box being, all right we need a little bit of data from the ADS automated driving system, but not a lot. And the white box being substantial access to the automated driving system. So what you were and we did that purposefully because we want to make sure that there, you could, if you are a, say, a regulator of some sort or an IOO infrastructure owner operator, that you can evaluate, The driving performance of one of these vehicles without having to have access to the automated driving system.

I think we thought that was very important. So that’s where that taxonomy came from. Your discussion about the OEDR uh, and it’s measuring that, that performance. You’re right. Right now, the AI is opaque. It’s it’s a black box. So you can’t get. You can understand or see what’s happening on the inside, you can get the input output and you can measure those things so we could develop more white box metrics.

We have a few in our in our metrics. And I should mention that the task force is working on S. A. J. 32 37. Hopefully that will be published in the early in the new year. Under the SEA’s On Road Automated Driving Committee that has these metrics black, gray, and white, we have some white box metrics, but we didn’t get extensively into them because of that sensitivity from the OEMs.

We didn’t want to scare them off and not want to work with us. We were very judicious in having a few white box metrics require for that you, for which you require ADS data. And we could certainly look at different subsystems. You could get, you could have white box metrics that measure the perception, you could have white box metrics that measure the path planning both of those subsystems, you could do it and you could get into them.

You could say, look at your localization, you could look at your classification, you could look at your tracking and getting into all of those details. You can them. probe those aspects of the unmade driving system if you have basically unfettered access to it, but we have stayed away from them for now.

Maybe the next revision of J3237 will include some more of those. And maybe the industry will decide that’s important and we need to do that. We need to not just look at the vehicle’s performance from the system, from the vehicle itself. Did it go through a red light? Is it speeding did it avoid that object easy to see from a, you don’t even need to be on the vehicle.

You can get, you could have an infrastructure camera that could capture all of that, but maybe we’ll decide, no, we need to now dig deep into the ADS and see why it was doing what we saw at the system or vehicle level. Since

Michael: we’re on our segment, since we’re on perception real quick, I was just wondering, I had a question generally, but, perception, you have basically four categories of things that the vehicle’s doing there, you’re detecting the object, you’re tracking the object, if it moves, you’re getting a depth estimation for it, but then you also have this, Area called segmentation where, it’s basically the vehicle or the computer is classifying objects into appropriate categories and just looking at those four to me and also looking at, how that is generally where even in humans, we tend to screw up the most is then how we categorize things in the world and what we believe their attributes are before we act.

Is that the most challenging part? of

Dr. Jeffery Wishart: the perception. It’s, there are a few things that are challenging, but that’s, that is very true. You look at the Uber incident from several years ago, which is actually quite close to my house in Tempe, Arizona. The vehicle identified, misidentified her several times.

She, it was a challenging situation because it was a pedestrian crossing the road perpendicular to the vehicle. And she was walking a bike. And so it misclassified her as a pedestrian, as a cyclist, I think, for some reason, I think at one point, maybe even a plastic bag, but it switched back and forth several times from what is this?

And which is, as you say, Michael, very important to know, because you’re going to have very different characteristics, both from a what’s it, what speed are you likely to be moving at, but also your ability to change direction quickly, a pedestrian can change direction a lot faster than a car can and more you can go off much more obliquely.

So yeah, it’s. It’s a huge challenge that I, we don’t know, I don’t work for Waymo, so I don’t know how much better they are than Uber was back in 2018. So it’s, I, it’s probably is one of the biggest challenges they face, but because we don’t get data from them, we can’t say, okay, they are this good at it now.

So maybe to Fred’s point, we need some more of those white box metrics. To so that we have, we understand that I’m trying to, without the metrics that we’re doing in J 32 37, a backdoor way, if you if you don’t perceive things correctly. Eventually, you’re not going to drive appropriately, so if you don’t perceive that pedestrian, or you think that pedestrian is a car, you’re going to act differently.

We’re trying to understand from that perspective, but to really get an understanding of how the vehicle is performing top to bottom, yeah, you need a lot more of those perception, path planning metrics.

Anthony: So when you guys are in these SAE meetings, do you actually, uh, feedback from the OEMs saying, Hey, we do want regulations here.

We do want guidelines. I’ve never heard them say that.

Dr. Jeffery Wishart: Yeah, we don’t get into the politics necessarily or the regulatory scheme, but they, there are some very good people working from those companies working on the standards with Fred and I, and they very involved, very engaged and some of them are excellent collaborators. Yeah, it’s not all the companies are doing that.

I haven’t seen a Tesla. Representative in any of these standards committees, for example. And I wish they would engage more. I think that they do. I don’t know if you saw today’s news about the recall of insert

Michael: my shocked face there where you say they haven’t been at those meetings.

Dr. Jeffery Wishart: I wish they would.

I think it’s be better for them. It’d be better for the industry, be better for the public. But no, Anthony, yeah, we don’t talk about, at least not on the record. Talk about the regulatory schemes.

Anthony: I think it was more regulatory schemes, more like if I’m the engineer, I want to know, okay what are the guidelines I have to go for?

Basically what you’re saying is like, what are the standards I have to hit? Cause if not, it’s just the wild west. And one of my concerns from,

Dr. Jeffery Wishart: go ahead. As I said, there are no centers you have to hit, right? That’s the thing, right? They don’t need to. And until they’re forced to they’re not going to, I think that’s the crux of it.

Fred: That’s one of the problems when governments defer to industry. Consensus standards and industry documents that you’ve got to remember that the human beings involved in these meetings, myself included, Jeffrey included, and a lot of other people really do their best to inject their engineering judgment into what these standards are going to cover.

But it’s also important to note that almost all of these people are paid by their. Industry sponsors, and most of them work directly for companies, and even though they are supposed to represent themselves as individuals and not their companies in these meetings, the fact is that it’s hard for people to ignore the person who is They’re paying their, they’re giving them a paycheck every couple of weeks.

Dr. Jeffery Wishart: Yeah, that’s interesting because the IEEE is different than SAE. So SAE, they have these statements they read before every meeting. They say, yeah, you’re, as Fred said, you’re representing yourself, not any other organization. But then some people in their name, they put Say Jeffrey Wishart, Arizona Councilor, ACA, right?

So then it’s very clear are you representing yourself? I’ve just identified myself with this organization. Very different from IEEE, where you are explicitly, in those standards meetings, you’re explicitly representing your organization. Your organization has to pay to be a part of that. SAE, as Fred said, we are all individuals.

But yeah, it’s you don’t find Very frequently someone saying something that their organization wouldn’t like. I don’t think, I don’t think that’s gonna, I don’t know that, for sure. But I can’t imagine that they are saying things that go against what their

Fred: company thinks.

No, I think the other extreme is the organization called the AVSC or What does stands for automated vehicle consortium, which you refer to in your book as an SAE based consortium. But I always think of them as the resistance underground from people who want to really defy the government because they put out these documents as though they’re quasi official documents.

Yet there’s no. government organizations involved in it. There are no consumer oriented organizations involved in it. And to say that they’re SAE based somehow gives them an imprimatur, which I don’t think they deserve. I think they’re You know, in a sense, dangerous because they’re putting a pretty face on something that people don’t really understand what’s underneath the, the

Dr. Jeffery Wishart: tent here.

Yeah. Interesting. I guess I’m a little bit, I have a better view of them than you do, Fred. I, they, SCE has a. As a several different parts to this organization. So it’s run by one, one division of SAE. They’re called their ITC, which I can’t remember what that stands for. Technologies consortium. I can’t remember what the I is, but regardless the, yes, you, the process is opaque.

You can, not just anybody can. That can join it. So it’s cut paid companies that are doing it and they are very careful in their calls. I understand, but, they’re putting them out as best practices. And I think that’s useful. It’s like when you guys did your bill of rights. I think putting it out to the community for standards body, like SAE.

They toss over themselves or over the fence to themselves to, to adopt, to take on and make into a standard, I think is quite useful. So that’s one of the things that happened with the metrics work, J3237 the IEM, we had our research and we put that out into the public domain, AVSE put it out, theirs out, and what started out as an information report and SAE has three different levels to their documents, information report, recommended practice, and then standard.

In increasing amount of industry consensus. So what started as information report got upgraded to a recommended practice because the IAM’s work and the AVSC’s work was philosophically aligned. So I see what the AVSC puts out as useful and helpful to the industry. Yes, it’s not a standard, it’s not a regulation, but I think it’s there, they put out useful work in my view.

Oh, thank you for that.

Anthony: In your opinion, now how soon before we have fully autonomous vehicles?

Dr. Jeffery Wishart: Oh my god that’s a billion dollar, that’s a twenty billion dollar question,

Anthony: right? It’s a trick question because it happened two years ago, according to Elon

Dr. Jeffery Wishart: Musk. Yeah, exactly. I, so I live in Arizona and I see Waymos with no vehicles, or no driver Have you gotten into them?

Yes, I have. Oh what’s the experience? Transcribed Experience is quite good. I haven’t had any issues, although my girlfriend got into one and it went past the road that was, would have turned to our house and took a circuitous route for some reason. I’m not sure why, but other than that, it’s been pretty seamless.

Arizona, as you might know, has big wide streets and the infrastructure is quite good. It’s a block system. It’s pretty simple. I know your favorite person, Kyle. You used to say that their cruise vehicles in San Francisco would learn 48 times per mile in San Francisco than what they learned in Phoenix.

You come to Phoenix, it’s with the training wheels. It’s much easier. We do have traffic, but it’s not crazy traffic, and we don’t have as many pedestrians or cyclists, these sorts of things as San Francisco, at least in such a concentrated way. It was actually quite, it was, the experience was quite good in the Waymo vehicle.

I, I have my thoughts on deployment without the safety standards, but from my experience, it was quite good. Do you have a concern

Anthony: that the Waymo was trying to take your girlfriend away from you? Oh, you don’t want to go to Jeffrey’s house. Come on, let’s go get a drink.

Dr. Jeffery Wishart: Bring her back to the depot.

Yeah, I hadn’t thought of that, Anthony. Now I’m going to have nightmares.

Anthony: Sorry.

Dr. Jeffery Wishart: But I guess the, the, so I guess your question, Anthony, but when are we going to see them? We do see them in small numbers. I think that we’ve I don’t think that we’re going to be, Cruz, their plan was 10 times growth every year until.

Until things went awry I think that we’re going to see, I think Waymo as much is pretty cautious in, in, at least in terms of scaling or more cautious. So I think that we’re going to see them grow their footprint. We may get a couple of other companies Robotaxi companies. Starting, I think emotional is I think it’s going to probably start to deploy, uh, at some point.

I don’t have any inside information, but I think we’ll see some more robo taxi services. I think we’ll see some more of the tracking side of things. There’s some people think that’s a better use case because, or are these a simpler use case? Because you’re on public, you’re on highways, you’re at highway speeds, but you don’t have the pedestrians and cyclists and.

It’s not as complicated. It’s called the operational design domain. So that may be a better use case. Then you’ve got your shuttles that are moving at low speed on a university campus or a business campus or an airport. So because they’re low speed, the stakes are lower. There are some companies that are doing that and maybe we’ll get there.

Or then we’ve got the Neuros the grocery delivery companies that are, they could be riding on the sidewalk at very slow speeds. So I think depending on the use case we’ll see them they’ll come in varying degrees and at varying speeds. I should mention drones. We’ll see drones, we’ll see automated drones at some point too.

There are lots of different use cases out there and the timelines are going to be different for each of them, I think.

Anthony: Great, good to know. Hey I don’t know if you guys have any other questions. If not, I’m going to jump to some of our listener mail. So we’ve only got a couple of minutes left.

Michael: I think that’s pretty good.

Fred: Okay, Jeff. Thanks so much. This has been a great conversation. Yeah.

Michael: I feel like we could have gone a lot longer and maybe we’ll have to have you back soon to talk about. I wanted to talk about the smart drive program and some of the things they’re doing in Arizona, like preventing emergency vehicles from hitting intersections and all that stuff.

We can dive into that another time.

Anthony: Yes. You definitely have to come back for a part two so feel free to stick around for the next couple of minutes. But Hey, listeners, we want to thank Dr. Jeffrey Wishart, uh, with the Arizona Commerce Authority and the Science Foundation of Arizona. And he’s a professor at Arizona State University.

The book is titled Fundamentals of Connected and Automated Vehicles. It’s available from SAE International. We’ve got a couple good pieces of listener feedback this week. The number one, and I think Michael, you really have to address this is how long have you been funded by big auto?

Michael: That was a very interesting question that came in response from a flood of people who were not very happy with our comments about the cyber truck.

fairly unsafe. But we are actually, we’ve never been funded by Big Auto and we never will be. That’s really one of our founding principles is independence from any large corporate donors that are going to direct what we do. That’s what allows us to say anything we want and what allows Anthony to make fun of Kyle on a weekly basis.

Anthony: I’ve moved on from Kyle.

Dr. Jeffery Wishart: But we

Fred: would accept sponsorship from Piggly

Dr. Jeffery Wishart: Wiggly, wouldn’t we? Oh my God.

Michael: We will accept, we will accept donations from anyone that’s not, a tier one manufacturer or, we don’t want anyone who we don’t want anyone basically controlling our operations.

We we’re, like most. Ralph Nader founded groups. We are hedged towards independence from all the money that’s flying around the world these days.

Anthony: So we’ve made poor life choices. All right. So Robert writes in, he says, with the cyber trucks, almost unbreakable glass, how are emergency responders going to be able to quickly extricate the vehicle’s occupants if they are unresponsive and, or the doors are jammed in an accident?

That’s an excellent question. That’s

Dr. Jeffery Wishart: a great

Michael: question. That’s a, that’s one that I don’t know if there’s really a good answer to yet because we haven’t seen any incidents where emergency responders are having to deal with that sort of thing. I believe and from what I’ve been able to gather, very, on the web and from sources that aren’t exactly.

Direct law enforcement or life saving sources is that the jaws of life can deal with those windows and can deal with the stainless steel, but it will probably be a little more difficult. It may be slower for extractions. But it also points to the need for Tesla to develop better ways for passengers and drivers to exit post crash.

We know that some of their emergency releases on their automatic doors are hidden and in places that make it difficult for, Someone who hasn’t read their owner’s manual, which no one does, to be able to extricate themselves from the vehicle in the event of a crash. We know people have gotten trapped in the event of fires and in those situations.

And it’s something that I think, especially when you’re designing a battle tank of a vehicle that you need to really focus on is making sure that post crash safety remains on your radar.

Anthony: I’d like to point out that I’ve read my owner’s manual, and I’ve also been picked last in gym class a number of times.

Last question from Frank, another Tesla related question. Hello, I’ve observed that the current model of Tesla’s Y, Tesla TeslaWise rear taillights are exceptionally bright and that Anthony can’t speak. Even while driving, the lights are so bright that they look like two red laser beams aimed at the driver behind them.

Again, these are not stopping lights, but the regular driving night lights. Is this something you can investigate? Is there a standard around lumens?

Michael: I don’t know. There are, that’s an interesting one. I get that. I’ve never really heard anyone’s. Talking about we get a lot of feedback and then there’s even a number of groups out there that are strongly suggesting that led lights and some other things on vehicles today are excessively bright.

We’re hoping that some of the adaptive headlight. Rules that are coming down from NHTSA and just the new designs of those are going to mitigate some of that and ensure that headlights are properly aimed. That’s also another huge problem is that headlights move around on vehicles all the time due to vibration use collisions and aren’t aligned so that they can properly illuminate the road ahead of you and not blind other drivers.

So it’s something that is, we hope gets better with time, but. In the case of a rear, dry operating light that’s on all the time. I’ve never heard of that complaint. So I would encourage, if anyone has a similar complaint don’t just let us know, let NHTSA know, because that’s what they need to look into.

There are standards for the illumination of rear tail lights.

Anthony: Alright, and with that, again, I’d like to thank our guest, Dr. Jeffrey Wishart. The book is called Fundamentals of Connected and Automated Vehicles. Get it now, it’s a great Christmas gift. And I’d say it’s a great book. And thanks everybody.

Till next week.

Fred: Thanks very much. Bye bye. Bye everybody.

Dr. Jeffery Wishart: For more information, visit www. autosafety. org.