Partnership with City of Memphis to Improve City Services

Problem Statement:

Like many cities and towns in the U.S., the City of Memphis has a problem with potholes.  In February 2019, they sought SpringML’s help to come up with a proof of concept (POC) to address this issue.

Proposed Solution:

With the advances in machine learning and Artificial Intelligence, we can detect potholes through video footage of the road and share the information workers that can repair them.

Here is a snapshot:

 

With the POC, the city of Memphis hoped to replace the current antiquated, time-consuming, and expensive process of identifying potholes with the latest in Artificial Intelligence and Computer Vision.  Last year, we shared in a blog post about how we can use machine learning for pothole detection. We used this in the POC to take it even further.

General Process:

1. Use videos with front cameras which pointed towards the roads.
2. Extract frames from the videos.
3. Segregate the images to training and validation dataset and annotate them.
4. Train the model with around 5000 iterations and save the model weights.

Challenges:

1. Data acquisition was a challenge since videos were delayed by a month.
2. Resolution on the videos was as low which made it very difficult to check pothole boundaries.
3. Weather challenges such as rain obstructed the view of potholes. Similarly, videos recorded in early mornings/night time were not useful for model construction.
4. Filled potholes, bigger cracks, semi-filled manhole covers were sometimes misinterpreted as potholes.

Some highlights:

1. We overlaid the detection of potholes on a static video to show the results on a previously recorded video.  
2. With a google 360 camera mounted on a car, a GPS coordinate is tied together with the detection of the pothole as the car thoroughly drives through all the streets.  That information can then create a help ticket for someone to fix it.
3. We were also able to detect and visualize multiple custom objects and train it on a VM on the Compute Engine.

User Interface (UI):

We created a UI to demonstrate how this model could be used for the City of Memphis:

By clicking on one of the Google pins, You can pull up information on a detected pothole, with various information to decide whether to submit a help ticket or not.

Here is the link to try it out yourself here, built using App Engine from Google.

Potential Next Phases:

1. Further develop the custom multi-object detection model to detect many other objects in one pass through a video.

   

2. Integrate the model into their existing platform to send out help tickets for workers to work on the pothole repairs, going beyond just a demo.
3. Create a route optimization plan to repair the potholes in the most efficient path.
4. Establish additional machine learning models to predict which road segments are likely to get more potholes than others. Include other features such as weather data, current ratings, road quality, etc.
5. Detect which potholes have increased in size over time and create additional tickets for them.
6. Shift from bus videos to insta 360 footage with higher resolution to track the depth of the potholes and understand the amount of road materials required to fill them.

We look forward to continuing work with the City of Memphis to develop this further and apply these concepts to other use cases for other clients.

Feel free to contact us at [email protected] if you have any questions or need help with any similar use cases.