Temporal city modeling using street level imagery

Estimation of the temporal changes to a city is useful for city management, disaster recovery operations, and understanding natural phenomena. When several types of data are available for this task, the optimal type should be chosen depending on the changes that need to be detected. However, data of the desired type are not always available, particularly historical data. In this study, we propose two methods for detecting changes in a city, which can be used in complement to process available data types and detect changes in selected targets. The first method estimates the presence of buildings by comparing street-level images and a 2D city map of buildings created at different points in time. This method uses the Structure from Motion (SfM) technique to reconstruct a point cloud of the structures of the city, and matches the point cloud with the 3D building structures recovered from its 2D map. While 2D city maps are available for most cities, most are not very accurate. Therefore, this method is designed to overcome these inaccuracies and thus is widely applicable. On the other hand, the method cannot detect the following types of scene change: wall paintings, buildings that were reconstructed and closely restored to their previous shape, pedestrians, cars, and vegetation. The second method uses a pair of street-level images that are roughly aligned with GPS data collected at different points in time to detect such scene changes. This method uses the features of a convolutional neural network (CNN) in combination with superpixel segmentation to address inaccurate image alignment and it also enables change detection with pixel-level accuracy. Additionally, the second method is scalable for large-scale estimation because it can quickly detect scene changes by merely using an image pair without performing large-scale SfM. The authors consider the proper use of these two methods to enable temporal city modeling in various situations. We experimentally apply these methods to cities damaged by the tsunami that struck Japan in 2011 and the results show their effectiveness.