图1.非视距(NLOS)成像依赖于精确测量超短光脉冲从激光源到高速检测器(如单光子雪崩检测器(SPAD))的飞行时间。红色实线表示超短脉冲通过窗口到遮罩的路径“relay wall” that diffusely reflects light into a room, where it illuminates targets. The dashed blue line shows diffuse reflection back to the 中继墙 from a desk and table in the front of the room, and from a “bad guy” hiding in the background, who is illuminated indirectly. The green dashed line shows how the 中继墙 then diffusely reflects some of that light back through the window to the detector. The detector times the arrival of the few photons that return and calculates their path to map the interior of the room and spot the lurking “bad guy.”

寻求光学器件在拐角处看到的要求是古老的。波兰天文学家约翰尼斯·赫维留斯(Johannes Hevelius)于1647年制造了一个原始的潜望镜,但直到1854年法国发明家希波吕特(HippolyteMarié-Davy)才发明了海军版本。 1926年,试图查看视线外的仪表盘,导致RCA无线电工程师Clarence W. Hansell发明了用于成像的柔性光纤束。现在所说的 非视距(NLOS)成像 已经成为光子学中的热门话题,并出现在军事规划师的愿望清单上,这些人试图防御潜伏在等待攻击部队的老建筑中的叛乱分子。

The modern quest is not to invent a new type of periscope, but to peer around corners into hidden areas not directly viewable with mirrors or other conventional optics that see only objects in the line of sight. The hot new approach that revived the quest is directing very short pulses of light at a “relay wall” visible from outside so the light is scattered through the hidden area and illuminates objects out of direct sight. Those hidden objects then scatter some light back to the 中继墙, where it can be observed from outside. Using ultrashort pulses and detectors makes it possible to measure the time of flight of the scattered light, and those observations could yield 恩 ough information to map what is in the hidden area.


Like a lidar, the NLOS system starts by firing a short pulse of light, but the light is directed at a point on the 中继墙 which will scatter the light into the hidden zone. Triggering the pulse also triggers a detection system that watches another point on the wall for light scattered from the hidden area. Objects in the hidden zone then scatter the light from the 中继墙; some of the scattered light goes directly to the wall, but other light scatters to other hidden objects, which scatter some light back to the wall. As in lidar, only a very small fraction of the photons in the original pulse wind up in the return signal sensed by the detector, which records their time of flight from the start of the pulse.

An imaging processor collects the return signals, their location on the wall, and their time of flight through the hidden area. Then, the processing system algorithms reconstruct the path of the light inside the hidden area, including where the light 恩 countered objects that scattered it back to the 中继墙. The processing system collects data returned from a series of pulses scattered by the 中继墙 into the hidden area to build up a digital 3D map of the collection of objects in the room. Think of the whole imaging system as a virtual camera that reconstructs the scene computationally; Figure 1 shows how the system works.

拉梅什·拉斯卡(Ramesh Raskar)麻省理工学院媒体实验室的相机文化小组(马萨诸塞州剑桥) 在2010年开发并发布了该概念.1 安德烈亚斯(Andreas Velten)后来在麻省理工学院任教,现在是威斯康星大学(威斯康星州麦迪逊分校)的教授,继续论证了这一概念。在2012年,他和同事报告说使用条纹相机记录返回的信号并处理信号条纹相机图像,以记录隐藏体积内小模特的三维图像。2 散射过程导致光中的大多数空间信息丢失,但是在飞行时间上捕获超快信息可以使它们补偿损耗。

Their demonstration was impressive, yet it was limited to collecting photons that had made only three scattering bounces: from the relay wall to the surface of the object facing the wall, from the object back to the wall, and from the wall to the streak camera. That could show only the three-dimensional side of the mannequin facing the 中继墙. Some light scattered from the wall then was scattered off the mannequin or other objects in the hidden area, but the scattering process was so inefficient that few photons remained after their additional bounces off the other objects and their signal was too weak to detect. At the time, Velten expressed hope that better lasers and sensors and more powerful algorithms could recover 恩 ough information on reflectance, refraction, and scattering to allow reconstruction of hidden objects in the scene.



DARPA计划认识到需要新技术来克服最初演示的局限性。在2020年6月发表的评论文章中, 格拉斯哥大学的Daniele Faccio (苏格兰格拉斯哥),费尔滕和 斯坦福大学的Gordon Wetzstein (加利福尼亚州斯坦福)描述了实际使用所需的三个开发。4 Perhaps the most obvious need was to fill in the picture behind the front layer of the scene that scattered the most light back to the 中继墙. More scattering events could distribute light deeper into more hidden parts of the scene, but the strength of the return signal drops rapidly with the number of scatterings (see Fig. 2). Detecting such faint signals as well as brighter ones requires single-photon detectors that can be gated or have a high dynamic range.FIGURE 2. Return from the setup shown in Figure 1. Timing starts at the start of the laser pulse, but the detector ignores early photons reflected from the 中继墙. The first returns recorded are from light that bounces back from the desk and table in the front of the room. Weaker signals are returned later after multiple bounces from the “bad guy” in the back of the room.FIGURE 2. Return from the setup shown in Figure 1. Timing starts at the start of the laser pulse, but the detector ignores early photons reflected from the 中继墙. The first returns recorded are from light that bounces back from the desk and table in the front of the room. Weaker signals are returned later after multiple bounces from the “bad guy” in the back of the room.



本文回顾了许多最新研究,并得出结论,用于NLOS成像的最有前途的方法是将超短脉冲光源与单光子探测器结合在一起的时间分辨系统。首选光源是聚焦在中继墙上的超短激光脉冲,因此光会散射到隐藏区域中。光以100 ps的速度传播约3厘米,因此脉冲持续时间必须不再为获得合理的时空分辨率。 100 fs的脉冲会将时间冻结到空间中较短的30μm切片中。激光束扫描中继壁,以使来自脉冲的散射光从中继壁上的点辐射出去,以照亮隐藏区域。一些散射光从其他壁或隐藏对象的一部分反射到隐藏区域中的其他点,因此在原始散射脉冲的几个光子返回中继壁之前,光可能需要多次反射,然后可以由探测器记录下来。或相机。图3将实际场景与从返程时间计算出的图像进行了比较。图3.测试场景(a)与根据返回值(b)计算的信号的比较;数据是通过许多脉冲建立的。图3.测试场景(a)与根据返回值(b)计算的信号的比较;数据是通过许多脉冲建立的。(由安德烈亚斯·韦尔滕提供)

灵敏的高速检测器或摄像机对于时间和空间的高分辨率也是必不可少的。单光子雪崩二极管(SPAD)已用于许多NLOS成像实验。5 SPAD是在高偏置电压下工作的雪崩光电二极管,因此在盖革模式下检测单个光子会产生雪崩的载流子和高信号。该检测系统计算从发射激光脉冲到SPAD检测到单光子返回的时间,并将光子的到达记录在直方图上进行分析。为了进行精确测量,检测器在检测到光子后会关闭数十或数百纳秒,从而有时间记录多次反射的返回值。 SPAD最多可以检测40%的入射光子。

在商业上, SPAD可用作单元素和阵列检测器,两者均可用于NLOS成像。当前,大多数SPAD阵列都是为激光雷达开发的,这些设计需要修改以在NLOS成像中获得最佳性能。需要改进的地方包括更好的时间分辨率,更高的填充因子,选通直射光以及光子时标的更大灵活性。


The key information collected on the hidden scene is time-of-flight of multiply reflected photons relative to the laser pulse. Scattered photons that bounce only once, directly back to the 中继墙, constitute most of the raw return signal, but are screened out by not starting detection until after they have reached the detector. The detector waits to start collecting data until only multiple reflected photons remain in flight to the sensor.

进一步的处理更具挑战性,因为它需要从观察到的散射光向后进行操作,以在隐藏的场景中创建散射光的对象的图像。正在开发各种方法。有些是启发式方法,可以进行近似测试,并在准确度与响应速度之间进行权衡。其他方法包括反投影模型,线性逆方法,具有非线性元素的逆光传输,使用波光学而不是几何光学的模型,对象跟踪以及基于训练中收集的数据的神经网络方法。 Faccio等。4 包含更多详细信息和参考。


在SPIE安全+国防数字论坛(2020年9月20日至25日)期间,Velten在邀请的演讲中总结了最近的结果。6 “We can computationally recreate any imaging system your heart desires” by using a 中继墙 as the aperture of a virtual camera, he said. His group illuminated one point on the 中继墙, and then scanned the SPAD detector across the rest of the 中继墙 to collect the response from the hidden zone. The researchers used a phasor phase technique to combine that data with a virtual illumination wave to get the returned phase, and used Rayleigh-Sommerfeld diffraction operators to compute the image by back-propagating in space and time.7

频率-波长(-)迁移被认为比NLOS成像重建数据的速度快于相量反投影,尽管不如相量方法灵活。8 Velten说,他的相量方法现在已经赶上了速度,只需要2.8 s就可以在MATLAB中计算快速傅里叶变换。此外,他说,它需要更少的存储器,对噪声更鲁棒,并且不需要共聚焦数据,从而允许他们使用SPAD阵列而不是单像素检测器。

A video showed a thin sheet of light scattered from the 中继墙 propagating through the hidden zone, illuminating contents of a model office. The incoming light washed back over a chair in front of the room, reached the back wall, and then washed forward again over other objects in the office on its fourth and fifth bounces.

Velten还描述了另一项进步。大多数NLOS成像都基于平坦的静态Lambertian曲面,并且仅基于隐藏场景中的静止物体。现在,由他在威斯康星州同事Marco La Manna领导的一个项目表明,中继表面不必是平坦的或固定的。他们设计了一个带有两个单独的探测器的NLOS系统,该探测器收集飞行时间数据。一个监测动态继电器表面的表面;另一个监测动态继电器表面的表面。其他监视光从隐藏场景散射到中继表面上。他们的动态表面很简单一对白色的窗帘挂在框架上,并用风扇吹动以使它们运动。当监视移动屏幕的系统打开并提供位置数据时,他们可以使用移动屏幕散射的光对隐藏的静止对象成像。9



Velten对进步感到乐观。他说:“我认为我们掌握了所有理论,可以从大约一公里的较大对峙距离重建和捕获视频速率的NLOS图像。” “所有组件都已演示或将很快展示。此后,需要大量的工程和系统开发。根据兴趣和资金的不同,这可能只是一两年的问题。”

所需的关键硬件进步是专用于NLOS系统而不是激光雷达的SPAD阵列的设计。重建图像所需的时间将随着阵列中SPAD像素数量的增加而减少。 Velten说,还需要紧凑的皮秒激光。 “我们可以在高端硬件上进行实时重建,但是能够在更低端的系统上进行优化将是不错的选择。在目前的状态下,激光器大约占成本和功耗的90%。”



1. A. Kirmani,T。Hutchison,J。Davis和R. Raskar, 进程IEEE国际Conf。计算可见2009年,159–166(2009)。

2. A. Velten等, 纳特社区 3,745(2012)。

3. See //bit.ly/NLOSRef3.

4. D. Faccio,A。Velten和G. Wetzstein, 纳特物理牧师,2,318(2020年6月); //doi.org/10.1038/s42254-020-0174-8.

5. See //bit.ly/NLOSRef5.

6. A. Velten, 进程SPIE,11540,115400T(2020年9月20日); doi:10.1117 / 12.2574737。

7. X. Liu,S。Bauer和A. Velten, 纳特社区 11,1645(2020); //doi.org/10.1038/s41467-020-15157-4.

8.劳伦兹(M. Laurenzis), 进程SPIE,11540、115400U(2020年9月20日); doi:10.1117 / 12.2574198。

9. M. La Manna等, 选择。表现,28,4,53315339(2020年2月17日); //doi.org/10.1364/oe.383586.