基于体素的3D目标检测网络：VoxelNet

开始

数据集解压

约15分钟.

%cd /home/aistudio/
!rm -rf kitti/
!mkdir -p kitti/training/velodyne_reduced
!mkdir -p kitti/testing/velodyne_reduced
!unzip data/data50186/data_object_calib.zip -d kitti/
!unzip data/data50186/image_training.zip -d kitti/training/
!unzip data/data50186/data_object_label_2.zip -d kitti/training/
!unzip data/data50186/velodyne_training_1.zip -d kitti/training/
!unzip data/data50186/velodyne_training_2.zip -d kitti//training/
!unzip data/data50186/velodyne_training_3.zip -d kitti/training/
!unzip data/data50186/image_testing.zip -d kitti/testing/
!unzip data/data50186/velodyne_testing_1.zip -d kitti/testing/
!unzip data/data50186/velodyne_testing_2.zip -d kitti/testing/
!unzip data/data50186/velodyne_testing_3.zip -d kitti/testing/
!mv kitti/training/training/* kitti/training/
!rm -rf kitti/training/training/
!mv kitti/testing/testing/* kitti/testing/
!rm -rf kitti/testing/testing/
!mkdir kitti/training/velodyne
!mv kitti/training/velodyne_training_1/* kitti/training/velodyne/
!mv kitti/training/velodyne_training_2/* kitti/training/velodyne/
!mv kitti/training/velodyne_training_3/* kitti/training/velodyne/
!rm -rf kitti/training/velodyne_training_1
!rm -rf kitti/training/velodyne_training_2
!rm -rf kitti/training/velodyne_training_3
!mkdir kitti/testing/velodyne
!mv kitti/testing/velodyne_testing_1/* kitti/testing/velodyne
!mv kitti/testing/velodyne_testing_2/* kitti/testing/velodyne
!mv kitti/testing/velodyne_testing_3/* kitti/testing/velodyne
!rm -rf kitti/testing/velodyne_testing_1
!rm -rf kitti/testing/velodyne_testing_2
!rm -rf kitti/testing/velodyne_testing_3
!mv kitti data/

至此，数据集的结构：

└── KITTI_DATASET_ROOT
       ├── training    <-- 7481 train data
       |   ├── image_2 <-- for visualization
       |   ├── calib
       |   ├── label_2
       |   ├── velodyne
       |   └── velodyne_reduced <-- empty directory
       └── testing     <-- 7580 test data
           ├── image_2 <-- for visualization
           ├── calib
           ├── velodyne
           └── velodyne_reduced <-- empty directory

3,712 data samples fortraining and 3,769 data samples for validation

安装依赖

最适合的环境配置：

• python版本：3.7.4
• PaddlePaddle框架版本：2.2.1
• CUDA 版本： NVIDIA-SMI 450.51.06 Driver Version: 450.51.06 CUDA Version: 11.0 cuDNN:7.6

注意： 由于PaddlePaddle/cuDNN本身的BUG，CUDA 10.1版本当batch size > 2时会报如下错误：

OSError: (External) CUDNN error(7), CUDNN_STATUS_MAPPING_ERROR. 
  [Hint: 'CUDNN_STATUS_MAPPING_ERROR'.  An access to GPU memory space failed, which is usually caused by a failure to bind a texture.  To correct, prior to the function call, unbind any previously bound textures.  Otherwise, this may indicate an internal error/bug in the library.  ] (at /paddle/paddle/fluid/operators/conv_cudnn_op.cu:758)

因此单卡如果环境不是CUDA 11.0以上，config文件中batch size设置为2即可，后续通过训练的accum_step参数开启梯度累加起到增大bs的效果。设置accum_step=8即表示bs=16，并做相应config文件的初始学习率调整。

!pip install distro shapely pybind11 pillow fire memory_profiler psutil scikit-image==0.14.2!pip install numpy==1.17.0!pip install numba==0.48.0

（由于Notebook不支持导入当前整个项目到Python环境，以下操作在终端命令行执行）

准备部分

1. 为numba设置cuda环境

export NUMBAPRO_CUDA_DRIVER=/usr/lib/x86_64-linux-gnu/libcuda.soexport NUMBAPRO_NVVM=/usr/local/cuda/nvvm/lib64/libnvvm.soexport NUMBAPRO_LIBDEVICE=/usr/local/cuda/nvvm/libdevice

2. 将当前项目加到环境中

export PYTHONPATH=$PYTHONPATH:/home/aistudio/VoxelNet

3. 数据预处理

从label中分类别抽取真值信息以及对点云进行降采样。（约7分钟）

cd /home/aistudio/VoxelNet/voxelnet/
python create_data.py create_kitti_info_file --data_path=/home/aistudio/data/kitti  #  Create kitti infospython create_data.py create_reduced_point_cloud --data_path=/home/aistudio/data/kitti # Create kitti reduced pointpython create_data.py create_groundtruth_database --data_path=/home/aistudio/data/kitti # Create kitti gt

打印信息如下：

Generate info. this may take several minutes.
Kitti info train file is saved to /home/aistudio/data/kitti/kitti_infos_train.pkl
Kitti info val file is saved to /home/aistudio/data/kitti/kitti_infos_val.pkl
Kitti info trainval file is saved to /home/aistudio/data/kitti/kitti_infos_trainval.pkl
Kitti info test file is saved to /home/aistudio/data/kitti/kitti_infos_test.pkl
[100.0%][===================>][40.86it/s][01:44>00:00]   
[100.0%][===================>][35.31it/s][01:47>00:00]   
[100.0%][===================>][39.13it/s][03:49>00:00] 
[100.0%][===================>][28.71it/s][01:53>00:00]     
load 14357 Car database infosload 2207 Pedestrian database infosload 734 Cyclist database infosload 1297 Van database infosload 56 Person_sitting database infosload 488 Truck database infosload 224 Tram database infosload 337 Misc database infos

4. 修改配置文件

voxelnet/configs/car.configs

train_input_reader: {
  ...
  database_sampler {
    database_info_path: "/home/aistudio/data/kitti/kitti_dbinfos_train.pkl"
    ...
  }
  kitti_info_path: "/home/aistudio/data/kitti/kitti_infos_train.pkl"
  kitti_root_path: "/home/aistudio/data/kitti"}
...
eval_input_reader: {
  ...
  kitti_info_path: "/home/aistudio/data/kitti/kitti_infos_val.pkl"
  kitti_root_path: "/home/aistudio/data/kitti"}

设置注意事项：

1、若训练要开启梯度累加选项，则：

• 学习率的decay_steps按照梯度累加后的batch size对应的总steps来设置。
• train_config.steps则按未梯度累加时对应的初始batch size对应的总steps来设置

2、 配置文件需放置于voxelnet/configs/***.py

快速开始

1. 训练

训练一个epoch, V100 16G大约15分钟。显存占用11G左右。

python ./pypaddle/train.py train --config_path=./configs/config.py --model_dir=./output

2. 评估

V100 16G 大约5分钟

python ./pypaddle/train.py evaluate --config_path=./configs/config.py --model_dir=./output --ckpt_path=./output/voxelnet-278400.ckpt

3. 可视化预测

3D可视化需要GUI，Notebook环境不支持动态GUI调用显示。需在本地测试。 详细查看README.md。

4. 一个简单的BEV视角可视化例子

为了方便查看预测结果，下面的cell提供了一个在notebook中查看二维bev视角的可视化例子，可以在notebook执行。 由于只保留了相机视角范围内的结果(Points that are projectedoutside of image boundaries are removed(in Paper Section 3.1))，所以车身后面没有检测框。

%cd /home/aistudio/VoxelNet/
!export PYTHONPATH=$PYTHONPATH:/home/aistudio/VoxelNetimport paddleimport numpy as npimport matplotlib.pyplot as pltimport picklefrom pathlib import Pathimport voxelnet.pypaddle.builder.voxelnet_builder as voxelnet_builderimport voxelnet.builder.voxel_builder as voxel_builderimport voxelnet.builder.target_assigner_builder as target_assigner_builderimport voxelnet.pypaddle.builder.box_coder_builder as box_coder_builderfrom voxelnet.data.preprocess import merge_voxelnet_batchfrom voxelnet.configs import cfg_from_config_py_filefrom voxelnet.utils import visdef example_convert_to_paddle(example, dtype=paddle.float32,                             ) -> dict:
    example_paddle = {}
    float_names = [        "voxels", "anchors", "reg_targets", "reg_weights", "bev_map", "rect",        "Trv2c", "P2"
    ]    for k, v in example.items():        if k in float_names:
            example_paddle[k] = paddle.to_tensor(v, dtype=dtype)        elif k in ["coordinates", "labels", "num_points"]:
            example_paddle[k] = paddle.to_tensor(
                v, dtype=paddle.int32)        elif k in ["anchors_mask"]:
            example_paddle[k] = paddle.to_tensor(
                v, dtype=paddle.uint8)        else:
            example_paddle[k] = v    return example_paddle

paddle.set_device('gpu') # 设置cpu/gpuconfig_path = "home/aistudio/VoxelNet/voxelnet/configs/config.py"config = cfg_from_config_py_file(config_path)
input_cfg = config.eval_input_reader
model_cfg = config.model.voxelnet

ckpt_path = "/home/aistudio/VoxelNet/voxelnet/output/voxelnet-278400.ckpt"model_cfg.voxel_generator.point_cloud_range = [0, -40, -3, 70.4, 40, 1]
voxel_generator = voxel_builder.build(model_cfg.voxel_generator)####################### BUILD TARGET ASSIGNER######################bv_range = voxel_generator.point_cloud_range[[0, 1, 3, 4]]
box_coder = box_coder_builder.build(model_cfg.box_coder)
target_assigner_cfg = model_cfg.target_assigner
target_assigner = target_assigner_builder.build(target_assigner_cfg,
                                                bv_range, box_coder)
net = voxelnet_builder.build(model_cfg, voxel_generator, target_assigner)
net.eval()

state = paddle.load(ckpt_path)
net.set_state_dict(state)

out_size_factor = model_cfg.rpn.layer_strides[0] // model_cfg.rpn.upsample_strides[0]
grid_size = voxel_generator.grid_size
feature_map_size = grid_size[:2] // out_size_factor
feature_map_size = [*feature_map_size, 1][::-1]

anchors = target_assigner.generate_anchors(feature_map_size)["anchors"]
anchors = anchors.reshape((1, -1, 7))

info_path = input_cfg.kitti_info_path
root_path = Path(input_cfg.kitti_root_path)with open(info_path, 'rb') as f:
    infos = pickle.load(f)

info = infos[564] # 测试目标点云# print(info)v_path = info['velodyne_path']
v_path = str(root_path / v_path)
points = np.fromfile(
    v_path, dtype=np.float32, count=-1).reshape([-1, 4])
voxels, coords, num_points = voxel_generator.generate(points, max_voxels=40000)print(voxels.shape)# add batch idx to coords# coords = np.pad(coords, ((0, 0), (1, 0)), mode='constant', constant_values=0)image_idx = info['image_idx']
rect = info['calib/R0_rect'].astype(np.float32)
Trv2c = info['calib/Tr_velo_to_cam'].astype(np.float32)
P2 = info['calib/P2'].astype(np.float32)

example = {    "anchors": anchors,    "voxels": voxels,    "num_points": num_points,    "num_voxels": np.array([voxels.shape[0]], dtype=np.int64),    "coordinates": coords,    "rect": rect,    "P2": P2,    "Trv2c":Trv2c,    'image_idx': image_idx

}
batch_example = [example]
examples = merge_voxelnet_batch(batch_example)
examples = example_convert_to_paddle(examples)with paddle.no_grad():
    pred = net(examples)[0]

boxes_lidar = pred["box3d_lidar"].detach().cpu().numpy()
vis_voxel_size = [0.1, 0.1, 0.1]
vis_point_range = [-50, -30, -3, 50, 30, 1]
bev_map = vis.point_to_vis_bev(points, vis_voxel_size, vis_point_range)
bev_map = vis.draw_box_in_bev(bev_map, vis_point_range, boxes_lidar, [0, 255, 0], 2)# plt.savefig('/home/aistudio/val564.png')plt.imshow(bev_map)
paddle.device.cuda.empty_cache()

/home/aistudio/VoxelNet
cfg_file must be located in ./configs/***.py...
load config from home/aistudio/VoxelNet/voxelnet/configs/config.py...
(13282, 35, 4)

结语

复现心得：

这篇论文在第四届论文复现赛的时候我就进行了复现，并未成功，相差甚远。由于论文没有开源代码，网络上也没有达到论文精度的项目，很难从头开始自己全部重写。于是我换了一个思路，既然这篇论文是非常经典的论文，后续肯定有人基于这个思路进行改进。果然找到了second。second这篇论文几乎重现了voxelnet的所有方法（但据作者说并没有完全复现原始的voxelnet），不过加入了稀疏卷积使得速度和精度得到了巨大提升。于是我的思路就变成了从second中去掉它改进的部分内容，使其复原原始的voxelnet，来减少自己重写代码的工作量。

由于之前第四届比赛时的经验，这次遇到的问题都不多，或者说当时已经遇到了，直接拿我当时的代码进行替换，一个函数一个函数输入输出对齐测试即可。并且只需要对照X2Paddle进行对齐即可。

第四届和这一次遇到的主要问题都是内存泄漏问题。不过这一次，找到了问题所在。

1.这里总结一下几个常犯的内存泄漏错误原因。

• loss在用于取值打印或者用于其他计算时，没有.detach()或者.numpy()
• 如果model本身内部要存一些每一次前向计算后的指标，指标一定也要在前向计算后detach再赋值到model内的变量（这是我找了超级久的泄漏问题）。

2.enisum函数。这个函数虽然paddle2.2提供了，但是还有bug。用的时候报错了。

我在paddlenlp（https://github.com/PaddlePaddle/PaddleNLP/blob/develop/paddlenlp/ops/einsum.py） 中找到了可以正常使用的版本。

3.mask赋值。问题如下: 写了一个mask（bool类型）的赋值函数：

def mask_slice_v1(data, mask):
    """
    问题：
    data.shape = [x,y], type: float...
    mask.shape = [x] ,type: bool
    in torch, can do it by data[mask] to get shape:[x,y] result.
    但是padlde目前还不行。
    """
    data_shape = data.shape
    mask = mask.unsqueeze(-1) # [x,1]
    mask = paddle.tile(mask,[1,data_shape[1]]) # [x,y]
    slice = paddle.masked_select(data,mask) # [x*y]
    return slice.reshape([-1,data_shape[1]]) # [x,y]

相关信息:

引用

• Thanks for yan yan's second.pytorch project.

@inproceedings{Yin2018voxelnet,
    author={Yin Zhou, Oncel Tuzel},
    title={VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection},
    booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    year = {2018}
}