本文复现了CVPR 2020的GSM视频分类模型，用Paddle2.0在Diving48数据集实现。该模型通过分解3D卷积核实现时空域视频理解，参数量小。项目处理了数据集，将视频转帧、生成训练测试文件，经训练测试，最佳模型在测试集准确率达0.223。
☞☞☞AI 智能聊天, 问答助手, AI 智能搜索, 免费无限量使用 DeepSeek R1 模型☜☜☜
项目背景
GSM是2020年CVPR上的一篇论文Gate-Shift Networks for Video Action Recognition中提出的视频分类模型。本项目即对其进行复现。
项目简介
本项目使用paddle2.0实现了视频动作分类GSM模型，并在Diving48数据集上进行了训练和验证。
Diving48数据集的划分是按数据集官方网站的划分方法进行划分的。
模型简介
Gate-Shift Networks for Video Action Recognition (GSM) 的核心是对进行时空域视频理解的3D卷积核进行分解的一种方法。该方法和已有方法的比较如下图所示。从图中可以看出，该模型跟其它网络模块相比具有参数量小的优点。
图1 GSM和其它网络结构比较图
模型细节
模型实现细节如下图所示。
图2 GSM实现细节图
GSM模块可以加在已有网络的不同部分，论文作者通过实验得出，将GSM模块加在下图所示的网络分支示例中具有较好的效果，即卷积核尺寸较小的分支中。
图3 GSM的plug-and-play示意图
论文原文：GSM: Gate-Shift Networks for Video Action Recognition
数据集介绍
本项目使用Diving48数据集进行训练和测试. 数据集官方网站为Diving48数据集
Diving48是跳水竞赛不同动作的细粒度视频数据集。该数据集包含48类不同的跳水标准动作，共18k左右个视频片段。由于不同的跳水动作可能在起跳、腾空和入水的三个不同阶段中的任一个阶段发生不同，需要模型学习视频中的长时间段间的特征关系。动作示例如下图所示：
图4 Diving48动作示意图
这48个类别中的每一个都是由三个不同的动作构成的，分别是起跳动作 (dive groups), 空中动作(somersaults and/or twists), 入水动作 (dive positions)。 下图展示了这48个类别的不同动作组成与互相间的关系：
图5 Diving48中不同动作间关系示意图
文件结构
| 文件名或文件夹名 | 功能 |
|---|---|
| model | 模型 |
| best_model | 训练好的模型 |
| result | 训练过程中的数据 |
| dataset | 数据文件夹 |
| checkpoints_models | 模型文件 |
| pretrained_model | 预训练模型 |
| extract_frames_diving48.py | 从视频中抽帧 |
| process_dataset_diving.py | 生成训练与测试的数据文件 |
| dataset.py | 自定义数据 |
| datasets_video.py | 读取数据 |
| transforms.py | 数据增强 |
| train.py | 模型训练 |
| test.py | 模型测试 |
模型实现及训练、验证
解压视频数据集
In [ ]
import osif not os.path.exists('dataset/Diving48'):
os.makedirs('dataset/Diving48')
In [ ]
!tar -xzf data/data56350/Diving48_rgb.tar.gz -C dataset/Diving48
创建标签文件
仅在预测时使用
In [ ]
import jsonimport numpy as np
label_dir = {}
i = 0with open('data/data56350/Diving48_vocab.json', 'r') as f:
vocab_list = json.load(f)
for vocab in vocab_list:
label_dir['_'.join(vocab)] = i
i += 1np.save('work/label_dir.npy', label_dir)print(label_dir)
{'Back_15som_05Twis_FREE': 0, 'Back_15som_15Twis_FREE': 1, 'Back_15som_25Twis_FREE': 2, 'Back_15som_NoTwis_PIKE': 3, 'Back_15som_NoTwis_TUCK': 4, 'Back_25som_15Twis_PIKE': 5, 'Back_25som_25Twis_PIKE': 6, 'Back_25som_NoTwis_PIKE': 7, 'Back_25som_NoTwis_TUCK': 8, 'Back_2som_15Twis_FREE': 9, 'Back_2som_25Twis_FREE': 10, 'Back_35som_NoTwis_PIKE': 11, 'Back_35som_NoTwis_TUCK': 12, 'Back_3som_NoTwis_PIKE': 13, 'Back_3som_NoTwis_TUCK': 14, 'Back_Dive_NoTwis_PIKE': 15, 'Back_Dive_NoTwis_TUCK': 16, 'Forward_15som_1Twis_FREE': 17, 'Forward_15som_2Twis_FREE': 18, 'Forward_15som_NoTwis_PIKE': 19, 'Forward_1som_NoTwis_PIKE': 20, 'Forward_25som_1Twis_PIKE': 21, 'Forward_25som_2Twis_PIKE': 22, 'Forward_25som_3Twis_PIKE': 23, 'Forward_25som_NoTwis_PIKE': 24, 'Forward_25som_NoTwis_TUCK': 25, 'Forward_35som_NoTwis_PIKE': 26, 'Forward_35som_NoTwis_TUCK': 27, 'Forward_45som_NoTwis_TUCK': 28, 'Forward_Dive_NoTwis_PIKE': 29, 'Forward_Dive_NoTwis_STR': 30, 'Inward_15som_NoTwis_PIKE': 31, 'Inward_15som_NoTwis_TUCK': 32, 'Inward_25som_NoTwis_PIKE': 33, 'Inward_25som_NoTwis_TUCK': 34, 'Inward_35som_NoTwis_TUCK': 35, 'Inward_Dive_NoTwis_PIKE': 36, 'Reverse_15som_05Twis_FREE': 37, 'Reverse_15som_15Twis_FREE': 38, 'Reverse_15som_25Twis_FREE': 39, 'Reverse_15som_35Twis_FREE': 40, 'Reverse_15som_NoTwis_PIKE': 41, 'Reverse_25som_15Twis_PIKE': 42, 'Reverse_25som_NoTwis_PIKE': 43, 'Reverse_25som_NoTwis_TUCK': 44, 'Reverse_35som_NoTwis_TUCK': 45, 'Reverse_Dive_NoTwis_PIKE': 46, 'Reverse_Dive_NoTwis_TUCK': 47}
视频转为jpg格式图片
视频转换需要较长时间，不低于1个小时，建议转换到非data文件夹以便后续启动项目后可以直接使用。
In [10]
!python work/extract_frames_diving48.py
ffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
libavutil 54. 31.100 / 54. 31.100
libavcodec 56. 60.100 / 56. 60.100
libavformat 56. 40.101 / 56. 40.101
libavdevice 56. 4.100 / 56. 4.100
libavfilter 5. 40.101 / 5. 40.101
libavresample 2. 1. 0 / 2. 1. 0
libswscale 3. 1.101 / 3. 1.101
libswresample 1. 2.101 / 1. 2.101
libpostproc 53. 3.100 / 53. 3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dataset/Diving48/rgb/nOlRwoxsDJ0_00555.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf57.71.100
Duration: 00:00:03.18, start: 0.000000, bitrate: 838 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 4:3 DAR 16:9], 830 kb/s, 60 fps, 60 tbr, 15360 tbn, 120 tbc (default)
Metadata:
handler_name : VideoHandler[swscaler @ 0x2204380] deprecated pixel format used, make sure you did set range correctlyOutput #0, image2, to 'dataset/Diving48/frames/nOlRwoxsDJ0_00555/frames%05d.jpg':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf56.40.101
Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 250x250 [SAR 16:9 DAR 16:9], q=2-31, 200 kb/s, 60 fps, 60 tbn, 60 tbc (default)
Metadata:
handler_name : VideoHandler
encoder : Lavc56.60.100 mjpeg
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
frame= 191 fps=0.0 q=24.8 Lsize=N/A time=00:00:03.18 bitrate=N/A
video:703kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
ffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
libavutil 54. 31.100 / 54. 31.100
libavcodec 56. 60.100 / 56. 60.100
libavformat 56. 40.101 / 56. 40.101
libavdevice 56. 4.100 / 56. 4.100
libavfilter 5. 40.101 / 5. 40.101
libavresample 2. 1. 0 / 2. 1. 0
libswscale 3. 1.101 / 3. 1.101
libswresample 1. 2.101 / 1. 2.101
libpostproc 53. 3.100 / 53. 3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dataset/Diving48/rgb/OFxuiqI5G44_00406.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf57.71.100
Duration: 00:00:03.77, start: 0.000000, bitrate: 1028 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 4:3 DAR 16:9], 1021 kb/s, 60 fps, 60 tbr, 15360 tbn, 120 tbc (default)
Metadata:
handler_name : VideoHandler[swscaler @ 0x18f1400] deprecated pixel format used, make sure you did set range correctlyOutput #0, image2, to 'dataset/Diving48/frames/OFxuiqI5G44_00406/frames%05d.jpg':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf56.40.101
Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 250x250 [SAR 16:9 DAR 16:9], q=2-31, 200 kb/s, 60 fps, 60 tbn, 60 tbc (default)
Metadata:
handler_name : VideoHandler
encoder : Lavc56.60.100 mjpeg
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
frame= 226 fps=0.0 q=24.8 Lsize=N/A time=00:00:03.76 bitrate=N/A
video:862kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
ffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
libavutil 54. 31.100 / 54. 31.100
libavcodec 56. 60.100 / 56. 60.100
libavformat 56. 40.101 / 56. 40.101
libavdevice 56. 4.100 / 56. 4.100
libavfilter 5. 40.101 / 5. 40.101
libavresample 2. 1. 0 / 2. 1. 0
libswscale 3. 1.101 / 3. 1.101
libswresample 1. 2.101 / 1. 2.101
libpostproc 53. 3.100 / 53. 3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dataset/Diving48/rgb/LkluZoNfKu8_00121.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf57.71.100
Duration: 00:00:09.40, start: 0.000000, bitrate: 816 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 4:3 DAR 16:9], 814 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
Metadata:
handler_name : VideoHandler[swscaler @ 0x1b7d3c0] deprecated pixel format used, make sure you did set range correctlyOutput #0, image2, to 'dataset/Diving48/frames/LkluZoNfKu8_00121/frames%05d.jpg':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf56.40.101
Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 250x250 [SAR 16:9 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)
Metadata:
handler_name : VideoHandler
encoder : Lavc56.60.100 mjpeg
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
frame= 235 fps=0.0 q=24.8 Lsize=N/A time=00:00:09.40 bitrate=N/A
video:922kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
ffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
libavutil 54. 31.100 / 54. 31.100
libavcodec 56. 60.100 / 56. 60.100
libavformat 56. 40.101 / 56. 40.101
libavdevice 56. 4.100 / 56. 4.100
libavfilter 5. 40.101 / 5. 40.101
libavresample 2. 1. 0 / 2. 1. 0
libswscale 3. 1.101 / 3. 1.101
libswresample 1. 2.101 / 1. 2.101
libpostproc 53. 3.100 / 53. 3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dataset/Diving48/rgb/Q--70-ELqLw_00068.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf57.71.100
Duration: 00:00:06.56, start: 0.000000, bitrate: 774 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 4:3 DAR 16:9], 771 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
Metadata:
handler_name : VideoHandler[swscaler @ 0x2108060] deprecated pixel format used, make sure you did set range correctlyOutput #0, image2, to 'dataset/Diving48/frames/Q--70-ELqLw_00068/frames%05d.jpg':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf56.40.101
Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 250x250 [SAR 16:9 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)
Metadata:
handler_name : VideoHandler
encoder : Lavc56.60.100 mjpeg
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
frame= 164 fps=0.0 q=24.8 Lsize=N/A time=00:00:06.56 bitrate=N/A
video:606kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
ffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
libavutil 54. 31.100 / 54. 31.100
libavcodec 56. 60.100 / 56. 60.100
libavformat 56. 40.101 / 56. 40.101
libavdevice 56. 4.100 / 56. 4.100
libavfilter 5. 40.101 / 5. 40.101
libavresample 2. 1. 0 / 2. 1. 0
libswscale 3. 1.101 / 3. 1.101
libswresample 1. 2.101 / 1. 2.101
libpostproc 53. 3.100 / 53. 3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dataset/Diving48/rgb/_tigfCJFLZg_00205.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf57.71.100
Duration: 00:00:04.00, start: 0.000000, bitrate: 1545 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 4:3 DAR 16:9], 1541 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
Metadata:
handler_name : VideoHandler[swscaler @ 0x1d4e780] deprecated pixel format used, make sure you did set range correctlyOutput #0, image2, to 'dataset/Diving48/frames/_tigfCJFLZg_00205/frames%05d.jpg':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf56.40.101
Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 250x250 [SAR 16:9 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)
Metadata:
handler_name : VideoHandler
encoder : Lavc56.60.100 mjpeg
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
frame= 100 fps=0.0 q=24.8 Lsize=N/A time=00:00:04.00 bitrate=N/A
video:479kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
ffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
libavutil 54. 31.100 / 54. 31.100
libavcodec 56. 60.100 / 56. 60.100
libavformat 56. 40.101 / 56. 40.101
libavdevice 56. 4.100 / 56. 4.100
libavfilter 5. 40.101 / 5. 40.101
libavresample 2. 1. 0 / 2. 1. 0
libswscale 3. 1.101 / 3. 1.101
libswresample 1. 2.101 / 1. 2.101
libpostproc 53. 3.100 / 53. 3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dataset/Diving48/rgb/sk8TafuB3lU_00696.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf57.71.100
Duration: 00:00:01.27, start: 0.000000, bitrate: 1286 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 4:3 DAR 16:9], 1276 kb/s, 60 fps, 60 tbr, 15360 tbn, 120 tbc (default)
Metadata:
handler_name : VideoHandler[swscaler @ 0x1f20b00] deprecated pixel format used, make sure you did set range correctlyOutput #0, image2, to 'dataset/Diving48/frames/sk8TafuB3lU_00696/frames%05d.jpg':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf56.40.101
Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 250x250 [SAR 16:9 DAR 16:9], q=2-31, 200 kb/s, 60 fps, 60 tbn, 60 tbc (default)
Metadata:
handler_name : VideoHandler
encoder : Lavc56.60.100 mjpeg
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
frame= 76 fps=0.0 q=24.8 Lsize=N/A time=00:00:01.26 bitrate=N/A
video:199kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
ffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
libavutil 54. 31.100 / 54. 31.100
libavcodec 56. 60.100 / 56. 60.100
libavformat 56. 40.101 / 56. 40.101
libavdevice 56. 4.100 / 56. 4.100
libavfilter 5. 40.101 / 5. 40.101
libavresample 2. 1. 0 / 2. 1. 0
libswscale 3. 1.101 / 3. 1.101
libswresample 1. 2.101 / 1. 2.101
libpostproc 53. 3.100 / 53. 3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dataset/Diving48/rgb/xbQCwTHcGN8_00158.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf57.71.100
Duration: 00:00:03.56, start: 0.000000, bitrate: 1580 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 4:3 DAR 16:9], 1576 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
Metadata:
handler_name : VideoHandler[swscaler @ 0x1bbd3c0] deprecated pixel format used, make sure you did set range correctlyOutput #0, image2, to 'dataset/Diving48/frames/xbQCwTHcGN8_00158/frames%05d.jpg':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf56.40.101
Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 250x250 [SAR 16:9 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)
Metadata:
handler_name : VideoHandler
encoder : Lavc56.60.100 mjpeg
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
frame= 84 fps=0.0 q=24.8 Lsize=N/A time=00:00:03.36 bitrate=N/A
video:406kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Exiting normally, received signal 2.
ffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
libavutil 54. 31.100 / 54. 31.100
libavcodec 56. 60.100 / 56. 60.100
libavformat 56. 40.101 / 56. 40.101
libavdevice 56. 4.100 / 56. 4.100
libavfilter 5. 40.101 / 5. 40.101
libavresample 2. 1. 0 / 2. 1. 0
libswscale 3. 1.101 / 3. 1.101
libswresample 1. 2.101 / 1. 2.101
libpostproc 53. 3.100 / 53. 3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dataset/Diving48/rgb/l6ZoMjD1UtA_00156.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf57.71.100
Duration: 00:00:05.60, start: 0.000000, bitrate: 908 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 4:3 DAR 16:9], 905 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
Metadata:
handler_name : VideoHandler[swscaler @ 0x209c500] deprecated pixel format used, make sure you did set range correctlyOutput #0, image2, to 'dataset/Diving48/frames/l6ZoMjD1UtA_00156/frames%05d.jpg':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf56.40.101
Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 250x250 [SAR 16:9 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)
Metadata:
handler_name : VideoHandler
encoder : Lavc56.60.100 mjpeg
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
生成训练和测试的数据文本文件
In [ ]
!python work/process_dataset_diving.py
15026/15027
在之前训练基础上微调
In [ ]
!python work/train.py --epoch 5 --batch_size 32 --seg_num 16 --seg_len 1 --pretrain True --weights 'pretrained/inception_pretrained' \ --data_dir 'result/best_model.npz' --best_model_save 'best_model/best_model' --model_arc 'inception'
You are using Paddle compiled with TensorRT, but TensorRT dynamic library is not found. Ignore this if TensorRT is not needed. W0119 22:07:28.029369 15355 device_context.cc:320] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.1, Runtime API Version: 10.1 W0119 22:07:28.034546 15355 device_context.cc:330] device: 0, cuDNN Version: 7.6. video number:15027 video number:1970 /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/nn/layer/norm.py:634: UserWarning: When training, we now always track global mean and variance. "When training, we now always track global mean and variance.") Loss at epoch 0 step 0: [3.657299], acc: [0.25] Loss at epoch 0 step 200: [3.6588974], acc: [0.25] Loss at epoch 0 step 400: [3.6196995], acc: [0.28125] The best result is 0.22030456 Loss at epoch 1 step 0: [3.5962734], acc: [0.3125] Loss at epoch 1 step 200: [3.5641773], acc: [0.34375] Loss at epoch 1 step 400: [3.6840277], acc: [0.21875] The best result is 0.22081219 Loss at epoch 2 step 0: [3.4678974], acc: [0.4375] Loss at epoch 2 step 200: [3.4990504], acc: [0.40625] Loss at epoch 2 step 400: [3.622469], acc: [0.28125] The best result is 0.22335026 Loss at epoch 3 step 0: [3.6552052], acc: [0.25] Loss at epoch 3 step 200: [3.5325003], acc: [0.375] Loss at epoch 3 step 400: [3.6104255], acc: [0.28125] Loss at epoch 4 step 0: [3.6843047], acc: [0.21875] Loss at epoch 4 step 200: [3.6228292], acc: [0.28125] Loss at epoch 4 step 400: [3.7475114], acc: [0.15625] Final loss: [3.6923075] time duration is 5594.123642206192
图示训练和验证过程
In [ ]
## 画图函数import numpy as npimport matplotlib.pyplot as pltdef draw_train_eval_process(title,step,train_reward, eval_reward, label_train_reward, label_eval_reward, color_train, color_eval, label_y):
plt.title(title, fontsize=24)
plt.xlabel("epochs", fontsize=20)
plt.ylabel(label_y, fontsize=20)
plt.plot(step, train_reward,color=color_train,label=label_train_reward)
plt.plot(step, eval_reward,color=color_eval,label=label_eval_reward)
plt.legend()
plt.grid()
plt.show()
In [ ]
result_gsm = np.load('result/best_model.npz')
gsm_train_rewards = result_gsm['all_train_rewards']
gsm_eval_rewards = result_gsm['all_test_rewards']
steps = [i for i in range(len(gsm_train_rewards))]
draw_train_eval_process("GSM training and eval",steps,gsm_train_rewards,gsm_eval_rewards,
"GSM trainning accuracy", "GSM eval accuracy", "red","green", "gsm_train_eval_accuracy")
<Figure size 432x288 with 1 Axes>
用保存的最佳训练模型测试
In [ ]
!python work/test.py --weights 'best_model/best_model'
You are using Paddle compiled with TensorRT, but TensorRT dynamic library is not found. Ignore this if TensorRT is not needed. [INFO: test.py: 132]: Namespace(batch_size=1, dataset='diving48', infer_topk=1, modality='RGB', model_arc='inception', seg_len=1, seg_num=16, weights='best_model/best_model') W0120 00:08:33.052891 25454 device_context.cc:320] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.1, Runtime API Version: 10.1 W0120 00:08:33.058135 25454 device_context.cc:330] device: 0, cuDNN Version: 7.6. video number:1970 测试集准确率为:0.22335025668144226
