lidargaitv2 open-source
This commit is contained in:
Noah
@@ -13,6 +13,7 @@ The corresponding [paper](https://openaccess.thecvf.com/content/CVPR2023/papers/
|
||||
The extension [paper](https://arxiv.org/pdf/2405.09138) has been accepted to TPAMI2025.
|
||||
|
||||
## What's New
|
||||
- **[Jun 2025]** [LidarGait++](https://openaccess.thecvf.com/content/CVPR2025/papers/Shen_LidarGait_Learning_Local_Features_and_Size_Awareness_from_LiDAR_Point_CVPR_2025_paper.pdf) has been accepted to CVPR2025🎉 and open-source in [configs/lidargaitv2](./configs/lidargaitv2/README.md).
|
||||
- **[Jun 2025]** The extension paper of [OpenGait](https://arxiv.org/pdf/2405.09138), further strengthened by the advancements of [DeepGaitV2](https://github.com/ShiqiYu/OpenGait/blob/master/opengait/modeling/models/deepgaitv2.py), SkeletonGait, and [SkeletonGait++](opengait/modeling/models/skeletongait%2B%2B.py), has been accepted for publication in TPAMI🎉. We sincerely acknowledge the valuable contributions and continuous support from the OpenGait community.
|
||||
- **[Feb 2025]** The diffusion-based [DenoisingGait](https://arxiv.org/pdf/2505.18582) has been accepted to CVPR2025🎉 Congratulations to [Dongyang](https://scholar.google.com.hk/citations?user=1xA5KxAAAAAJ)! This is his SECOND paper!
|
||||
- **[Feb 2025]** Chao successfully defended his Ph.D. thesis in Oct. 2024🎉🎉🎉 You can access the full text in [*Chao's Thesis in English*](https://www.researchgate.net/publication/388768400_Gait_Representation_Learning_and_Recognition?_sg%5B0%5D=qaGVpS8gKWPyR7olHoFd4bCs40AZdJzaM96P3TSnxrpiP9zCIUTxzeEq8YhQOlE4WemB7iMF2fHvcJFAYHTlJhTIB2J6faVa5s-xcQVj.4112nauMM4MWUNSyUa9eMeF0MEeplptpFOgb5kSgIk3lMcfPK6TdPX1bW1y_bKSdbwXuBf29GloRsVwBdexhug&_tp=eyJjb250ZXh0Ijp7ImZpcnN0UGFnZSI6ImhvbWUiLCJwYWdlIjoicHJvZmlsZSIsInByZXZpb3VzUGFnZSI6InByb2ZpbGUiLCJwb3NpdGlvbiI6InBhZ2VDb250ZW50In19) or [*樊超的学位论文（中文版）*](https://www.researchgate.net/publication/388768605_butaitezhengxuexiyushibiesuanfayanjiu).
|
||||
@@ -39,6 +40,7 @@ Our team's latest checkpoints for projects such as DeepGaitv2, SkeletonGait, Ske
|
||||
- [Mar 2022] Dataset [GREW](https://www.grew-benchmark.org) is supported in [datasets/GREW](./datasets/GREW). -->
|
||||
|
||||
## Our Works
|
||||
- [**CVPR'25**] LidarGait++: Learning Local Features and Size Awareness from LiDAR Point Clouds for 3D Gait Recognition. [*Paper*](https://openaccess.thecvf.com/content/CVPR2025/papers/Shen_LidarGait_Learning_Local_Features_and_Size_Awareness_from_LiDAR_Point_CVPR_2025_paper.pdf) and [*LidarGait++ Code*](configs/lidargaitv2/README.md)
|
||||
- [**TPAMI'25**] OpenGait: A Comprehensive Benchmark Study for Gait Recognition Towards Better Practicality. [*Paper*](https://arxiv.org/pdf/2405.09138). _This extension includes a key update with in-depth insights into emerging trends and challenges of gait recognition in Sec. VII_.
|
||||
- [**CVPR'25**] On Denoising Walking Videos for Gait Recognition. [*Paper*](https://arxiv.org/pdf/2505.18582) and [*DenoisingGait Code* (coming soon)]
|
||||
- [**Chao's Thesis**] Gait Representation Learning and Recognition, [Chinese Original](https://www.researchgate.net/publication/388768605_butaitezhengxuexiyushibiesuanfayanjiu) and [English Translation](https://www.academia.edu/127496287/Gait_Representation_Learning_and_Recognition).
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
# LidarGait++: Learning Local Features and Size Awareness from LiDAR Point Clouds for 3D Gait Recognition
|
||||
|
||||
This [paper](https://openaccess.thecvf.com/content/CVPR2025/papers/Shen_LidarGait_Learning_Local_Features_and_Size_Awareness_from_LiDAR_Point_CVPR_2025_paper.pdf) has been accepted by CVPR 2025.
|
||||
|
||||
|
||||
|
||||
## Prepare dataset
|
||||
**SUSTech1K**:
|
||||
- Step 1. Apply for [SUSTech1K](https://lidargait.github.io/).
|
||||
|
||||
**FreeGait** (Optional):
|
||||
|
||||
- Step 1. Download [FreeGait](https://drive.google.com/drive/folders/1I9zOCmqUuBUcOmvO1cgZtUC6uSfmAq7h) first.
|
||||
|
||||
- Then rearrange the folder structure like SUSTech1K/CASIA-B to fit OpenGait framework.
|
||||
```
|
||||
python datasets/FreeGait/rearrange_freegait.py --input_path yout_freegait_path
|
||||
```
|
||||
|
||||
## Train
|
||||
To train on SUSTech1K, run
|
||||
```
|
||||
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python -m torch.distributed.launch --nproc_per_node=4 opengait/main.py --cfgs ./configs/lidargaitv2/lidargaitv2_sustech1k.yaml --phase train
|
||||
```
|
||||
or train on FreeGait, run
|
||||
```
|
||||
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python -m torch.distributed.launch --nproc_per_node=4 opengait/main.py --cfgs ./configs/lidargaitv2/lidargaitv2_freegait.yaml --phase train
|
||||
```
|
||||
|
||||
## Citation
|
||||
|
||||
```bibtex
|
||||
@inproceedings{shen2023lidargait,
|
||||
title={Lidargait: Benchmarking 3d gait recognition with point clouds},
|
||||
author={Shen, Chuanfu and Fan, Chao and Wu, Wei and Wang, Rui and Huang, George Q and Yu, Shiqi},
|
||||
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
|
||||
pages={1054--1063},
|
||||
year={2023}
|
||||
}
|
||||
|
||||
@inproceedings{shen2025lidargait++,
|
||||
title={LidarGait++: Learning Local Features and Size Awareness from LiDAR Point Clouds for 3D Gait Recognition},
|
||||
author={Shen, Chuanfu and Wang, Rui and Duan, Lixin and Yu, Shiqi},
|
||||
booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference},
|
||||
pages={6627--6636},
|
||||
year={2025}
|
||||
}
|
||||
```
|
||||
@@ -0,0 +1,111 @@
|
||||
data_cfg:
|
||||
dataset_name: FreeGait
|
||||
dataset_root: your_path
|
||||
dataset_partition: ./datasets/FreeGait/FreeGait.json
|
||||
num_workers: 1
|
||||
data_in_use: [false, false, true, false, false]
|
||||
remove_no_gallery: false # Remove probe if no gallery for it
|
||||
test_dataset_name: FreeGait
|
||||
|
||||
|
||||
evaluator_cfg:
|
||||
enable_float16: false #true #false #true
|
||||
restore_ckpt_strict: true
|
||||
restore_hint: 80000
|
||||
save_name: lidargaitv2
|
||||
eval_func: evaluate_FreeGait
|
||||
sampler:
|
||||
batch_shuffle: false
|
||||
points_in_use: # For point-based gait recognition using point clouds
|
||||
pointcloud_index: 0
|
||||
points_num: 256 #312 #256 #2048
|
||||
batch_size: 8
|
||||
frames_num_fixed: 10 # fixed frames number for training
|
||||
frames_skip_num: 0
|
||||
#sample_type: fixed_ordered
|
||||
sample_type: all_ordered # all indicates whole sequence used to test, while ordered means input sequence by its natural order; Other options: fixed_unordered
|
||||
frames_all_limit: 720 # limit the number of sampled frames to prevent out of memory
|
||||
metric: euc # cos
|
||||
transform:
|
||||
- type: PointCloudsTransform
|
||||
xyz_only: true
|
||||
scale_aware: true
|
||||
|
||||
loss_cfg:
|
||||
- loss_term_weight: 1.0
|
||||
margin: 0.2
|
||||
type: TripletLoss
|
||||
log_prefix: triplet
|
||||
lazy: False
|
||||
- loss_term_weight: 0.1
|
||||
scale: 31 #16
|
||||
type: CrossEntropyLoss
|
||||
log_prefix: softmax
|
||||
log_accuracy: true
|
||||
|
||||
|
||||
model_cfg:
|
||||
model: LidarGaitPlusPlus
|
||||
pool: PPP_HAP
|
||||
sampling: knn
|
||||
channel: 32
|
||||
npoints: [256, 192, 128]
|
||||
nsample: 16
|
||||
scale_aware: true
|
||||
normalize_dp: true
|
||||
SeparateFCs:
|
||||
in_channels: 256
|
||||
out_channels: 256
|
||||
parts_num: 31
|
||||
SeparateBNNecks:
|
||||
class_num: 500
|
||||
in_channels: 256
|
||||
parts_num: 31
|
||||
scale:
|
||||
- 1
|
||||
- 2
|
||||
- 4
|
||||
- 8
|
||||
- 16
|
||||
|
||||
|
||||
optimizer_cfg:
|
||||
lr: 0.1
|
||||
momentum: 0.9
|
||||
solver: SGD
|
||||
weight_decay: 0.0005
|
||||
|
||||
scheduler_cfg:
|
||||
T_max: 80000
|
||||
eta_min: 0.0001
|
||||
scheduler: CosineAnnealingLR
|
||||
|
||||
|
||||
trainer_cfg:
|
||||
enable_float16: false #true # half_percesion float for memory reduction and speedup
|
||||
fix_BN: false
|
||||
with_test: true
|
||||
log_iter: 100
|
||||
restore_ckpt_strict: true
|
||||
restore_hint: 0
|
||||
save_iter: 5000
|
||||
save_name: lidargaitv2
|
||||
sync_BN: true
|
||||
total_iter: 80000
|
||||
sampler:
|
||||
batch_shuffle: true
|
||||
batch_size:
|
||||
- 32 # TripletSampler, batch_size[0] indicates Number of Identity
|
||||
- 4 # batch_size[1] indicates Samples sequqnce for each Identity
|
||||
frames_num_fixed: 10 # fixed frames number for training
|
||||
sample_type: fixed_unordered # fixed control input frames number, unordered for controlling order of input tensor; Other options: unfixed_ordered or all_ordered
|
||||
type: TripletSampler
|
||||
points_in_use:
|
||||
pointcloud_index: 0
|
||||
points_num: 256
|
||||
transform:
|
||||
- type: PointCloudsTransform
|
||||
xyz_only: true
|
||||
scale_aware: true
|
||||
scale_prob: 0.2
|
||||
flip_prob: 0.1
|
||||
@@ -0,0 +1,110 @@
|
||||
data_cfg:
|
||||
dataset_name: SUSTech1K
|
||||
dataset_root: your_path # download from https://lidargait.github.io/ if you don't have dataset
|
||||
dataset_partition: ./datasets/SUSTech1K/SUSTech1K.json
|
||||
num_workers: 4
|
||||
data_in_use: [true, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false]
|
||||
remove_no_gallery: false # Remove probe if no gallery for it
|
||||
test_dataset_name: SUSTech1K
|
||||
|
||||
evaluator_cfg:
|
||||
enable_float16: true
|
||||
restore_ckpt_strict: true
|
||||
restore_hint: 40000
|
||||
save_name: lidargaitv2
|
||||
eval_func: evaluate_indoor_dataset
|
||||
sampler:
|
||||
batch_shuffle: false
|
||||
points_in_use: # For point-based gait recognition using point clouds
|
||||
pointcloud_index: 0
|
||||
points_num: 1024 #2048
|
||||
batch_size: 4
|
||||
frames_num_fixed: 10 # fixed frames number for training
|
||||
frames_skip_num: 0
|
||||
#sample_type: fixed_ordered
|
||||
sample_type: all_ordered # all indicates whole sequence used to test, while ordered means input sequence by its natural order; Other options: fixed_unordered
|
||||
frames_all_limit: 720 # limit the number of sampled frames to prevent out of memory
|
||||
metric: euc # cos
|
||||
transform:
|
||||
- type: PointCloudsTransform
|
||||
xyz_only: true
|
||||
scale_aware: true
|
||||
|
||||
loss_cfg:
|
||||
- loss_term_weight: 1.0
|
||||
margin: 0.2
|
||||
type: TripletLoss
|
||||
log_prefix: triplet
|
||||
lazy: False
|
||||
- loss_term_weight: 0.1
|
||||
scale: 31
|
||||
type: CrossEntropyLoss
|
||||
log_prefix: softmax
|
||||
log_accuracy: true
|
||||
|
||||
|
||||
model_cfg:
|
||||
model: LidarGaitPlusPlus
|
||||
pool: PPP_HAP
|
||||
sampling: knn
|
||||
channel: 16
|
||||
npoints: [512, 256, 128]
|
||||
nsample: 32
|
||||
scale_aware: true
|
||||
normalize_dp: true
|
||||
SeparateFCs:
|
||||
in_channels: 256
|
||||
out_channels: 256
|
||||
parts_num: 31
|
||||
SeparateBNNecks:
|
||||
class_num: 250
|
||||
in_channels: 256
|
||||
parts_num: 31
|
||||
scale:
|
||||
- 1
|
||||
- 2
|
||||
- 4
|
||||
- 8
|
||||
- 16
|
||||
|
||||
|
||||
optimizer_cfg:
|
||||
lr: 0.1
|
||||
momentum: 0.9
|
||||
solver: SGD
|
||||
weight_decay: 0.0005
|
||||
|
||||
scheduler_cfg:
|
||||
T_max: 40000
|
||||
eta_min: 0.0001
|
||||
scheduler: CosineAnnealingLR
|
||||
|
||||
|
||||
trainer_cfg:
|
||||
enable_float16: false #true # half_percesion float for memory reduction and speedup
|
||||
fix_BN: false
|
||||
with_test: true
|
||||
log_iter: 100
|
||||
restore_ckpt_strict: true
|
||||
restore_hint: 0
|
||||
save_iter: 5000
|
||||
save_name: lidargaitv2
|
||||
sync_BN: true
|
||||
total_iter: 40000
|
||||
sampler:
|
||||
batch_shuffle: true
|
||||
batch_size:
|
||||
- 32 # TripletSampler, batch_size[0] indicates Number of Identity
|
||||
- 4 # batch_size[1] indicates Samples sequqnce for each Identity
|
||||
frames_num_fixed: 10 # fixed frames number for training
|
||||
sample_type: fixed_unordered # fixed control input frames number, unordered for controlling order of input tensor; Other options: unfixed_ordered or all_ordered
|
||||
type: TripletSampler
|
||||
points_in_use:
|
||||
pointcloud_index: 0
|
||||
points_num: 1024
|
||||
transform:
|
||||
- type: PointCloudsTransform
|
||||
xyz_only: true
|
||||
scale_aware: true
|
||||
scale_prob: 1
|
||||
flip_prob: 0.15
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,29 @@
|
||||
import argparse
|
||||
import os
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from tqdm import tqdm
|
||||
|
||||
|
||||
|
||||
def rearrange(input_path: Path) -> None:
|
||||
for id in tqdm(os.listdir(input_path)):
|
||||
for device in os.listdir(os.path.join(input_path,id)):
|
||||
for seq in os.listdir(os.path.join(input_path,id,device)):
|
||||
for pkl in ['image', 'lidar', 'range_pkl', 'smpl', 'kp3d']:
|
||||
pkl_dir = os.path.join(input_path, id, device, seq, pkl, pkl+'.pkl')
|
||||
target_dir = os.path.join(input_path, id, device, seq, pkl+'.pkl')
|
||||
shutil.move(pkl_dir, target_dir)
|
||||
os.rmdir(os.path.join(input_path, id, device, seq, pkl))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
parser = argparse.ArgumentParser(description='FreeGait rearrange tool')
|
||||
parser.add_argument('-i', '--input_path', required=True, type=str,
|
||||
help='Root path of raw dataset.')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
input_path = Path(args.input_path).resolve()
|
||||
rearrange(input_path)
|
||||
print('Done!')
|
||||
@@ -33,6 +33,8 @@ class CollateFn(object):
|
||||
if self.sampler == 'all' and 'frames_all_limit' in sample_config:
|
||||
self.frames_all_limit = sample_config['frames_all_limit']
|
||||
|
||||
self.points_in_use = sample_config.get('points_in_use')
|
||||
|
||||
def __call__(self, batch):
|
||||
batch_size = len(batch)
|
||||
# currently, the functionality of feature_num is not fully supported yet, it refers to 1 now. We are supposed to make our framework support multiple source of input data, such as silhouette, or skeleton.
|
||||
@@ -88,7 +90,14 @@ class CollateFn(object):
|
||||
|
||||
for i in range(feature_num):
|
||||
for j in indices[:self.frames_all_limit] if self.frames_all_limit > -1 and len(indices) > self.frames_all_limit else indices:
|
||||
sampled_fras[i].append(seqs[i][j])
|
||||
point_cloud_index = self.points_in_use.get('pointcloud_index')
|
||||
if self.points_in_use is not None and point_cloud_index is not None and i == point_cloud_index:
|
||||
points_num = self.points_in_use.get('points_num')
|
||||
sample_points = (random.choices(range(len(seqs[i][j])), k=points_num)
|
||||
if points_num is not None else list(range(len(seqs[i][j]))))
|
||||
sampled_fras[i].append(np.asarray([seqs[i][j][p] for p in sample_points]))
|
||||
else:
|
||||
sampled_fras[i].append(seqs[i][j])
|
||||
return sampled_fras
|
||||
|
||||
# f: feature_num
|
||||
@@ -112,4 +121,4 @@ class CollateFn(object):
|
||||
batch[-1] = np.asarray(seqL_batch)
|
||||
|
||||
batch[0] = fras_batch
|
||||
return batch
|
||||
return batch
|
||||
+116
-1
@@ -228,6 +228,121 @@ def get_transform(trf_cfg=None):
|
||||
return transform
|
||||
raise "Error type for -Transform-Cfg-"
|
||||
|
||||
# **************** For LidarGait++ ****************
|
||||
# Shen, et al: LidarGait++: Learning Local Features and Size Awareness from LiDAR Point Clouds for 3D Gait Recognition, CVPR2025
|
||||
|
||||
def normalize_point_cloud(batch_data):
|
||||
"""Normalize the batch data using coordinates of the block centered at origin.
|
||||
|
||||
Input:
|
||||
batch_data: BxNxC array
|
||||
Output:
|
||||
BxNxC array
|
||||
"""
|
||||
centroids = np.mean(batch_data, axis=1, keepdims=True) # shape: (B, 1, C)
|
||||
centered = batch_data - centroids
|
||||
scales = np.max(np.linalg.norm(centered, axis=2), axis=1, keepdims=True) # shape: (B, 1)
|
||||
scales = scales.reshape(batch_data.shape[0], 1, 1) # (B, 1, 1) for broadcasting
|
||||
return centered / scales
|
||||
|
||||
|
||||
def dropout_point_cloud(batch_data, max_dropout_ratio=0.875, prob=0.2):
|
||||
"""Randomly drop points in each point cloud.
|
||||
|
||||
Input:
|
||||
batch_data: BxNx3 array
|
||||
Output:
|
||||
BxNx3 array, with dropped points replaced by the first point in each cloud.
|
||||
"""
|
||||
if np.random.rand() >= prob:
|
||||
return batch_data
|
||||
B, N, C = batch_data.shape
|
||||
# 为每个点云生成一个 dropout_ratio（范围 0 ~ max_dropout_ratio）
|
||||
dropout_ratio = np.random.rand(B, 1) * max_dropout_ratio # shape: (B, 1)
|
||||
random_matrix = np.random.rand(B, N)
|
||||
drop_mask = random_matrix <= dropout_ratio # shape: (B, N)
|
||||
# 构造每个点云第一个点重复 N 次的数组，用于替换被 dropout 的点
|
||||
first_points = np.repeat(batch_data[:, :1, :], N, axis=1)
|
||||
return np.where(drop_mask[..., None], first_points, batch_data)
|
||||
|
||||
def shift_point_cloud(batch_data, shift_range=0.1, prob=0.2):
|
||||
""" Randomly shift point cloud. Shift is per point cloud.
|
||||
Input:
|
||||
BxNx3 array, original batch of point clouds
|
||||
Return:
|
||||
BxNx3 array, shifted batch of point clouds
|
||||
"""
|
||||
if np.random.rand() >= prob:
|
||||
return batch_data
|
||||
B, N, C = batch_data.shape
|
||||
shifts = np.random.uniform(-shift_range, shift_range, (B, N,3))
|
||||
batch_data += shifts
|
||||
return batch_data
|
||||
|
||||
|
||||
def scale_point_cloud(batch_data, scale_low=0.8, scale_high=1.25, prob=0.2):
|
||||
""" Randomly scale the point cloud. Scale is per point cloud.
|
||||
Input:
|
||||
BxNx3 array, original batch of point clouds
|
||||
Return:
|
||||
BxNx3 array, scaled batch of point clouds
|
||||
"""
|
||||
if np.random.rand() >= prob:
|
||||
return batch_data
|
||||
B, N, C = batch_data.shape
|
||||
scales = np.random.uniform(scale_low, scale_high, B)
|
||||
for batch_index in range(B):
|
||||
batch_data[batch_index,:,:] *= scales[batch_index]
|
||||
return batch_data
|
||||
|
||||
def jitter_point_cloud(batch_data, std=0.01, clip=0.05, prob=0.2):
|
||||
if np.random.rand() >= prob:
|
||||
return batch_data
|
||||
B, N, C = batch_data.shape
|
||||
jittered_data = np.random.normal(loc=0.0, scale=std, size=(B, N, C))
|
||||
jittered_data = np.clip(jittered_data, -clip, clip)
|
||||
batch_data += jittered_data
|
||||
return batch_data
|
||||
|
||||
def flip_point_cloud_y(batch_data, prob=0.25):
|
||||
if np.random.rand() >= prob:
|
||||
return batch_data
|
||||
batch_data[:, :, 1] = -batch_data[:, :, 1]
|
||||
return batch_data
|
||||
|
||||
|
||||
def getxyz(batch_data,col = 2,to_ground=False):
|
||||
B,N,C = batch_data.shape
|
||||
last_col = batch_data[:, :, col]
|
||||
result = last_col.reshape((B, N, 1))
|
||||
if to_ground:
|
||||
result -= result.min(axis=1,keepdims=True)
|
||||
return result
|
||||
|
||||
class PointCloudsTransform():
|
||||
def __init__(self, xyz_only=True, scale_aware=False, drop_prob=0, shift_prob=0, jit_prob=0,scale_prob=0, flip_prob=0):
|
||||
self.scale_aware = scale_aware
|
||||
self.xyz_only = xyz_only
|
||||
|
||||
self.flip_prob, self.shift_prob, self.jit_prob, self.scale_prob, self.drop_prob = flip_prob, shift_prob, jit_prob, scale_prob, drop_prob
|
||||
def __call__(self, points):
|
||||
if self.xyz_only:
|
||||
points = points[:,:,:3]
|
||||
|
||||
heights = getxyz(points, col = 2, to_ground=True)
|
||||
points = normalize_point_cloud(points)
|
||||
|
||||
points = flip_point_cloud_y(points, prob=self.flip_prob)
|
||||
points = shift_point_cloud(points, prob=self.shift_prob)
|
||||
points = jitter_point_cloud(points, prob=self.jit_prob)
|
||||
points = scale_point_cloud(points, prob=self.scale_prob)
|
||||
points = dropout_point_cloud(points, prob=self.drop_prob)
|
||||
|
||||
if self.scale_aware:
|
||||
points = np.concatenate([points,heights],axis=-1)
|
||||
return points
|
||||
|
||||
|
||||
# **************** For GaitSSB ****************
|
||||
# Fan, et al: Learning Gait Representation from Massive Unlabelled Walking Videos: A Benchmark, T-PAMI2023
|
||||
|
||||
@@ -587,4 +702,4 @@ class MSGGTransform():
|
||||
|
||||
def __call__(self, x):
|
||||
result=x[...,self.mask,:].copy()
|
||||
return result
|
||||
return result
|
||||
@@ -456,4 +456,45 @@ def evaluate_scoliosis(data, dataset, metric='euc'):
|
||||
print(f"{cls} Specificity: {TNR[i] * 100:.2f}%")
|
||||
print(f"Accuracy: {accuracy * 100:.2f}%")
|
||||
|
||||
return result_dict
|
||||
return result_dict
|
||||
|
||||
def evaluate_FreeGait(data, dataset, metric='euc'):
|
||||
msg_mgr = get_msg_mgr()
|
||||
|
||||
features, labels, cams, time_seqs = data['embeddings'], data['labels'], data['types'], data['views']
|
||||
import json
|
||||
probe_sets = json.load(
|
||||
open('./datasets/FreeGait/FreeGait.json', 'rb'))['PROBE_SET']
|
||||
|
||||
probe_mask = []
|
||||
for id, ty, sq in zip(labels, cams, time_seqs):
|
||||
if '-'.join([id, ty, sq]) in probe_sets:
|
||||
probe_mask.append(True)
|
||||
else:
|
||||
probe_mask.append(False)
|
||||
probe_mask = np.array(probe_mask)
|
||||
|
||||
# probe_features = features[:probe_num]
|
||||
probe_features = features[probe_mask]
|
||||
# gallery_features = features[probe_num:]
|
||||
gallery_features = features[~probe_mask]
|
||||
# probe_lbls = np.asarray(labels[:probe_num])
|
||||
# gallery_lbls = np.asarray(labels[probe_num:])
|
||||
probe_lbls = np.asarray(labels)[probe_mask]
|
||||
gallery_lbls = np.asarray(labels)[~probe_mask]
|
||||
|
||||
results = {}
|
||||
msg_mgr.log_info(f"The test metric you choose is {metric}.")
|
||||
dist = cuda_dist(probe_features, gallery_features, metric).cpu().numpy()
|
||||
cmc, all_AP, all_INP = evaluate_rank(dist, probe_lbls, gallery_lbls)
|
||||
|
||||
mAP = np.mean(all_AP)
|
||||
mINP = np.mean(all_INP)
|
||||
for r in [1, 5, 10]:
|
||||
results['scalar/test_accuracy/Rank-{}'.format(r)] = cmc[r - 1] * 100
|
||||
results['scalar/test_accuracy/mAP'] = mAP * 100
|
||||
results['scalar/test_accuracy/mINP'] = mINP * 100
|
||||
|
||||
# print_csv_format(dataset_name, results)
|
||||
msg_mgr.log_info(results)
|
||||
return results
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
import torch.nn.functional as F
|
||||
from einops import rearrange
|
||||
|
||||
from .lidargaitv2_utils import PointNetSetAbstraction, PPPooling, PPPooling_UDP,NetVLAD
|
||||
|
||||
from ..base_model import BaseModel
|
||||
from ..modules import SeparateFCs, SeparateBNNecks
|
||||
|
||||
|
||||
class LidarGaitPlusPlus(BaseModel):
|
||||
def build_network(self, model_cfg):
|
||||
C = model_cfg['channel']
|
||||
C_out = model_cfg['SeparateFCs']['in_channels']
|
||||
scale_aware = model_cfg['scale_aware']
|
||||
normalize_dp = model_cfg['normalize_dp']
|
||||
sampling = model_cfg['sampling']
|
||||
|
||||
npoints = model_cfg.get('npoints', [512, 256, 128])
|
||||
nsample = model_cfg.get('nsample', 32)
|
||||
in_channel = 4 if scale_aware else 3
|
||||
|
||||
self.sa1 = PointNetSetAbstraction(npoint=npoints[0], radius=0.1, nsample=nsample, in_channel=in_channel, mlp=[2*C, 2*C, 4*C], group_all=False, sampling=sampling, scale_aware=scale_aware, normalize_dp=normalize_dp)
|
||||
self.sa2 = PointNetSetAbstraction(npoint=npoints[1], radius=0.2, nsample=nsample, in_channel=4*C + in_channel, mlp=[4*C, 4*C, 8*C], group_all=False, sampling=sampling, scale_aware=scale_aware, normalize_dp=normalize_dp)
|
||||
self.sa3 = PointNetSetAbstraction(npoint=npoints[2], radius=0.4, nsample=nsample, in_channel=8*C + in_channel, mlp=[8*C, 8*C, 16*C], group_all=False, sampling=sampling, scale_aware=scale_aware, normalize_dp=normalize_dp)
|
||||
self.sa4 = PointNetSetAbstraction(npoint=None, radius=None, nsample=None, in_channel=16*C + in_channel, mlp=[16*C, 16*C, C_out], group_all=True, sampling=sampling, scale_aware=scale_aware, normalize_dp=normalize_dp)
|
||||
|
||||
if model_cfg['pool'] == 'VLAD':
|
||||
self.pool = NetVLAD(num_clusters=16, dim=C_out, alpha=1.0)
|
||||
elif model_cfg['pool'] == 'GMaxP':
|
||||
self.pool = PPPooling_UDP([1])
|
||||
elif model_cfg['pool'] == 'PPP_UDP':
|
||||
self.pool = PPPooling_UDP(model_cfg['scale'])
|
||||
elif model_cfg['pool'] == 'PPP_UAP':
|
||||
self.pool = PPPooling(scale_aware=False, bin_num=model_cfg['scale'])
|
||||
elif model_cfg['pool'] == 'PPP_HAP':
|
||||
self.pool = PPPooling(scale_aware=True, bin_num=model_cfg['scale'])
|
||||
|
||||
|
||||
|
||||
self.BNNecks = SeparateBNNecks(**model_cfg['SeparateBNNecks'])
|
||||
self.FCs = SeparateFCs(**model_cfg['SeparateFCs'])
|
||||
|
||||
|
||||
def forward(self, inputs):
|
||||
ipts, labs, _, views, seqL = inputs
|
||||
|
||||
xyz = ipts[0]
|
||||
B, T, N, C = xyz.shape
|
||||
xyz = rearrange(xyz, 'B T N C -> (B T) C N')
|
||||
|
||||
|
||||
l1_xyz, l1_points = self.sa1(xyz, None)
|
||||
l1_points = torch.max(l1_points, dim=-2)[0]
|
||||
|
||||
l2_xyz, l2_points = self.sa2(l1_xyz, l1_points)
|
||||
l2_points = torch.max(l2_points, dim=-2)[0]
|
||||
|
||||
l3_xyz, l3_points = self.sa3(l2_xyz, l2_points)
|
||||
l3_points = torch.max(l3_points, dim=-2)[0]
|
||||
|
||||
l4_xyz, l4_points = self.sa4(l3_xyz, l3_points)
|
||||
|
||||
x = self.pool(l4_points, l3_xyz)
|
||||
|
||||
|
||||
x = rearrange(x, '(B T) feat p -> B T feat p', B=B)
|
||||
feat = x.max(1)[0]# x.mean(1) # x.max(1)[0]
|
||||
embed = self.FCs(feat) # [n, c, p]
|
||||
embed_2, logits = self.BNNecks(embed) # [n, c, p]
|
||||
retval = {
|
||||
'training_feat': {
|
||||
'triplet': {'embeddings': embed, 'labels': labs},
|
||||
'softmax': {'logits': logits, 'labels': labs}
|
||||
},
|
||||
'visual_summary': {
|
||||
},
|
||||
'inference_feat': {
|
||||
'embeddings': embed,
|
||||
}
|
||||
}
|
||||
return retval
|
||||
@@ -0,0 +1,377 @@
|
||||
import torch.nn as nn
|
||||
import torch.nn.functional as F
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
import torch.utils.data
|
||||
import torch.nn.functional as F
|
||||
from einops import rearrange
|
||||
from torch.autograd import Variable
|
||||
import numpy as np
|
||||
|
||||
|
||||
def square_distance(src, dst):
|
||||
"""
|
||||
Calculate Euclid distance between each two points.
|
||||
src^T * dst = xn * xm + yn * ym + zn * zm；
|
||||
sum(src^2, dim=-1) = xn*xn + yn*yn + zn*zn;
|
||||
sum(dst^2, dim=-1) = xm*xm + ym*ym + zm*zm;
|
||||
dist = (xn - xm)^2 + (yn - ym)^2 + (zn - zm)^2
|
||||
= sum(src**2,dim=-1)+sum(dst**2,dim=-1)-2*src^T*dst
|
||||
Input:
|
||||
src: source points, [B, N, C]
|
||||
dst: target points, [B, M, C]
|
||||
Output:
|
||||
dist: per-point square distance, [B, N, M]
|
||||
"""
|
||||
B, N, _ = src.shape
|
||||
_, M, _ = dst.shape
|
||||
dist = -2 * torch.matmul(src, dst.permute(0, 2, 1))
|
||||
dist += torch.sum(src ** 2, -1).view(B, N, 1)
|
||||
dist += torch.sum(dst ** 2, -1).view(B, 1, M)
|
||||
return dist
|
||||
|
||||
|
||||
def index_points(points, idx):
|
||||
"""
|
||||
Input:
|
||||
points: input points data, [B, N, C]
|
||||
idx: sample index data, [B, S]
|
||||
Return:
|
||||
new_points:, indexed points data, [B, S, C]
|
||||
"""
|
||||
device = points.device
|
||||
B = points.shape[0]
|
||||
view_shape = list(idx.shape)
|
||||
view_shape[1:] = [1] * (len(view_shape) - 1)
|
||||
repeat_shape = list(idx.shape)
|
||||
repeat_shape[0] = 1
|
||||
batch_indices = torch.arange(B, dtype=torch.long).to(device).view(view_shape).repeat(repeat_shape)
|
||||
new_points = points[batch_indices, idx, :]
|
||||
return new_points
|
||||
|
||||
|
||||
def farthest_point_sample(xyz, npoint):
|
||||
"""
|
||||
Input:
|
||||
xyz: pointcloud data, [B, N, 3]
|
||||
npoint: number of samples
|
||||
Return:
|
||||
centroids: sampled pointcloud index, [B, npoint]
|
||||
"""
|
||||
device = xyz.device
|
||||
B, N, C = xyz.shape
|
||||
centroids = torch.zeros(B, npoint, dtype=torch.long).to(device)
|
||||
distance = torch.ones(B, N).to(device) * 1e10
|
||||
farthest = torch.randint(0, N, (B,), dtype=torch.long).to(device)
|
||||
batch_indices = torch.arange(B, dtype=torch.long).to(device)
|
||||
for i in range(npoint):
|
||||
centroids[:, i] = farthest
|
||||
centroid = xyz[batch_indices, farthest, :].view(B, 1, C)
|
||||
dist = torch.sum((xyz - centroid) ** 2, -1)
|
||||
mask = dist < distance
|
||||
distance[mask] = dist[mask]
|
||||
farthest = torch.max(distance, -1)[1]
|
||||
return centroids
|
||||
|
||||
|
||||
def ball_query(radius, nsample, xyz, new_xyz):
|
||||
"""
|
||||
Input:
|
||||
radius: local region radius
|
||||
nsample: max sample number in local region
|
||||
xyz: all points, [B, N, 3]
|
||||
new_xyz: query points, [B, S, 3]
|
||||
Return:
|
||||
group_idx: grouped points index, [B, S, nsample]
|
||||
"""
|
||||
device = xyz.device
|
||||
B, N, C = xyz.shape
|
||||
_, S, _ = new_xyz.shape
|
||||
group_idx = torch.arange(N, dtype=torch.long).to(device).view(1, 1, N).repeat([B, S, 1])
|
||||
xyz = xyz[:,:,:3]
|
||||
new_xyz = new_xyz[:,:,:3]
|
||||
sqrdists = square_distance(new_xyz, xyz)
|
||||
group_idx[sqrdists > radius ** 2] = N
|
||||
group_idx = group_idx.sort(dim=-1)[0][:, :, :nsample]
|
||||
group_first = group_idx[:, :, 0].view(B, S, 1).repeat([1, 1, nsample])
|
||||
mask = group_idx == N
|
||||
group_idx[mask] = group_first[mask]
|
||||
return group_idx
|
||||
|
||||
|
||||
def knn_query(k, xyz, new_xyz):
|
||||
"""
|
||||
Input:
|
||||
k: number of nearest neighbors to query
|
||||
xyz: all points, [B, N, 3]
|
||||
new_xyz: query points, [B, S, 3]
|
||||
Return:
|
||||
group_idx: indices of k-nearest neighbors, [B, S, k]
|
||||
"""
|
||||
B, N, C = xyz.shape
|
||||
_, S, _ = new_xyz.shape
|
||||
xyz = xyz[:,:,:3]
|
||||
new_xyz = new_xyz[:,:,:3]
|
||||
dists = square_distance(new_xyz, xyz)
|
||||
#scaling_factor = torch.Tensor([1, 1, 0.6]).to(new_xyz.device)
|
||||
#dists = torch.sum(torch.square(new_xyz.unsqueeze(2) - xyz.unsqueeze(1)) / scaling_factor, dim=-1)
|
||||
group_idx = dists.sort(dim=-1)[1][:, :, :k]
|
||||
return group_idx
|
||||
|
||||
|
||||
def sample_and_group(npoint, radius, nsample, xyz, points, returnfps=False, sampling='ball',scale_aware=False, normalize_dp=False):
|
||||
"""
|
||||
Input:
|
||||
npoint:
|
||||
radius:
|
||||
nsample:
|
||||
xyz: input points position data, [B, N, 3]
|
||||
points: input points data, [B, N, D]
|
||||
Return:
|
||||
new_xyz: sampled points position data, [B, npoint, nsample, 3]
|
||||
new_points: sampled points data, [B, npoint, nsample, 3+D]
|
||||
"""
|
||||
B, N, C = xyz.shape
|
||||
S = npoint
|
||||
fps_idx = farthest_point_sample(xyz[:,:,:3], npoint) # [B, npoint, C]
|
||||
new_xyz = index_points(xyz, fps_idx)
|
||||
|
||||
if sampling == 'ball':
|
||||
idx = ball_query(radius, nsample, xyz, new_xyz)
|
||||
elif sampling == 'knn':
|
||||
idx = knn_query(nsample, xyz, new_xyz)
|
||||
else:
|
||||
raise ValueError("Unsupported sampling type. Use 'ball' or 'knn'.")
|
||||
|
||||
grouped_xyz = index_points(xyz, idx) # [B, npoint, nsample, C]
|
||||
grouped_xyz_norm = grouped_xyz - new_xyz.view(B, S, 1, C)
|
||||
if normalize_dp: # and sampling!='knn':
|
||||
grouped_xyz_norm /= radius
|
||||
grouped_xyz_norm = grouped_xyz_norm if scale_aware else grouped_xyz_norm[:,:,:,:3]
|
||||
|
||||
|
||||
if points is not None:
|
||||
grouped_points = index_points(points, idx)
|
||||
new_points = torch.cat([grouped_xyz_norm, grouped_points], dim=-1) # [B, npoint, nsample, C+D]
|
||||
else:
|
||||
new_points = grouped_xyz_norm
|
||||
if returnfps:
|
||||
return new_xyz, new_points, grouped_xyz, fps_idx
|
||||
else:
|
||||
return new_xyz, new_points
|
||||
|
||||
|
||||
def sample_and_group_all(xyz, points, scale_aware=False):
|
||||
"""
|
||||
Input:
|
||||
xyz: input points position data, [B, N, 3]
|
||||
points: input points data, [B, N, D]
|
||||
Return:
|
||||
new_xyz: sampled points position data, [B, 1, 3]
|
||||
new_points: sampled points data, [B, 1, N, 3+D]
|
||||
"""
|
||||
device = xyz.device
|
||||
B, N, C = xyz.shape
|
||||
new_xyz = torch.zeros(B, 1, C).to(device)
|
||||
grouped_xyz = xyz.view(B, 1, N, C)
|
||||
grouped_xyz = grouped_xyz if scale_aware else grouped_xyz[:,:,:,:3]
|
||||
if points is not None:
|
||||
new_points = torch.cat([grouped_xyz, points.view(B, 1, N, -1)], dim=-1)
|
||||
else:
|
||||
new_points = grouped_xyz
|
||||
return new_xyz, new_points
|
||||
|
||||
|
||||
class PointNetSetAbstraction(nn.Module):
|
||||
def __init__(self, npoint, radius, nsample, in_channel, mlp, group_all, sampling='ball', scale_aware=False,normalize_dp=False):
|
||||
super(PointNetSetAbstraction, self).__init__()
|
||||
self.npoint = npoint
|
||||
self.radius = radius
|
||||
self.nsample = nsample
|
||||
self.mlp_convs = nn.ModuleList()
|
||||
self.mlp_bns = nn.ModuleList()
|
||||
last_channel = in_channel
|
||||
for out_channel in mlp:
|
||||
self.mlp_convs.append(nn.Conv2d(last_channel, out_channel, 1))
|
||||
self.mlp_bns.append(nn.BatchNorm2d(out_channel))
|
||||
last_channel = out_channel
|
||||
self.group_all = group_all
|
||||
self.scale_aware = scale_aware
|
||||
self.normalize_dp = normalize_dp
|
||||
self.sampling = sampling
|
||||
def forward(self, xyz, points):
|
||||
"""
|
||||
Input:
|
||||
xyz: input points position data, [B, C, N]
|
||||
points: input points data, [B, D, N]
|
||||
Return:
|
||||
new_xyz: sampled points position data, [B, C, S]
|
||||
new_points_concat: sample points feature data, [B, D', S]
|
||||
"""
|
||||
xyz = xyz.permute(0, 2, 1)
|
||||
if points is not None:
|
||||
points = points.permute(0, 2, 1)
|
||||
|
||||
if self.group_all:
|
||||
new_xyz, new_points = sample_and_group_all(xyz, points, scale_aware=self.scale_aware)
|
||||
else:
|
||||
new_xyz, new_points = sample_and_group(self.npoint, self.radius, self.nsample, xyz, points, sampling=self.sampling, scale_aware=self.scale_aware,normalize_dp=self.normalize_dp)
|
||||
# new_xyz: sampled points position data, [B, ], C]
|
||||
# new_points: sampled points data, [B, npoint, nsample, C+D]
|
||||
new_points = new_points.permute(0, 3, 2, 1) # [B, C+D, nsample,npoint]
|
||||
for i, conv in enumerate(self.mlp_convs):
|
||||
bn = self.mlp_bns[i]
|
||||
new_points = F.relu(bn(conv(new_points)))
|
||||
|
||||
new_points = new_points
|
||||
new_xyz = new_xyz.permute(0, 2, 1)
|
||||
return new_xyz, new_points
|
||||
|
||||
class PPPooling_UDP():
|
||||
"""
|
||||
Hierarchically Clustered Point Pooling
|
||||
"""
|
||||
|
||||
def __init__(self, bin_num=None):
|
||||
if bin_num is None:
|
||||
bin_num = [16, 8, 4, 2, 1]
|
||||
self.bin_num = bin_num
|
||||
|
||||
def __call__(self, x, xyz):
|
||||
"""
|
||||
x : [n, c, h, w]
|
||||
xyz: [n, 3, p]
|
||||
ret: [n, c, p]
|
||||
"""
|
||||
#print(xyz.shape)
|
||||
#x = rearrange(x, 'b n c -> b c n 1')
|
||||
n, c = x.size()[:2]
|
||||
_, idx = xyz[:, 2, :].sort()
|
||||
x = x.gather(2, idx.unsqueeze(1).unsqueeze(-1).expand_as(x))
|
||||
features = []
|
||||
for b in self.bin_num:
|
||||
z = x.view(n, c, b, -1)
|
||||
z = z.mean(-1) + z.max(-1)[0]
|
||||
features.append(z)
|
||||
return torch.cat(features, -1)
|
||||
|
||||
|
||||
class PPPooling():
|
||||
def __init__(self, scale_aware=False, bin_num=None):
|
||||
# 默认设置多个分辨率的分bin数量
|
||||
self.bin_num = bin_num if bin_num is not None else [16, 8, 4, 2, 1]
|
||||
self.scale_aware = scale_aware
|
||||
|
||||
def __call__(self, point_clouds, points):
|
||||
# 调整维度：输入 point_clouds: B x C x N x 1 转换为 B x N x C，
|
||||
# points: B x C x N 转换为 B x N x C
|
||||
point_clouds = rearrange(point_clouds, 'B C N 1 -> B N C')
|
||||
points = rearrange(points, 'B C N -> B N C')
|
||||
B, N, C = point_clouds.shape
|
||||
|
||||
if self.scale_aware: # PPPooling_HAP
|
||||
z = points[:, :, 3] # shape: (B, N)
|
||||
# 固定的 z 范围（例如：0 到 2）
|
||||
z_min, z_max = 0.0, 2.0
|
||||
else:
|
||||
# PPPooling_UAP
|
||||
# 使用 points 的第 3 个通道作为 z 坐标，归一化到 [0, 1]
|
||||
z = points[:, :, 2] # shape: (B, N)
|
||||
z_min = z.min(dim=1, keepdim=True)[0][0].item()
|
||||
z_max = z.max(dim=1, keepdim=True)[0][0].item()
|
||||
z_range = z_max - z_min + 1e-6
|
||||
z = (z - z_min) / z_range # shape: (B, N)
|
||||
z_min, z_max = 0.0, 1.0
|
||||
|
||||
all_pooled = []
|
||||
for M in self.bin_num:
|
||||
# 由于 z 已归一化，直接构造均匀分布的 bin 边界
|
||||
edges = torch.linspace(z_min, z_max, steps=M + 1, device=point_clouds.device)
|
||||
# 利用 bucketize 将每个点分配到 [0, M-1] 内的 bin（不需要额外处理首尾）
|
||||
# 注意：这里使用 edges[1:-1] 作为分界，保证边界值归到正确 bin
|
||||
bin_idx = torch.bucketize(z.contiguous(), edges[1:-1], right=False) # shape: (B, N)
|
||||
|
||||
# 为每个 bin计算 max 和 mean 池化值，利用 scatter_reduce 与 scatter_add 操作：
|
||||
# 构造初始 tensor，形状均为 (B, M, C)
|
||||
pooled_max = torch.full((B, M, C), float('-inf'), device=point_clouds.device, dtype=point_clouds.dtype)
|
||||
pooled_sum = torch.zeros((B, M, C), device=point_clouds.device, dtype=point_clouds.dtype)
|
||||
counts = torch.zeros((B, M, 1), device=point_clouds.device, dtype=point_clouds.dtype)
|
||||
|
||||
# 将 bin_idx 扩展到与 point_clouds 对应的维度 (B, N, C)
|
||||
bin_idx_exp = bin_idx.unsqueeze(-1).expand(-1, -1, C)
|
||||
# max 池化：scatter_reduce 计算每个 bin 内的最大值
|
||||
pooled_max = pooled_max.scatter_reduce(1, bin_idx_exp, point_clouds, reduce='amax', include_self=True)
|
||||
# sum 池化：scatter_add 计算每个 bin 内的和
|
||||
pooled_sum = pooled_sum.scatter_add(1, bin_idx_exp, point_clouds)
|
||||
# 计算每个 bin 的计数
|
||||
counts = counts.scatter_add(1, bin_idx.unsqueeze(-1), torch.ones((B, N, 1), device=point_clouds.device))
|
||||
# 计算 mean 池化
|
||||
pooled_mean = pooled_sum / counts.clamp(min=1)
|
||||
# 这里采用 max 与 mean 的和作为最终池化结果（也可以用 concat）
|
||||
pooled = pooled_max + pooled_mean
|
||||
# 将没有点（max为 -inf）的 bin 置 0
|
||||
pooled[pooled == float('-inf')] = 0
|
||||
|
||||
all_pooled.append(pooled)
|
||||
|
||||
# 将各分辨率下的池化结果在 bin 维度上拼接，并调整为 B x C x M_total
|
||||
output = torch.cat(all_pooled, dim=1)
|
||||
output = rearrange(output, 'B M C -> B C M')
|
||||
return output
|
||||
|
||||
|
||||
class NetVLAD(nn.Module):
|
||||
"""NetVLAD layer implementation"""
|
||||
|
||||
def __init__(self, num_clusters=64, dim=128, alpha=100.0,
|
||||
normalize_input=True):
|
||||
"""
|
||||
Args:
|
||||
num_clusters : int
|
||||
The number of clusters
|
||||
dim : int
|
||||
Dimension of descriptors
|
||||
alpha : float
|
||||
Parameter of initialization. Larger value is harder assignment.
|
||||
normalize_input : bool
|
||||
If true, descriptor-wise L2 normalization is applied to input.
|
||||
"""
|
||||
super(NetVLAD, self).__init__()
|
||||
self.num_clusters = num_clusters
|
||||
self.dim = dim
|
||||
self.alpha = alpha
|
||||
self.normalize_input = normalize_input
|
||||
self.conv = nn.Conv2d(dim, num_clusters, kernel_size=(1, 1), bias=True)
|
||||
self.centroids = nn.Parameter(torch.rand(num_clusters, dim))
|
||||
self._init_params()
|
||||
|
||||
def _init_params(self):
|
||||
self.conv.weight = nn.Parameter(
|
||||
(2.0 * self.alpha * self.centroids).unsqueeze(-1).unsqueeze(-1)
|
||||
)
|
||||
self.conv.bias = nn.Parameter(
|
||||
- self.alpha * self.centroids.norm(dim=1)
|
||||
)
|
||||
|
||||
def forward(self, x, xyz):
|
||||
N, C = x.shape[:2]
|
||||
|
||||
if self.normalize_input:
|
||||
x = F.normalize(x, p=2, dim=1) # across descriptor dim
|
||||
|
||||
# soft-assignment
|
||||
soft_assign = self.conv(x).view(N, self.num_clusters, -1)
|
||||
soft_assign = F.softmax(soft_assign, dim=1)
|
||||
|
||||
x_flatten = x.view(N, C, -1)
|
||||
|
||||
# calculate residuals to each clusters
|
||||
residual = x_flatten.expand(self.num_clusters, -1, -1, -1).permute(1, 0, 2, 3) - \
|
||||
self.centroids.expand(x_flatten.size(-1), -1, -1).permute(1, 2, 0).unsqueeze(0)
|
||||
residual *= soft_assign.unsqueeze(2)
|
||||
vlad = residual.sum(dim=-1)
|
||||
|
||||
vlad = F.normalize(vlad, p=2, dim=2) # intra-normalization
|
||||
vlad = vlad.view(x.size(0), -1) # flatten b num c -> b num c
|
||||
vlad = F.normalize(vlad, p=2, dim=1) # L2 normalize
|
||||
return vlad.unsqueeze(-1)
|
||||
Reference in New Issue
Block a user