Robbyant/lingbot-touch

Overview

LingBot-Touch is a JEPA-based tactile representation model for dexterous robotic manipulation. It learns a shared latent space across magnetic, piezoresistive and visuotactile sensors, so a model pretrained on one sensor type can transfer to different tactile hardware without extra training. This repository ships the frozen target encoder, sensor adapters and inference utilities for extracting token-level features; pretraining code is not included. It is aimed at robotics and embodied-AI researchers working with tactile sensing.

Getting started

Download the matched config.json and encoder.pt for either LingBot-Touch-Tiny or LingBot-Touch-Small from Hugging Face or ModelScope, and place them in checkpoints/lingbot-touch-tiny/ or checkpoints/lingbot-touch-small/. Then set up a Linux/Python 3.12 environment and install the package:

conda create -n lingbot-touch python=3.12.13 pip=26.0.1 -y
conda activate lingbot-touch
python -m pip install "torch==2.11.0" --index-url https://download.pytorch.org/whl/cu130
python -m pip install .

Run inference on the bundled Sharpa sample, which uses the visuotactile branch:

mkdir -p outputs/sharpa
python infer.py --input-format sharpa \
  --bundle checkpoints/lingbot-touch-tiny \
  --input sample_data/sharpa/001_left.pt \
  --output outputs/sharpa/001_left_features.pt \
  --device cpu

Each output file contains latent of shape [B,N,C] plus time_id, finger_id and pos_xyz addressing each token. The XHand example follows the same pattern with --input-format xhand and the magnetic branch. The repository also includes 120 single-hand sample files across both platforms, which can be processed in a loop Example.

Data requirements

Inputs must be aligned to the format the adapters expect: the Sharpa sample is [1,16,5,200] scalar values, the XHand sample [1,16,5,120,3] three-axis values with a verified layout_id. New raw taxel layouts need verified fingertip calibration parameters (rotation, origin, radius, offset) before running the geometry normalization; tensor shape compatibility alone is not enough, and the XHand mapping is tied to its specific point-order and signal-axis conventions. Optional resampling supports specific pretraining point counts per modality. For readout, the README recommends attention-based querying over all tokens rather than mean/max pooling or per-token linear projection, because JEPA tokens already mix spatiotemporal context. See the data interface guide for input contracts and calibration usage.

HotRepo's take

On its first recorded day on the new-project list (2026-11-10), the repository gained 12 stars over a roughly one-hour observation window, going from 105 to 117 stars. That is a short observation and does not establish a trend. What is concrete is the release shape: a paper-linked model with open weights on two hosting platforms, frozen-encoder inference code, two sample sensor platforms and an Apache-2.0 license. The repository explicitly excludes pretraining from this release, so it is a consumption toolkit for the representation rather than a full training stack.