OPEN-SOURCE RESEARCH REPOSITORY
Full-parameter post-training on Ascend SuperPOD
A reproducible stack for scaling DeepSeek-V4 training on Ascend 910C and specializing DeepSeek-V4-Flash for solver-grounded Operations Research.
REPORT OVERVIEW
SLAI T-Rex starts from one premise: better domain capability requires a training substrate that is stable, observable, and efficient enough to support controlled experiments. A trillion-parameter MoE model stresses memory capacity, pipeline balance, collective communication, and irregular operators, while an Operations Research task also requires mathematical fidelity, executable solver code, and structural equivalence. The project handles these requirements as one connected pipeline, from Ascend runtime work to the final benchmark.
The system study uses DeepSeek-V4-Pro as the large-scale workload and Ascend CloudMatrix384 SuperPOD with Ascend 910C as the execution environment. It starts from profiling rather than from an isolated optimization trick, decomposing a training step into useful computation, exposed communication, pipeline bubble, and device-side idle time. The current Pro SFT implementation organizes the response into two levels: first tune the parameters and environment of existing MindSpeed-LLM capabilities, then introduce custom fused operators and framework patches for selected computation paths.
The model study uses DeepSeek-V4-Flash as the main experimentation platform. The starting checkpoint already contains useful knowledge of common linear and mixed-integer programming templates, but its errors reveal that familiarity with optimization language is not enough. A generated program may compile while using the wrong variable domain, omit a conservation constraint, mishandle a ratio, or produce a graph that is structurally different from the reference formulation. CPT is therefore used to broaden the model's domain prior, while SFT is used to make solver-facing behavior and output contracts reliable at inference time.
This distinction also explains the organization of the open-source repository. Data construction is kept separate from MindSpeed-LLM training. The data tools create self-contained documents and instruction examples; conversion scripts turn them into the indexed format expected by training; launchers provide environment-driven recipes; kernel and framework directories expose the Ascend-specific acceleration path; evaluation scripts measure both execution and formulation structure.
| Stage | Role in the loop | Output | Repository entry |
|---|---|---|---|
| Training substrate | Make the model and runtime stable enough for full-parameter updates on Ascend. | Parallel execution, communication overlap, memory strategy, and optimized operators. | kernel/, docker/ |
| Data and adaptation | Use the stable substrate to inject OR knowledge, then align it with solver-facing instructions. | Solver-verified CPT documents and Clean-CoT SFT examples. | data/, training/ |
| Measurement and feedback | Check whether system and data choices become reliable downstream behavior. | Four OR benchmark scores, general-capability checks, logs, and registries. | eval/, training/ |
THE CONTINUOUS PIPELINE
SLAI T-Rex is one continuous post-training workflow. Ascend-aware parallelism, memory management, and kernel execution first create a stable training substrate. The same substrate then carries solver-verified OR data through CPT and SFT, and finally exposes whether the adaptation worked through execution and structural-equivalence benchmarks.
01 / FOUNDATION
The system optimizations are the first part of the model adaptation story. They make full-parameter MoE updates practical on Ascend, so the later OR data and checkpoint experiments can be interpreted without confusing infrastructure instability with modeling quality.
The OR stage is not an independent application layer. It reuses the training runtime, conversion path, checkpoint format, monitoring, and evaluation boundary established by the system work. CPT supplies domain structure; SFT makes that structure appear as valid, executable, and benchmark-aligned output.
SYSTEM STUDY / ASCEND SUPERPOD
At trillion-parameter scale, device time is shaped by the interaction of parallelism, communication, memory, and operator implementation. The report measures these layers together so that a kernel speedup is interpreted in the context of the training step that it actually affects.
CURRENT PRO SFT
Level 1 tunes MindSpeed-LLM parameters, parallel scheduling, communication, and the runtime environment without introducing new operator implementations. Level 2 then replaces selected high-cost computation paths with fused operators and framework patches. The resulting Pro SFT configuration reaches the best measured MFU of 38.57%.
Tune VPP, MoE All-to-All overlap, GEMM gradient-accumulation fusion, task queue behavior, HCCL settings, and the NPU memory allocator. No new operator implementation is introduced at this level.
Apply framework patches and load custom operator artifacts. The optimized paths cover fused SwiGLU, MoE permutation, DSA index scaling, CPU binding, SparseFlashMla, and LightningIndexer.
This level stays inside MindSpeed-LLM's existing feature set. Its purpose is to expose less communication, reduce pipeline idle time, and avoid unnecessary memory traffic without adding a new operator dependency.
The 512-rank run uses TP=2, PP=8, EP=64, ETP=1, CP=1, and DP=32 with Sequence Parallel. Setting four layers per virtual pipeline stage introduces interleaved VPP scheduling: it shortens idle gaps between pipeline stages, while accepting more scheduling and point-to-point communication.
TP=2 · PP=8 · EP=64 · DP=32 · VPP=4 layers
MoE All-to-All is overlapped with expert computation so that token dispatch is less visible on the critical path. GEMM gradient accumulation is fused into the weight-gradient calculation, removing a separate Add and repeated memory access. Distributed optimizer, full recompute, optimizer swap, and manual garbage collection keep the full-parameter run within the memory budget.
MoE overlap · GEMM grad fusion · recompute · optimizer swap
Task queue level 2 increases asynchronous host-to-device submission. A 256 MB HCCL buffer and topology-aware All-to-All strategy target large-message throughput, while the AIV communication path is selected for this workload. Expandable allocator segments mitigate fragmentation from dynamic tensors and communication buffers. These settings can consume more peak memory and must be revalidated when CANN or HCCL changes.
TASK_QUEUE_ENABLE=2 · HCCL_BUFFSIZE=256 · AIV
This level adds compiled custom operators and the framework patches that route execution into them. It reduces launch overhead and intermediate memory traffic, and changes the sparse-attention path itself rather than only tuning its schedule.
The activation path combines value limiting, SwiGLU, routing-probability multiplication, and the corresponding backward work where applicable. MoE token permutation uses a fused mask-map path, and the DSA indexer combines two constant scales into one operation. These changes remove temporary tensors, dtype conversions, copies, and small elementwise launches.
limited SwiGLU · routing probability · MoE permute · index scaling
LightningIndexer first selects the top 1,024 candidate positions. SparseFlashMla then computes attention only on those positions, its fused backward handles the sparse gradient path, and the fused KL-loss backward updates the indexer. The reduction in selected positions is especially valuable for long sequences, where dense attention would otherwise dominate computation and memory access.
LightningIndexer → topk=1024 → SparseFlashMla → fused backward
The generic FlashAttention switch and the sparse-attention switch select different paths: the latter depends on the custom operator artifacts and matching framework patches. CPU affinity is also applied per local rank to reduce process migration and NUMA jitter, but it is a runtime optimization rather than an NPU kernel. Correctness, compiler compatibility, memory use, and end-to-end MFU must all be checked together after an environment change.
General FlashAttention and DSA sparse attention use distinct execution paths.
The baseline training step spends 40.76 seconds in communication, which accounts for 52.2% of total device kernel time in the reported profile. Tensor and data parallel collectives contribute the largest share, followed by pipeline point-to-point traffic. Expert-parallel AllToAllV is important for MoE token dispatch, but it is a smaller part of this particular breakdown because the SuperPOD interconnect handles that traffic comparatively well.
| Parallelism dimension | Primitives | Latency (s) | Share of communication | Share of total |
|---|---|---|---|---|
| Tensor / Data | AllReduce, AllGather, ReduceScatter | 19.71 | 48.4% | 25.2% |
| Pipeline | Send, Receive | 14.95 | 36.7% | 19.2% |
| Expert | AllToAllV | 5.91 | 14.5% | 7.6% |
| Misc. | Broadcast and others | 0.18 | 0.4% | 0.2% |
| Communication total | 40.76 | 100% | 52.2% |
The system response is hierarchical. Expert-tensor parallelism reduces unnecessary tensor-parallel pressure in MoE layers while keeping the pipeline configuration stable. Load-balanced scheduling addresses uneven work between pipeline stages. Communication and computation are overlapped where the execution graph permits it, and token dispatch is organized so that synchronization does not expose avoidable idle time. The optimizer path uses memory-aware techniques, including a double-buffered swap strategy, to keep full-parameter training within device constraints.
Profiling also shows why generic operator optimization is insufficient. Sparse attention is memory- and control-bound: the vector compute pipeline remains lightly used while local-buffer movement and scalar index or gather logic dominate. By contrast, MatMul and grouped expert computation are closer to cube-compute saturation. The long tail contains many tiny Cast, Add, reshape, and copy operations. Each is short, but the repeated launch and memory traffic costs accumulate across the step, making them natural targets for fusion.
| Operator | Task-duration share | Calls | Average (us) | Observed behavior |
|---|---|---|---|---|
| SparseAttnSharedkvGrad | 16.86% | 224 | 27,593 | Sparse attention backward; low vector compute use |
| MatMulV3 | 12.77% | 5,600 | 836 | Load-heavy dense matrix operation |
| GroupedMatmul | 9.50% | 1,344 | 2,592 | MoE expert computation; cube-bound |
| Cast | 6.02% | 40,063 | 55 | Frequent memory-bound conversion |
| SparseLightningIndexerGradKLLoss | 5.52% | 96 | 21,082 | Sparse indexer gradient; scalar and load bound |
| 92 remaining operator types | 32.88% | 140,357 | about 86 | Small, memory-bound vector and copy kernels |
AuraKernel combines a hardware-grounded harness, iterative optimization memory, parallel candidate evaluation, and solver-guided tiling search. The optimization loop first compiles and checks a candidate for correctness on real Ascend hardware. Only feasible candidates enter performance comparison. This separates mathematical or functional correctness from performance selection, and makes hardware counters useful as a diagnosis of whether a candidate is compute-bound, movement-bound, or scalar-bound.
Within Level 2, operator results are reported at two measurement scopes. AuraKernel improves already-visible bottleneck kernels, including sparse attention, the lightning indexer gradient, and weight-free RMSNorm. The fusion path changes the execution unit itself by removing eager intermediate tensors, dtype round trips, and autograd-visible boundaries. An isolated kernel speedup and a fused module speedup answer different questions, but both contribute to the final 38.57% MFU.
| Operator or module | Baseline context | Speedup | Optimization idea |
|---|---|---|---|
| SparseAttnSharedkv | cmp. ratio 128 | 1.23x | Reduce memory movement in sparse attention |
| SparseAttnSharedkvGrad | cmp. ratio 128 | 1.24x | Restructure memory- and control-bound backward work |
| SparseLightningIndexerGradKLLoss | CFA / SCFA paths | 1.16x / 1.21x | Reduce scalar address calculation and global loads |
| RMSNormWithoutWeightFwd | tp = 2 | 11.90x | 2D blocking, multi-row processing, FP32 accumulation |
| RMSNormWithoutWeightBwd | tp = 2 | 3.42x | Vectorize the reduction and reduce scalar work |
| Limited SwiGLU forward | Fused module | 8.86x | Keep clamp and gate chain on chip |
| RoPE backward | Fused module | 3.54x | Remove eager slice-assignment scatter chain |
WHY TWO STAGES
Operations Research asks a model to preserve meaning across several layers at once: a natural-language problem, a mathematical formulation, a data interface, and an executable solver program. A response that looks plausible or returns a solution is not necessarily faithful to the intended model.
The report's error analysis found that structural equivalence was the largest failure family, while protocol and API mistakes were the most visible Pass@1 failures. Other recurring problems included continuous variables used for discrete entities, incomplete conservation constraints, unit and ratio errors, and nonlinear expressions that were not solver-compatible.
Variable families, constraint counts, and the canonical formulation graph do not match the reference.
Code fences, malformed Python, unsafe indexing, or incorrect Gurobi usage block evaluation before reasoning can be scored.
Counts, assignments, vehicles, and selections need integer or binary domains instead of an LP relaxation.
02 / WORKFLOW
Every stage has a distinct job: preserve mathematical fidelity, inject domain priors, align output contracts, then measure what survives.
Generate and filter optimization seeds with contracts, static checks, and independent solver execution.
Use OR-dominant CPT to teach terminology, formulation patterns, solver APIs, and structural priors.
Clean and distill instruction data around code-only, feasibility, and structural-equivalence contracts.
Run four OR benchmarks and inspect both executable feasibility and canonical model structure.
CPT
Continued pre-training exposes the model to OR terminology, recurring formulation families, solver-aware nonlinear reasoning, data structures, and the relationship between business requirements and mathematical constraints. It is the knowledge-infusion stage of the pipeline.
SFT
Supervised fine-tuning teaches the model to follow code-only contracts, write safe Gurobi programs, handle data files and schemas, extract objective values, and produce canonical outputs without relying on few-shot demonstrations. It is the behavior-shaping stage.
DATA ENGINE
The OR-CPT Engine treats language generation as one step inside a controlled verification path. The release keeps the data and training boundaries separate: the engine establishes mathematical fidelity and lineage, while MindSpeed tokenizes, packs, samples, and optimizes the model.
A seed is a structured optimization instance with sets, coefficients, objective direction, variable domains, bounds, constraint families, and executable or LP artifacts. Infeasible, degenerate, unstable, or duplicated seeds are filtered before language generation.
Canonical and semantic intermediate representations make it possible to vary domains, entities, objectives, variable families, constraints, numerical ranges, and data shapes without copying the source problem. Samples are rendered as data-in-problem (DP), data-in-table (DT), or data-problem-separate (DPS).
Accepted documents pass natural-language quality checks, contract validation, static checks, independent execution, and objective comparison. Agreement on one numerical optimum is not treated as proof of symbolic equivalence, so structural checks remain necessary.
TRAINING DESIGN
The training experiment is designed around transfer. A checkpoint is useful when it models the OR corpus better, remains numerically stable, retains general capability, and improves a later SFT run under matched conditions. This makes the CPT-to-SFT comparison more informative than comparing unrelated recipes.
The CPT corpus combines solver-verified OR documents with collected OR resources, mathematical reasoning data, code data, and general English. The roles are deliberately different. OR documents provide formulation semantics and structural modeling patterns; mathematical data supports symbolic derivation and reformulation; code data supports executable solver implementation; general English protects broad language understanding. The report therefore treats mixture composition as a recipe parameter rather than assuming that pure domain data is always better.
The practical lesson is important for reproduction. A domain-specialized model should not be selected from OR validation loss alone. Intermediate checkpoints are checked on held-out OR likelihood, general validation signals, representative general benchmarks, and downstream OR transfer. The best checkpoint is the one that gives a useful balance across these objectives and continues to improve the next training stage.
The controlled comparison uses the same Clean-CoT SFT data, optimizer configuration, update budget, sequence length, and evaluation protocol for two routes: original checkpoint to SFT, and original checkpoint to CPT to SFT. The difference between those final models is therefore interpreted as the contribution of the CPT initialization. This design avoids attributing a gain to CPT when it may actually come from a different SFT dataset or learning-rate schedule.
The report also records a long-running 50K SFT stability run on eight nodes and 128 Ascend 910C NPUs. With global batch size 128, micro batch size 1, and sequence length 8192, the run reaches 800/800 iterations without skipped or NaN iterations. This is infrastructure evidence rather than the main quality result, but it establishes that the data conversion, MindSpeed-LLM launcher, model implementation, and Ascend environment can sustain the workflow.
03 / RESULTS
Matched-condition evaluation on DeepSeek-V4-Flash shows that CPT initialization adds to Clean-CoT SFT across all four zero-shot Pass@1 cells.
| Model | NL4OPT | OptiBench | B4O-Feasible | B4O-ORGEval | Average |
|---|---|---|---|---|---|
| DeepSeek-V4-Flash | 84.08 | 63.33 | 60.47 | 34.26 | 60.54 |
| SFT-Clean-CoT | 86.93 | 64.17 | 65.93 | 48.73 | 66.44 |
| SLAI T-Rex | 89.52 | 67.12 | 71.22 | 59.39 | 71.81 |
NL4OPT and OptiBench test natural-language-to-optimization modeling and problem understanding. B4O-Feasible checks whether the generated Gurobi program executes and produces a feasible solution. B4O-ORGEval goes one layer deeper: it evaluates structural equivalence between the generated formulation and the reference model, independently of implementation details.
Under matched SFT conditions, CPT initialization improves all four zero-shot Pass@1 scores, with the largest additional gain on B4O-ORGEval. The broader mixture also retains general capability: the report finds no sign of catastrophic forgetting, while pure-OR CPT causes a substantial drop on general benchmarks.
DATA QUALITY / SFT
The report's SFT scaling experiments show a non-monotonic pattern. Increasing a similar self-distilled dataset from 10K to 50K does not automatically improve every task. The more reliable improvement comes from identifying concrete modeling mistakes, repairing them under a contract, and validating the repaired answer before export.
The cleaning targets are practical rather than stylistic. They include wrong variable types, wrong objective direction, missing capacity or demand constraints, incorrect ratio or average handling, invalid nonlinear forms, unsafe data access, and incorrect Gurobi API calls. The goal is not to make every answer longer. It is to make the modeling checkpoints visible and repeatable: choose variable domains, determine the objective, check the critical constraints, handle ratios or nonlinear terms, then implement and execute the solver model.
Clean-only repairs the answer while preserving the original reasoning style as much as possible. Clean-CoT additionally inserts a short modeling checklist into tagged responses. A cleaner proposes a repair, an independent reviewer checks whether the repair is justified, local validation executes the code and checks the output contract, and export checking verifies the final artifact. The reviewer is important because an apparently helpful repair can introduce a new inconsistency, such as turning a continuous quantity into an integer or deleting a necessary constraint.
| Branch | Input samples | Reviewer passed | Exported after validation | What is added |
|---|---|---|---|---|
| Clean-only | 10,000 | 9,043 | 8,731 | Repairs modeling and implementation errors |
| Clean-CoT | 10,000 | 9,174 | 8,881 | Repairs errors and adds a concise modeling checklist |
The report identifies a tension between solver-API syntax and natural-language modeling. A single SFT stage can let gradients from code-only targets and reasoning-rich targets interfere with one another. The proposed progressive recipe separates them. Stage I uses code-only OR targets with a conservative 5e-7 learning rate to stabilize Gurobi calls, data loading, and output contracts. Stage II continues from that checkpoint on the complete Clean-CoT 10K set with a 1e-6 learning rate, so the model can learn richer formulation behavior without losing the syntactic backbone.
The same principle applies to general capability retention. The report adds 10% to 15% general-purpose instruction data in the later SFT stage, including open-domain question answering, multi-turn dialogue, and creative writing. In CPT, the analogous lesson is to interleave OR data with mathematical, code, and general-language support data. Pure OR CPT causes a 10 to 15 point drop on general benchmarks without improving the overall OR score, so domain adaptation needs an anchor rather than an all-domain-only mixture.
SCALE-UP VALIDATION
DeepSeek-V4-Flash is the main controlled platform for studying CPT initialization and SFT data quality. DeepSeek-V4-Pro is then used to test whether the complete OR-oriented workflow remains effective at trillion-parameter scale. The Pro result is a scale-up validation of the full pipeline, not an isolated CPT ablation.
On DeepSeek-V4-Pro, the original model starts at an average OR score of 70.16% under the reported zero-shot Pass@4 setting. SFT alone raises the average to 76.74%, while the complete SLAI T-Rex-Pro configuration reaches 77.33%. The strongest absolute movement remains on B4O-ORGEval, which rises from 43.91% for the original model to 61.93% for the complete configuration. OptiBench changes only slightly, which is a useful reminder that a single aggregate number hides different task sensitivities.
| Model | NL4OPT | OptiBench | B4O-Feasible | B4O-ORGEval | Overall | Gain |
|---|---|---|---|---|---|---|
| DeepSeek-V4-Pro | 86.51 | 70.00 | 80.23 | 43.91 | 70.16 | -- |
| DeepSeek-V4-Pro + SFT | 89.97 | 69.50 | 86.04 | 61.43 | 76.74 | +6.58 |
| SLAI T-Rex-Pro | 92.04 | 70.17 | 85.17 | 61.93 | 77.33 | +7.17 |
The general-capability check is deliberately reported alongside the OR result. On the Pro scale-up, SLAI T-Rex-Pro improves AIME 2024 from 83.33% to 90.00%, AIME 2025 from 70.00% to 86.67%, and LiveCodeBench from 55.00% to 67.00%. HumanEval remains at 100%; CMMLU decreases slightly from 93.88% to 93.12%. These results do not prove that forgetting is impossible, but they show why retention metrics must be part of checkpoint selection.
LIMITATIONS AND NEXT STEPS
The report is careful about what its validation proves. Matching an objective value on one numerical instance is a strong executable-consistency signal, but it is not a proof that two formulations are symbolically equivalent. Structural validation remains necessary, especially for canonical graph checks, nonlinear reformulations, ratio constraints, and quadratic terms.
The current release therefore exposes the pieces needed for inspection and reproduction while leaving room for stronger evaluators and broader data. Future work includes more diverse solver-verified corpora, better structural checks for nonlinear models, broader OR families, retention-aware curricula, and agentic reinforcement-learning workflows built on top of the verified training stack.
QUICK START
This is the shortest complete path through the public release. It first prepares a matched Ascend runtime, then converts a base checkpoint and CPT corpus, runs the public Flash training launchers, exports the final MCore checkpoint for serving, and sends an OpenAI-compatible endpoint through the four evaluation targets.
The commands below use placeholder mount points so that the page can be copied to another cluster without carrying private paths. Replace model, dataset, output, master-node, and endpoint values with the paths and service names in your environment. Multi-node jobs must run the same command on every node with the appropriate NODE_RANK.
The published image contains the pinned CANN, MindSpeed, MindSpeed-LLM, Megatron-LM, and Ascend operator environment. Pulling it is the fastest way to reproduce the public setup. Building from source is useful when you need to change versions or rebuild the kernels.
git clone https://github.com/SLAI-AITP/SLAI-T-Rex.git
cd SLAI-T-Rex
# Option A: use the published image
docker pull quay.io/slai-t-rex/slai-t-rex:v1.0.0-a3-cann9.1.0
# Option B: build on a Linux AArch64 Ascend build host
bash docker/build_image.sh
# Mount the repository and external artifacts in the runtime selected by
# your cluster scheduler. Device flags vary by deployment environment.
docker run --rm -it --network host --ipc=host --privileged \
-v "$PWD":/workspace/SLAI-T-Rex-src \
-v /path/to/models:/data/models \
-v /path/to/datasets:/data/datasets \
-v /path/to/outputs:/data/outputs \
quay.io/slai-t-rex/slai-t-rex:v1.0.0-a3-cann9.1.0 bashThe public Flash launchers expect an MCore checkpoint. When the starting model is an FP8 HuggingFace checkpoint, dequantize it to BF16 first, then convert BF16 HuggingFace weights to the MCore layout. The tokenizer path can point to the BF16 HuggingFace directory if it contains the tokenizer files.
export MINDSPEED_LLM_DIR=/workspace/MindSpeed-LLM
export TOKENIZER_PATH=/data/models/deepseek4_flash_bf16
export INPUT_FP8_HF_PATH=/data/models/deepseek4_flash_fp8
export OUTPUT_BF16_HF_PATH=/data/models/deepseek4_flash_bf16
bash /workspace/SLAI-T-Rex-src/training/convert/convert_ckpt_fp8_to_bf16.sh
export CONVERT_SCRIPT=/workspace/MindSpeed-LLM/convert_ckpt_v2.py
export HF_LOAD_DIR=/data/models/deepseek4_flash_bf16
export MCORE_SAVE_DIR=/data/models/deepseek4_flash_mcore
bash /workspace/SLAI-T-Rex-src/training/convert/convert_ckpt_hf_to_mcore.shCPT input is JSONL with one plain text document per row, for example {"text": "..."}. The generic conversion helper invokes the MindSpeed-LLM preprocessor, supports multiple workers and sub-processes, and emits the indexed text-document pair used by the CPT launcher.
cd /workspace/SLAI-T-Rex-src
bash training/convert/convert_data.sh \
--mindspeed-llm-dir "$MINDSPEED_LLM_DIR" \
--input /data/datasets/or_cpt_corpus.jsonl \
--output-prefix /data/datasets/processed/or_cpt_corpus \
--tokenizer "$TOKENIZER_PATH" \
--handler-name GeneralPretrainHandler \
--json-keys text \
--seq-length 4096 \
--workers 8 \
--n-subs 16
# Output:
# /data/datasets/processed/or_cpt_corpus_text_document.bin
# /data/datasets/processed/or_cpt_corpus_text_document.idxSet the weighted indexed dataset list and the cluster topology before launching. The public 4K Flash CPT template defaults to global batch size 128, micro batch size 1, tensor parallel 1, pipeline parallel 4, and expert parallel 32. On each node, keep the shared values identical and set NODE_RANK to that node's rank.
cd /workspace/SLAI-T-Rex-src
export MINDSPEED_LLM_DIR=/workspace/MindSpeed-LLM
export MINDSPEED_DIR=/workspace/MindSpeed
export MEGATRON_DIR=/workspace/Megatron-LM
export TOKENIZER_PATH=/data/models/deepseek4_flash_bf16
export CKPT_LOAD_DIR=/data/models/deepseek4_flash_mcore
export OUTPUT_ROOT=/data/outputs/slai_trex_cpt
export TRAIN_DATA_PATH=$'1.0 /data/datasets/processed/or_cpt_corpus_text_document'
export VALID_DATA_PATH=/data/datasets/processed/validation/val_text_document
export NNODES=8
export NPUS_PER_NODE=16
export MASTER_ADDR=MASTER_NODE_HOST
export MASTER_PORT=8192
export NODE_RANK=0
bash training/cpt/train_cpt_deepseek4_flash_4k.shThe repository includes a packed public SFT dataset under data/sft/datasets/or_sft. DATA_PATH is the prefix before the packed field names, so the example uses .../or_sft rather than .../or_sft_packed. The launcher can prepare the distributed index map once on rank 0 and make other nodes wait for the ready marker.
cd /workspace/SLAI-T-Rex-src
export MINDSPEED_LLM_DIR=/workspace/MindSpeed-LLM
export MINDSPEED_DIR=/workspace/MindSpeed
export MEGATRON_DIR=/workspace/Megatron-LM
export TOKENIZER_PATH=/data/models/deepseek4_flash_bf16
export CKPT_LOAD_DIR=/data/outputs/slai_trex_cpt/checkpoints/your_cpt_run
export OUTPUT_ROOT=/data/outputs/slai_trex_sft
export DATA_PATH=/workspace/SLAI-T-Rex-src/data/sft/datasets/or_sft/or_sft
export PREPARE_INDEXMAP=1
export NNODES=8
export NPUS_PER_NODE=16
export MASTER_ADDR=MASTER_NODE_HOST
export MASTER_PORT=8192
export NODE_RANK=0
bash training/sft/launch_sft_deepseek4_flash_8n16_910c.shAfter SFT, convert the selected MCore checkpoint back to HuggingFace format and serve it through an OpenAI-compatible endpoint. The evaluation launcher runs NL4OPT, OptiBench, B4O-Feasible, and B4O-ORGEval together in 1-pass mode. A Gurobi installation and valid license are required for the solver-backed evaluations.
export CONVERT_SCRIPT=/workspace/MindSpeed-LLM/convert_ckpt_v2.py
export MCORE_LOAD_DIR=/data/outputs/slai_trex_sft/checkpoints/your_sft_run
export HF_SAVE_DIR=/data/outputs/slai_trex_sft/hf
bash /workspace/SLAI-T-Rex-src/training/convert/convert_ckpt_mcore_to_hf.sh
# Start your model service separately, then evaluate all four OR targets.
cd /workspace/SLAI-T-Rex-src/eval
python3 -m pip install -r requirements.txt
export OPENAI_BASE_URL=http://MODEL_SERVICE_HOST:8000/v1
export MODEL_NAME=your-served-model-name
export OPENAI_API_KEY=none
BACKGROUND=false bash scripts/run_eval_1pass.sh
# Results: eval/results/<run-name>/
# Logs: eval/logs/full_<run-name>/For a first smoke test, use the packed SFT dataset already included in the repository and a small evaluation range. For a real CPT run, prepare a held-out validation prefix in addition to the training corpus. For custom SFT data, generate and validate the OpenAI-style JSONL with data/sft, then produce the packed input_ids, labels, and attention_mask files expected by the SFT launcher.
04 / THE STACK
The repository is organized around the reproducibility boundary: data preparation, training, acceleration, conversion, and evaluation are public; private cluster paths and production artifacts stay outside the release.
05 / RESOURCES
Read the technical report for the full system and modeling story, then use the repository modules and prebuilt Ascend image to reproduce the public workflow.
quay.io/slai-t-rex/slai-t-rex:v1.0.0-a3-cann9.1.0