Replies: 17 comments 125 replies
-
|
I'm happy to support moving the repo into the bytedeco org (or into a dedicated org if that makes more sense). I could also transfer the original repo currently under my personal namespace once it's up to date, as otherwise we would have a forked repo which is often treated as a second class citizen by GitHub compared to a non-forked one. I haven't had the time to really develop Storch further this year, and I doubt this will change much in the near future due other responsibilities and unforeseen things happening in my life. That said, I'll try to help as much as possible. Storch has come a long way and you've put an amazing amount of work into it @mullerhai! |
Beta Was this translation helpful? Give feedback.
-
|
@sbrunk If it's OK with you, please do transfer your repository to @bytedeco. We can update later as necessary. Thanks! @mullerhai Well, if we can find sponsors that sounds great! |
Beta Was this translation helpful? Give feedback.
-
@saudet @sbrunk ，wait we transfer the whole latest storch repo to bytedeco ，I think I will try to introduce storch as bytedeco organization incubate top level【🎖️】？project ，and let these company support us， these days I am prepare for Pytorch on Scala3 Online lesson on Youtube and Bilibili【China】，I think we could make big impact for world |
Beta Was this translation helpful? Give feedback.
-
|
I've just transferred the original repo, which is now at: https://github.com/bytedeco/storch Now we can look into getting the changes from bytedeco/storch#88 merged. |
Beta Was this translation helpful? Give feedback.
-
Perfectly! let me do more fix and clean code ,then make a pr |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
@saudet today I have talk with HUAWEI company AI leader Mr.Cheng , He is very interesting with Storch and javacpp-pytorch. it we could develop CANN[ like cuda] and [TensorRT-llm] with javacpp-pytorch. they want to support us |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for feedback! Yes, this is something that we can do. What would that support look like? |
Beta Was this translation helpful? Give feedback.
-
some Money sponsor and Gpu pc ，but gpu is HUAWEI NPU shengteng |
Beta Was this translation helpful? Give feedback.
-
|
This sounds interesting. The easiest thing for me is if they could become a sponsor through GitHub here: |
Beta Was this translation helpful? Give feedback.
-
|
Hi, @saudet @sbrunk , I want to transfer Storch-numpy Storch-pandas ,storch-opencv ,storch-ffmpeg to Bytedeco repo, to make more people use them . storch-opencv ,storch-ffmpeg is base on javacpp-opencv and javacpp-ffmpeg. how do you think ? |
Beta Was this translation helpful? Give feedback.
-
|
Sure, why not? |
Beta Was this translation helpful? Give feedback.
-
|
@sbrunk I have update storch-core code to the version 0.7.5-1.5.12 , you could check and validate review the code , in my test on pytorch_on_scala3_lesson is could correctly work |
Beta Was this translation helpful? Give feedback.
-
ohh，you know wechat？maybe use wechat or Tencent Meeting ，but they want to meet us in Shenzhen or Beijing to talk about storch and CANN [HuaWei cuda], the develop scheduler maybe is first generate Huawei CANN java glue layer code from cpp ,and try to suit with Huawei NPU clips, We could serach [pytorch-npu] it is build pytorch for huawei clips use python. second , storch invoke CANN Java glue layer code to run huawei gpu. finally they will use storch for Huawei cloud machine pc |
Beta Was this translation helpful? Give feedback.
-
|
I'm fine doing remote meetings, but I'm not going to go to China for that for now anyway. You could go in person, and others like me can join remotely though. |
Beta Was this translation helpful? Give feedback.
-
@saudet If Huawei makes any new progress, I will share the latest updates with you. I will continue to communicate and follow up with them. Currently, I've asked them to internally evaluate the suitability of javacpp-pytorch and Storch. However, it's important to note that Huawei is just one company. While their chip market share in China is currently around 5%, NVIDIA's CUDA still dominates with approximately 90% market share. Regardless, we must prioritize robust support for NVIDIA CUDA and NVIDIA NCCL. First, When do you plan to release javacpp-pytorch 2.9.0-1.5.13? These questions are not specifically aimed at the Huawei collaboration but rather to enhance the competitiveness of our project." |
Beta Was this translation helpful? Give feedback.
-
|
Regardless of what happens with CUDA and NVIDIA in China, I'm not aware of anyone anywhere doing distributed training on the Java platform. If Huawei wants to do that, that's interesting, but we're going to need a team to basically port all the tools required from the Python platform. NCCL is only a very small piece. But it's something you could ask them for starters. Could Huawei have one of their engineers spend some time to add support for NCCL to PyTorch on Java? |
Beta Was this translation helpful? Give feedback.
-
yeah ，no one use java for distribute training llm ，but just we want to make first ，it is should us aim，or we can not instead of python ai env，Huawei chips CANN gpu driven it also support distribute training |
Beta Was this translation helpful? Give feedback.
-
|
@sbrunk Hi Sbrunk, |
Beta Was this translation helpful? Give feedback.
-
|
Hi @saudet @sbrunk ,good news, I attended Huawei's chip-driven CANN salon meetup in Beijing today 2025-11-15. The event was vibrant and engaging, with many Chinese companies already having completed adaptations with them. I communicated with the event organizer—their next salon will feature presentations on javacpp-pytorch and storch, and I’ll probably get a 10-minute slot to speak. Most of the participating companies today still primarily use C++ and Python, and the venue was quite crowded. Their current research depth has reached the level of transformer Attention Matrix multiplication, tile blocking, and CUDA kernel optimization—essentially focusing on matrix blocking strategies, more efficient model inference communication across multiple GPUs and machines, and so on. Compared to them, our innovative javacpp-pytorch and storch are still in the shallow end, not yet diving into the kernel layer. We have a lot of work ahead. They’ll contact me next week to discuss the details of cooperation between CANN and Bytedeco. Do you have any questions? I’ll ask Huawei for their thoughts about your behalf. the CANN repo https://gitcode.com/cann |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the update! I guess my only question would be, given that we would need to invest a lot of effort, time and money to get all this working with Java, what are the use cases that are worth it? Why not just keep doing those tasks with Python? |
Beta Was this translation helpful? Give feedback.
-
|
I'm writing to provide a quick update on my communication with an engineering team from Huawei's chip division. The conversation was very positive, and they are very interested in collaborating with us. They are currently working on an internal budget and assessment, with a projected start for our official collaboration in 2026. As a first step, they want to evaluate the performance of Storch. I've already demonstrated some of its capabilities to them, and they were impressed, noting that it is very close to Python PyTorch. They then asked for a demonstration of Storch's training capabilities with a GPU. To show this, I had to demonstrate Storch integrated with CUDA. However, I've run into a significant technical issue. Initially, I tried on my Windows 11 machine with an NVIDIA 3060 GPU, but javacpp-pytorch was unable to communicate with CUDA. I then switched to a different laptop running Ubuntu with an NVIDIA 4060 GPU. I had to uninstall CUDA 12.4, as this machine had CUDA 13.0 installed. Even with the Ubuntu setup, javacpp-pytorch (version 2.7.1-1.5.12) still fails to recognize and use CUDA properly. Because of this, I had to reschedule the demonstration with them for either next week or in about two weeks. This is very time-sensitive for us, as the Huawei partnership is a key opportunity. Therefore, I am hoping a stable release of javacpp-pytorch 2.9-1.5.13 can be made available soon. This would allow me to use javacpp-cuda 1.3.0-1.5.13 to successfully demonstrate Storch's CUDA capabilities to them. Summary of Issues and Requests: Urgent Need for New Release: I am encountering CUDA recognition issues on Ubuntu with the current version (2.7.1-1.5.12). Your prompt release of version 1.5.13 would be crucial for this partnership. |
Beta Was this translation helpful? Give feedback.
-
|
What is the error message you're getting when you try to use the snapshots? |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
Please try to use the precompiled binaries |
Beta Was this translation helpful? Give feedback.
-
[INFO] Targeting /home/muller/Documents/code/javacpp-presets/pytorch/src/main/java/../../gen/java/org/bytedeco/pytorch/AllToAllOptions.java |
Beta Was this translation helpful? Give feedback.
-
|
Please try to build the pom.xml file in the samples directory here: |
Beta Was this translation helpful? Give feedback.
-
|
(base) muller@muller-Dell-G16-7630:~/Documents/code/javacpp-presets/pytorch$ mvn --version jdk 21 |
Beta Was this translation helpful? Give feedback.
-
|
hi, @saudet ,huawei company Ai engineer tell me ,they also want to try javacpp-pytorch with cuda 1.13.0 ,they are write python ,they don't want to like me build the all project ,just want to make dependency the jar, I think you need really release and publish to maven center repo |
Beta Was this translation helpful? Give feedback.
-
|
What error do you get with the pom.xml file in this samples directory? |
Beta Was this translation helpful? Give feedback.
-
|
@saudet the javacpp-pytorch all part build is so slowly and complex and often meet error , not suit for all people ,please consider ,make easier for most beginer |
Beta Was this translation helpful? Give feedback.
-
|
What error do you get with the pom.xml file in this samples directory? |
Beta Was this translation helpful? Give feedback.
-
(base) muller@muller-Dell-G16-7630: |
Beta Was this translation helpful? Give feedback.
-
|
Please follow the instructions here and try again: |
Beta Was this translation helpful? Give feedback.
-
|
the Huawei company ai engineer is eagerly try to use javacpp-pytorch with cuda 1.13.0, two choices,publish 1.5.13 snapshot version all to maven snap repo ,please not build by user , another release 1.5.13 version to maven center repo this week. |
Beta Was this translation helpful? Give feedback.
-
this RPC module is important, HUAWEI cloud user need use parameter server to do online training ,we need implement RPC |
Beta Was this translation helpful? Give feedback.
-
|
for EmbeddingImpl EmbeddingBagImpl both not found Embedding::from_pretrained , only have Options for the Embedding::from_pretrained ,does from_pretrained method implement by javacpp is hard ?@saudet. for llm we really need it |
Beta Was this translation helpful? Give feedback.
-
|
ProcessGroupNCCL I think we need implement in new version, I think the gloo problem meet is similar with nccl , |
Beta Was this translation helpful? Give feedback.
-
|
@saudet ,Mnist really have bug, I let other people try to load the dataset, also error |
Beta Was this translation helpful? Give feedback.
-
|
The MNIST dataset works just fine for me. I can't fix what isn't broken! As for the rest, I'm sure we can get everything running as per the C++ API. We just need someone to spend enough time on it. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @saudet ,I have write one gloo example with pytorch , after javacpp-pytorch gloo could work fun, I will implement these code in scala3, these code also like https://github.com/pytorch/examples/blob/main/cpp/distributed/dist-mnist.cpp we need to open two terminal windows then the output |
Beta Was this translation helpful? Give feedback.
-
|
I've added FileStore and TCPStore in commit bbf8c5b so please try again with that. |
Beta Was this translation helpful? Give feedback.
-
perfect I will try |
Beta Was this translation helpful? Give feedback.
-
oh, I delete the pytorch snapshot jar on my disk ,then I try to make new pytorch snapshot jar with mvn compile cmd, but download file is OT/pytorch-2.9.0-1.5.13-20251023.024851-17-* version ? have you publish the new commit and deploy new jar to the maven snapshot repo ? and I see the commit log you add two line in pytorch/src/main/java/org/bytedeco/pytorch/presets/torch.java file to compile so could you add ProcessGroupGlooCuda.cpp |
Beta Was this translation helpful? Give feedback.
-
|
NCCL isn't available on Windows, so that makes the build for Linux more complicated, but let me see... |
Beta Was this translation helpful? Give feedback.
-
Let's not hesitate; let's work and push to make it happen. I firmly believe the goal for javacpp-pytorch is to build an enterprise-grade framework. Now, it's rare for anyone to deploy deep learning frameworks on Windows. The deployment is almost exclusively done on Linux distributions, including: Debian, Ubuntu, Fedora, CentOS, and Red Hat. This is precisely the main battleground where NCCL shines. Alternatively, we could issue an additional declaration to users that certain classes are not available on Windows. This is determined by the underlying CUDA dependencies, which hopefully,maybe NVIDIA will support nccl on windows one day. NVIDIA, in fact, doesn't support Windows for many of its tools, including their own Spark DataFrame plugin called cuDF. This isn't our fault, but it means we should maximize the effectiveness of javacpp-pytorch in an enterprise Linux environment. I recently reached out to a tool called executorch, hoping they would support Linux with java, but they only support Android on the JVM. This is a significant drawback for its growth. On another front, we are working on implementing some of the APIs for huggingface Transformers model api and VLLM apinatively in Java. In the future, this will allow for seamless interaction using javacpp-pyorch's tensors. Also, regarding the pytorch.distributed.rpc module: could we consider implementing it? This would give us an additional channel for deploying distributed models for user invoke,will make more powerful. I have read Huawei NPU PYTORCH code, they also use pytorch rpc to inference and deploy the model Thank you for your hard work and consideration. |
Beta Was this translation helpful? Give feedback.
-
|
I write a gloo cuda example pytorch , it could work ,I think future in scala3 we just need implement like this @saudet |
Beta Was this translation helpful? Give feedback.
-
|
I've added FileStore and TCPStore in commit bbf8c5b so please try again with that. |
Beta Was this translation helpful? Give feedback.
-
|
@saudet I'm scheduled to give another demonstration to Huawei this weekend with the new javacpp-pytorch version 2.9.1-1.5.13, but gloo verification keeps failing, which is making me extremely anxious. Currently, the ProcessGroupGloo creation is failing and causing a JVM crash. I'm not sure if it's a bug in the software itself, due to my incorrect usage, or if I'm passing parameters incorrectly. the final I try ,also failed |
Beta Was this translation helpful? Give feedback.
-
the output error |
Beta Was this translation helpful? Give feedback.
-
|
after I change FileStore also can't connect to the rank java code output |
Beta Was this translation helpful? Give feedback.
-
|
I think there have some tiny bug , some logical code sometime could run , sometime cannot run ,get |
Beta Was this translation helpful? Give feedback.
-
|
There are at least issues with the wrappers for c10::intrusive_ptr. I'll let you know if I manage to fix them. |
Beta Was this translation helpful? Give feedback.
-
|
I've pushed a fix for c10::intrusive_ptr in commit a0dca3b and the following code doesn't crash anymore at least, so please try again with this update. FileStore store = new FileStore("/tmp/foo", 2);
ProcessGroupGloo.Options options = ProcessGroupGloo.Options.create();
options.timeout(new Milliseconds(50));
options.devices().push_back(
ProcessGroupGloo.createDeviceForHostname("127.0.0.1"));
ProcessGroupGloo processGroup = new ProcessGroupGloo(store, 1, 2, options); |
Beta Was this translation helpful? Give feedback.
-
|
now try to use TCPStore and FileStore with ProcessGroupGloo, I think if we could make the gloo multi ranks could discover each other ,the gloo maybe could wok , I have try how to set ,but all failed ,maybe is the gloo.Device cause? I am not sure, but the ProcessGroupGloo should already is running . |
Beta Was this translation helpful? Give feedback.
-
|
Please try to translate working C++ code. I can't guess how we're supposed to use the API |
Beta Was this translation helpful? Give feedback.
-
Sorry, I didn't quite understand what you meant. Are you saying that there's no need to fix ProcessGroup or ProcessGroupNCCL? Or is it that we're still unclear on how we'll use these APIs? My understanding is that ProcessGroup seems to be the parent class of ProcessGroupGloo, ProcessGroupCCL, ProcessGroupMPI, and ProcessGroupUCC. Also, ProcessGroup acts as a factory method; by specifying the backend and device, it can delegate to subclasses like ProcessGroupGLOO and ProcessGroupNCCL to run. I imagine that |
Beta Was this translation helpful? Give feedback.
-
|
I mean, for example, I asked Gemini, "Write a very small application that demonstrates how to use c10d::ProcessGroupGloo from the C++ API of PyTorch using CUDA as backend", and it came up with the following. It doesn't call setBackend(), so I don't think we need to call setBackend(), among other things. #include <torch/torch.h>
#include <c10d/GlooProcessGroup.hpp>
#include <c10d/FileStore.hpp>
#include <c10d/ProcessGroup.hpp>
#include <iostream>
#include <memory>
#include <chrono>
// Helper function to create the communication store and process group
std::shared_ptr<c10d::ProcessGroup> init_gloo_pg(int rank, int world_size, const std::string& master_path) {
// 1. Initialize the shared store (using FileStore for simplicity)
// FileStore uses a shared file path for synchronization. In a real multi-node
// setting, you would use c10d::TCPStore for network-based coordination.
auto store = std::make_shared<c10d::FileStore>(master_path, rank);
// 2. Set up Gloo Process Group Options
auto options = c10d::GlooOptions::create();
options->timeout = std::chrono::seconds(10); // Set a reasonable timeout
// 3. Create the Gloo Process Group
// Gloo handles the actual low-level communication and synchronization.
// Since the tensors are on CUDA, Gloo manages the peer-to-peer data transfers on the GPU.
return std::make_shared<c10d::ProcessGroupGloo>(
store,
rank,
world_size,
std::move(options)
);
}
int main(int argc, char* argv[]) {
// --- Argument Parsing ---
if (argc != 3) {
std::cerr << "Usage: " << argv[0] << " <RANK> <WORLD_SIZE>" << std::endl;
std::cerr << "Example for Rank 0: " << argv[0] << " 0 2" << std::endl;
return 1;
}
const int rank = std::stoi(argv[1]);
const int world_size = std::stoi(argv[2]);
// This file path must be accessible by both processes (e.g., in /tmp/ or an NFS mount)
const std::string master_file_path = "/tmp/pytorch_gloo_store";
std::cout << "Starting process [Rank " << rank << "/" << world_size << "]" << std::endl;
// --- CUDA Device Check ---
if (!torch::cuda::is_available()) {
std::cerr << "CUDA is NOT available. This example requires a CUDA device." << std::endl;
return 1;
}
// Use the first CUDA device
torch::Device device(torch::kCUDA, 0);
// --- 1. Initialize Gloo Process Group ---
auto process_group = init_gloo_pg(rank, world_size, master_file_path);
std::cout << "Process Group initialized successfully." << std::endl;
// --- 2. Create CUDA Tensor ---
// Create a tensor on the CUDA device, initialized based on the rank.
torch::Tensor tensor = torch::ones({2, 2}, device) * (rank + 1);
std::cout << "\nRank " << rank << " initial tensor on CUDA:\n" << tensor << std::endl;
// Rank 0 starts with: [[1, 1], [1, 1]]
// Rank 1 starts with: [[2, 2], [2, 2]]
// --- 3. Perform AllReduce Collective Operation ---
// AllReduce sums all tensors and sends the result back to all ranks.
// The operation is performed on the CUDA device.
std::vector<torch::Tensor> tensors_to_reduce = {tensor};
c10d::ProcessGroup::Options allreduce_opts(c10d::OpType::SUM);
// Start the collective operation
auto work = process_group->allreduce(tensors_to_reduce, allreduce_opts);
// Wait for the collective operation to complete
work->wait();
// --- 4. Verify Result ---
// Expected result (1 + 2 = 3): [[3, 3], [3, 3]]
std::cout << "\nRank " << rank << " final tensor after AllReduce (SUM):\n" << tensor << std::endl;
// Verification check
torch::Tensor expected_result = torch::ones({2, 2}, device) * (1 + world_size);
if (tensor.equal(expected_result)) {
std::cout << "\nRank " << rank << ": SUCCESS! AllReduce result is correct." << std::endl;
} else {
std::cerr << "\nRank " << rank << ": ERROR! AllReduce result is incorrect." << std::endl;
}
return 0;
} |
Beta Was this translation helpful? Give feedback.
-
I think you should be clear that in the init_gloo_pg method within the Gemini-generated code you provided, you actually used ProcessGroupGloo to create an instance of the parent class ProcessGroup, which is equivalent to an internal operation, essentially calling setBackend(torch.DeviceType.cpu, ProcessGroup.BackendType.GLOO, BackendOptional). I have also tested with Gemini, and I believe it can have hallucinations; I couldn't find any direct C++ usage of creating a gloo instance with ProcessGroup on GitHub, as it's mostly done with ProcessGroupGloo, ProcessGroupMPI, or ProcessGroupNCCL. However, I did see in C++ that ProcessGroup is a parent class with factory methods to create gloo and NCCL subclasses. Now, I believe that python pytorch's dist.init_process_group(backend, rank=rank, world_size=world_size) is a call to the ProcessGroup or Backend factory method. |
Beta Was this translation helpful? Give feedback.
-
HI @saudet ，I review my test code and try to run it, Today, I obtained some important error logs; we can analyze the reasons for the connection rejection together. I'm not sure if it's caused by my local computer, the code library, or the way I wrote the code example. I've checked my computer, and the port should not be in use. The VPN is also turned off, which suggests that some of its dependent tools may not have started, and the port is not taking effect. ProcessGroupGloo likely still requires some adjustments and checks. output 3ranks the javacode |
Beta Was this translation helpful? Give feedback.
-
|
I agree that gloo distributed training will be a key challenge we need to tackle before our new version release. After resolving the distributed communication and mutual rank discovery issues, gloo should be able to function normally. The error log shows "connection refused." I suggest you could run the Java code I wrote, starting three separate terminals and specifying rank=0, rank=1, and rank=2 respectively, to see what results you get. If you also get a connection refused error, it might be a problem with the javacpp-pytorch library; if the connection is successful, then it's an issue with my computer's network configuration. I tried again today and still couldn't connect on my computer. |
Beta Was this translation helpful? Give feedback.
-
Perhaps what you said is also correct; it might be that I'm using these APIs incorrectly. I haven't set up a C++ LibTorch environment on my computer yet, but I can try to set it up. However, I'm a bit rusty with C++. If I get it set up and try running some code, also, the PyTorch on GitHub indeed doesn't provide any usable gloo usage examples, but there seems to be a test file called However, it has to be admitted that the normal implementation and operation of Gloo in Python PyTorch definitely rely on Gloo in C++ LibTorch. Currently, though, we are not quite sure about the specific operations and configurations made in Python that enable Gloo to run properly. Actually, I would prefer it to be a configuration issue, but the failure of TCP socket connection should not be caused by the too-short timeout setting. Even when I set the timeout to 100 times the original value, it still remains in the waiting phase. It should be noted that this "wait allreduce" message is the print output I set myself, not from Gloo. Personally, I think the incorrect TCP socket connection is still the main cause, but I am not sure about the exact reason. [W1201 14:32:51.794314750 socket.cpp:472] [c10d] waitForInput: poll for socket SocketImpl(fd=80, addr=[localhost]:36124, remote=[localhost]:12455) returned 0, likely a timeout |
Beta Was this translation helpful? Give feedback.
-
|
FileStore doesn't work either, so there's definitely something wrong with PyTorch when running inside the JVM. That's going to be hard to debug... |
Beta Was this translation helpful? Give feedback.
-
|
Ah, no, after fixing up Gemini's output some more, this works just fine for me: import org.bytedeco.javacpp.*;
import org.bytedeco.javacpp.chrono.*;
import org.bytedeco.pytorch.*;
import org.bytedeco.pytorch.global.torch;
import org.bytedeco.pytorch.global.torch_cuda;
import static org.bytedeco.pytorch.global.gloo.*;
import static org.bytedeco.pytorch.global.torch.*;
import static org.bytedeco.pytorch.global.torch_cuda.*;
public class GlooCudaExample {
public static void main(String[] args) {
// Use a PointerScope to manage memory automatically when objects go out of scope
try (PointerScope scope = new PointerScope()) {
// --- Argument Parsing ---
if (args.length != 2) {
System.err.println("Usage: java GlooCudaExample <RANK> <WORLD_SIZE>");
System.err.println("Example for Rank 0: java GlooCudaExample 0 2");
return;
}
final int rank = Integer.parseInt(args[0]);
final int worldSize = Integer.parseInt(args[1]);
// This file path must be accessible by all processes for synchronization
final String masterFilePath = "/tmp/pytorch_gloo_store";
System.out.println("Starting process [Rank " + rank + "/" + worldSize + "]");
// --- CUDA Device Check ---
if (!torch_cuda.is_available()) {
System.err.println("CUDA is NOT available. This example requires a CUDA device.");
return;
}
// Use the first CUDA device
Device device = new Device(DeviceType.CUDA, (byte)0);
// --- 1. Initialize Gloo Process Group (Conceptual Placeholder) ---
FileStore store = new FileStore(masterFilePath, rank);
ProcessGroupGloo.Options options = ProcessGroupGloo.Options.create();
options.timeout(new Milliseconds(new Seconds(10)));
options.devices().push_back(
ProcessGroupGloo.createDeviceForHostname("127.0.0.1"));
ProcessGroupGloo processGroup = new ProcessGroupGloo(store, rank, worldSize, options);
// --- 2. Create CUDA Tensor ---
// Create a tensor on the CUDA device, initialized based on the rank (rank + 1).
// Note the use of .mul_() for in-place multiplication to match the C++ initialization.
Tensor tensor = torch.ones(new long[]{2, 2}, new TensorOptions(device)).mul_(new Scalar((float)(rank + 1)));
System.out.println("\nRank " + rank + " initial tensor on CUDA:\n" + tensor);
// Rank 0 starts with: [[1.0, 1.0], [1.0, 1.0]]
// Rank 1 starts with: [[2.0, 2.0], [2.0, 2.0]]
// --- 3. Perform AllReduce Collective Operation ---
// AllReduce sums all tensors and sends the result back to all ranks.
// The operation is performed on the CUDA device.
TensorVector tensorsToReduce = new TensorVector(tensor);
AllreduceOptions allreduceOps = new AllreduceOptions().reduceOp(new ReduceOp(ReduceOp.RedOpType.SUM));
// Start the collective operation
Work work = processGroup.allreduce(tensorsToReduce, allreduceOps);
// Wait for the collective operation to complete
work._wait();
// --- 4. Verify Result ---
// Expected result (1 + 2 = 3): [[3.0, 3.0], [3.0, 3.0]]
System.out.println("\nRank " + rank + " final tensor after AllReduce (SUM) Simulation:\n" + tensor);
// Verification check
Tensor expectedResult = torch.ones(new long[]{2, 2}, new TensorOptions(device)).mul_(new Scalar(1 + worldSize));
if (tensor.equal(expectedResult)) {
System.out.println("\nRank " + rank + ": SUCCESS! Simulated AllReduce result is correct (Expected value: " + (1 + worldSize) + ").");
} else {
System.err.println("\nRank " + rank + ": ERROR! Simulated AllReduce result is incorrect.");
}
} catch (Exception e) {
e.printStackTrace();
}
}
} |
Beta Was this translation helpful? Give feedback.
-
|
very nice ,very NB! yeah ,it is really work for me !!!! you are the HERO ,SO PERFECT. Now TcpStore and FileStore all could work. The ProcessGroupGloo is validate pass!!! it is magical , so just my fault that bad gloo use case before |
Beta Was this translation helpful? Give feedback.
-
|
First, it did succeed, with tcpstore and filestore. I looked at my previous code, and it would also succeed if the timeout was set longer. However, it seems that sometimes using filestore, initializing gloo would throw an error. The error was java.lang.RuntimeException: Gloo connectFullMesh failed with timed out connecting: SO_ERROR: Connection refused. But this is related to our programming code and should have nothing to do with javacpp. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @saudet I'm so eager to see the Linux version of ProcessGroupNCCL implemented with javacpp. Regarding suspending gloo and other issues, is it something we can implement? The javacpp pytorch on scala3 meetup will be held in Beijing, China on Saturday, December 13th. I will likely be the second speaker and will join via remote video conference. I'm really keen on recommending to the attendees that they try using javacpp. For the issue they care about most, NCCL, I'm truly looking forward to announcing to them the feasibility of our ProcessGroupNCCL support. This would be a huge draw for them to try javacpp-pytorch. They are currently making the event promotional poster, and I will share it with you all once it's done. I predict that if our implementation is robust enough, they will choose ours. |
Beta Was this translation helpful? Give feedback.
-
I have try to use the mvn compile exec:java to run the nccl example ,also failed, 
|
Beta Was this translation helpful? Give feedback.
-
|
I see, I think I found the issue. This should be fixed with commit 8a1a114. Please download the latest binaries and try again! |
Beta Was this translation helpful? Give feedback.
-
you are right , now nccl is working ,it could init but not sure nccl module all working correctly, I need time to validate , |
Beta Was this translation helpful? Give feedback.
-
|
one thing ,I think HashStore and ChunkSequentialDataLoader need to implement , how do you think |
Beta Was this translation helpful? Give feedback.
-
|
something make me confused , after create processgroupnccl ,but when I invoke ''pgnccl.isInitialized()" get false console log another things ,I think we should solve the processGroup the backendtype can not detect problem console log |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi @saudet
The latest version of Storch is now basically stable. Since all the lessons in the "PyTorch on Scala" course have been successfully completed, I believe I have verified most of the basic functionalities of PyTorch. It’s safe to say that Storch works properly and follows a coding style close to Python’s PyTorch. I highly recommend that everyone learns the Scala 3 version of PyTorch because it appears simpler and allows users to focus more on neural network construction, training, and inference.
After three years of verification and nearly 110,000 lines of Scala code, I can confirm that javacpp-pytorch is indeed an excellent glue layer. However, directly programming with javacpp-pytorch comes with many risks and drawbacks it not easy to use :
first maybe The code is quite verbose, especially when written directly in Java.
second It’s error-prone, with many parameters using types like Pointer and cpp...Vector, and vague names like var1 or var2. Without checking PyTorch’s official documentation, it’s hard to know the exact data types and actual meanings of these parameters.
thrid some Javacpp-pytorch still has some unresolved bugs, such as some padding type layers not working correctly and some hyperparameters failing to take effect use put() method .
Since Python’s PyTorch extends libtorch significantly, many features implemented in Python PyTorch are not available in libtorch, making it impossible for javacpp-pytorch to generate corresponding bindings.
Storch addresses most of these risks and issues, greatly improving the efficiency of using PyTorch. Based on the above, I believe that to promote more widespread use of PyTorch on the JVM platform, it’s better to encourage everyone to try Storch, which is built on javacpp-pytorch. What do you think?
Of course, I also anticipate that there might be a Java-based wrapper library for javacpp-pytorch called JTorch in the future. If necessary, I would like to transfer my extended Storch repository (based on @sbrunk Storch main branch code ) to the bytedeco organization. I also plan to consult @sbrunk sbrunk’s opinions, as I believe hosting it under bytedeco would attract more contributors to optimize Storch and foster ecosystem growth.
On another note, we have high expectations for the 1.5.13 version of pytorch. Looking through the code, I noticed some new additions:
For ProcessGroupGloo, new methods like:
The new class ProcessGroupStatus.
Additions to the global torch class:
I believe these updates show that Gloo support is quite robust. However, based on my understanding of large model training, most users still rely on NCCL-based distributed frameworks with DDP and DSFP modes. Therefore, I would like to request adding the following includes to the presets/torch_include.h generation script:
This should generate corresponding bindings after compilation. I understand this is challenging and may not produce correct code immediately, requiring further debugging. However, without ProcessGroupNCLL, we can only work on small single-machine models, making it difficult to optimize model quantization and CUDA memory usage. I sincerely hope that the 1.5.13 version of javacpp-pytorch will include functional ProcessGroupNCLL, enabling us to develop libraries like DeepSpeed, FairScale, or Colossal-AI to accelerate distributed training. I’d like to hear your thoughts and current challenges—I’m willing to assist with implementation.
Additionally, could the EmbeddingImpl layer include the Embedding::from_pretrained() method? This is crucial for future large-model fine-tuning, especially since we already have the EmbeddingFromPretrainedOptions class. Regarding JIT, I’m unsure if libtorch includes a method similar to torch::trace(jitmodule) and torch.jit.is_scripting in Python.
Beta Was this translation helpful? Give feedback.
All reactions